Async from scratch 3: Pinned against the wall

So, we've covered polling. We've tackled sleeping (and waking). Going back to the definition, that leaves us with one core concept left to conquer: pinning!

But before we get there, there was that one tiny other aside I'd like to go over, just so we can actually use the real trait this time. It'll be quick, I promise.¹ And then we'll be back to what you came here for.

Intermission: Letting our types associate

Let's ignore poll() completely for a second, and focus on another sneaky² change I pulled between Future and SimpleFuture:

trait Future {
    type Output;
}

trait SimpleFuture<Output> {}

What's the difference between these? Future::Output is an "associated type". Associated types are very similar to trait generics, but they aren't used to pick the right trait implementation.

The way I tend to think of this is that if we think of our type as a kind-of-a-function, then generics would be the arguments, while its associated types would be the return value(s).

We can define our trait implementations for any combination of generics, but for a given set of base type³, each associated type must resolve to exactly one real type.

For example, this is perfectly fine:

struct MyFuture;

impl SimpleFuture<u64> for MyFuture {}
impl SimpleFuture<u32> for MyFuture {}

Or this blanket implementation:

struct MyFuture;

impl<T> SimpleFuture<T> for MyFuture {}

But this isn't, because the implementations conflict with each other:⁴

struct MyFuture;

impl Future for MyFuture {
    type Output = u64;
}
impl Future for MyFuture {
    type Output = u32;
}

error[E0119]: conflicting implementations of trait `Future` for type `MyFuture`
  --> src/main.rs:13:1
   |
10 | impl Future for MyFuture {
   | ------------------------ first implementation here
...
13 | impl Future for MyFuture {
   | ^^^^^^^^^^^^^^^^^^^^^^^^ conflicting implementation for `MyFuture`

For more information about this error, try `rustc --explain E0119`.
error: could not compile `cargo0OpMSm` (bin "cargo0OpMSm") due to 1 previous error

We're also not allowed to do a blanket implementation that covers multiple types:

struct MyFuture;

impl<T> Future for MyFuture {
    type Output = T;
}

error[E0207]: the type parameter `T` is not constrained by the impl trait, self type, or predicates
  --> src/main.rs:10:6
   |
10 | impl<T> Future for MyFuture {
   |      ^ unconstrained type parameter

For more information about this error, try `rustc --explain E0207`.
error: could not compile `cargojraklM` (bin "cargojraklM") due to 1 previous error

So... why is this useful? Well, primarily it helps type inference do a better job: if we know the type of x, then we also know the type of f.await, since it can only have one (Into)Future implementation⁵, which can only have one Output type.⁶

There's also a bit of a convenience benefit: our generic code we can refer to the associated type as T::Output, rather than having to bind a new type parameter. These mean roughly the same thing:

fn run_simple_future<Output, F: SimpleFuture<Output>>() -> Output {
    todo!()
}

fn run_future<F: Future>() -> F::Output {
    todo!()
}

// Though this also works
fn run_future_2<Output, F: Future<Output = Output>>() -> Output {
    todo!()
}

Well, now that that's out of the way.. let's get back on track. You came here to get pinned, and I wouldn't want to disappoint...

But why, though?

Back in the ancient days of a-few-weeks-ago, I showed how we can translate any async fn into a state machine enum and a custom Future implementation.

Let's try doing that for (a slightly simplified version of) the trick-or-treat example that the whole series started with:

// No, I haven't read "Falsehoods programmers believe about addresses",
// why would you ask that?
struct House {
    street: String,
    house_number: u16,
}
struct Candy;

// Does nothing except wait
// (This one actually doesn't even do that.. we'll get there. Use tokio::task::yield_now.)
async fn yield_now() {}

async fn demand_treat(house: &House) -> Result<Candy, ()> {
    for _ in 0..house.house_number {
        // Walking to the house takes time
        yield_now().await;
    }
    Ok(Candy)
}
async fn play_trick(house: &House) {
    todo!();
}

async fn trick_or_treat() {
    // Address chosen by fair dice roll. Obviously. Don't worry about it.
    let house = House {
        street: "Riksgatan".to_string(),
        house_number: 3,
    };
    if demand_treat(&house).await.is_err() {
        play_trick(&house).await;
    }
}

Well that's simple enough, let's give it a go..

struct DemandTreat<'a> {
    house: &'a House,
}
impl SimpleFuture<Result<Candy, ()>> for DemandTreat<'_> {
    fn poll(&mut self) -> Poll<Result<Candy, ()>> { todo!() }
}
struct PlayTrick<'a> {
    house: &'a House,
}
impl SimpleFuture<()> for PlayTrick<'_> {
    fn poll(&mut self) -> Poll<()> { todo!() }
}

enum TrickOrTreat<'a> {
    Init,
    DemandTreat {
        house: House,
        demand_treat: DemandTreat<'a>,
    },
    PlayTrick {
        house: House,
        play_trick: PlayTrick<'a>,
    },
}
impl<'a> SimpleFuture<()> for TrickOrTreat<'a> {
    fn poll(&mut self) -> Poll<()> {
        loop {
            match self {
                TrickOrTreat::Init => {
                    let house = House {
                        street: "Riksgatan".to_string(),
                        house_number: 3,
                    };
                    *self = TrickOrTreat::DemandTreat {
                        house,
                        demand_treat: DemandTreat {
                            house: &house,
                        }
                    };
                }
                _ => todo!(),
            }
        }
    }
}

error[E0597]: `house` does not live long enough
  --> src/main.rs:76:36
   |
64 | impl<'a> SimpleFuture<()> for TrickOrTreat<'a> {
   |      -- lifetime `'a` defined here
...
69 |                     let house = House {
   |                         ----- binding `house` declared here
...
73 |                     *self = TrickOrTreat::DemandTreat {
   |                     ----- assignment requires that `house` is borrowed for `'a`
...
76 |                             house: &house,
   |                                    ^^^^^^ borrowed value does not live long enough
...
79 |                 }
   |                 - `house` dropped here while still borrowed

error[E0382]: borrow of moved value: `house`
  --> src/main.rs:76:36
   |
69 |                     let house = House {
   |                         ----- move occurs because `house` has type `House`, which does not implement the `Copy` trait
...
74 |                         house,
   |                         ----- value moved here
75 |                         demand_treat: DemandTreat {
76 |                             house: &house,
   |                                    ^^^^^^ value borrowed here after move
   |
note: if `House` implemented `Clone`, you could clone the value
  --> src/main.rs:4:1
   |
4  | struct House {
   | ^^^^^^^^^^^^ consider implementing `Clone` for this type
...
74 |                         house,
   |                         ----- you could clone this value

Some errors have detailed explanations: E0382, E0597.
For more information about an error, try `rustc --explain E0382`.
error: could not compile `cargorz58SU` (bin "cargorz58SU") due to 2 previous errors

..oh, right. Rust really doesn't like structs that borrow themselves. We can't even express this well in its type system: we can't bind the lifetime of the DemandTreat to the lifetime of the TrickOrTreat, it has to come from an external type parameter.⁷. We can't even construct TrickOrTreat::DemandTreat without the DemandTreat! What could we possibly do about this predicament?

Well. We could just pass the ownership of the House into DemandTreat, and then have it return it once finished. (That is, change the signature from async fn demand_treat(house: &House) -> Result<Candy, ()> to async fn demand_treat(house: House) -> (House, Result<Candy, ()>).) That works for our simple example⁸, but it breaks if we're borrowing the data ourselves, or if something else is also borrowing it at the same time as DemandTreat. Probably workable with enough elbow grease, but not great.

We could try wrapping the DemandTreat in an Option.. that'd solve the construction paradox at least. But it wouldn't do diddly to solve our lifetime problem.

We could try clone-ing the House.. but that assumes that it is cloneable⁹. We could get around that by wrapping the house in Arc-flavoured bubblewrap, but that assumes that we own it directly.¹⁰ Blech.

Well, that all sucks. Maybe there is something to that old "C" thing, after all. Y'know what. Clearly it's the compiler that is wrong. How about we just use some raw pointers instead. Clearly, I can be trusted with raw pointers. Right?

use std::task::ready;

struct DemandTreat {
    house: *const House,
    current_house: u16,
}
impl SimpleFuture<Result<Candy, ()>> for DemandTreat {
    fn poll(&mut self) -> Poll<Result<Candy, ()>> {
        if self.current_house == unsafe { (*self.house).house_number } {
            Poll::Ready(Ok(Candy))
        } else {
            self.current_house += 1;
            Poll::Pending
        }
    }
}
struct PlayTrick {
    house: *const House,
}
impl SimpleFuture<()> for PlayTrick {
    fn poll(&mut self) -> Poll<()> { todo!() }
}

enum TrickOrTreat {
    Init,
    DemandTreat {
        house: House,
        demand_treat: DemandTreat,
    },
    PlayTrick {
        house: House,
        play_trick: PlayTrick,
    },
}
impl SimpleFuture<()> for TrickOrTreat {
    fn poll(&mut self) -> Poll<()> {
        loop {
            match self {
                TrickOrTreat::Init => {
                    *self = TrickOrTreat::DemandTreat {
                        house: House {
                            street: "Riksgatan".to_string(),
                            house_number: 3,
                        },
                        demand_treat: DemandTreat {
                            house: std::ptr::null(),
                            current_house: 0,
                        },
                    };
                    let TrickOrTreat::DemandTreat { house, demand_treat } = self else { unreachable!() };
                    demand_treat.house = house;
                }
                TrickOrTreat::DemandTreat { house, demand_treat } => {
                    match ready!(demand_treat.poll()) {
                        Ok(_) => return Poll::Ready(()),
                        Err(_) => todo!(),
                    }
                }
                _ => todo!(),
            }
        }
    }
}

And it works compiles! I hear that's basically the same thing. Time to celebrate. Right?

...right?

...perhaps not yet.¹¹ As always, raw pointers come with a cost. First, we obviously lose the niceties of borrow checking. In fact, we arguably have a lifetime bug already!¹² But there's also a deeper problem in here. Pointers (and references) point at the absolute memory location. But once poll() has returned, whoever is running the future has full ownership. They're free to move it around as they please.

let mut future = TrickOrTreat::Init;
future.poll();
// future.demand_treat.house points at future.house
// move future somewhere else
let mut future2 = future;
// future2.demand_treat.house *still* points at future.house, not future2.house!
future2.poll();

...oh dear.¹³ And you don't even need to own it either, std::mem::swap and std::mem::replace are happy to move objects that are behind (mutable) references, as long as you have a valid object to replace them with:

let mut future = TrickOrTreat::Init;
future.poll();
let mut future2 = std::mem::replace(&mut future, TrickOrTreat::Init);
// future *is* now still a valid object, but not the one we meant to reference.
// And future.house definitely isn't valid, since we aren't on that branch of the enum.
future2.poll();

Welp. So how can we prevent ourselves from being moved, while still allowing other writes? We stick a Pin on that shit!

Pinning, actually

A Pin wraps a mutable reference of some kind (&mut T, Box<T>, and so on), but restricts us (in the safe API) to reading and replacing the value entirely, without the ability to move things out of it (or to mutate only parts of them¹⁴).

It looks like this:

use std::ops::{Deref, DerefMut};

// SAFETY: Don't access .0 directly
struct Pin<T>(T);

impl<T: Deref> Pin<T> {
    // SAFETY: `ptr` must never be moved after this function has been called
    unsafe fn new_unchecked(ptr: T) -> Self {
        Self(ptr)
    }

    fn get_ref(&self) -> &T::Target {
        &self.0
    }
}

impl<T: DerefMut> Pin<T> {
    // SAFETY: The returned reference must not be moved
    unsafe fn get_unchecked_mut(&mut self) -> &mut T::Target {
        &mut self.0
    }

    // Allow reborrowing Pin<OwnedPtr> as Pin<&mut T>
    fn as_mut(&mut self) -> Pin<&mut T::Target> {
        Pin(&mut self.0)
    }

    fn set(&mut self, value: T::Target) where T: DerefMut, T::Target: Sized {
        *self.0 = value;
    }
}

// As a convenience, `Deref` lets us call x.get_ref().y as x.y
impl<T: Deref> Deref for Pin<T> {
    type Target = T::Target;
    fn deref(&self) -> &Self::Target {
        self.get_ref()
    }
}

We can then create our object "normally" and then pin it (promising to uphold its requirements from that point onwards, but also gaining its self-referential powers):

struct Foo {
    bar: u64,
}
let mut foo = Foo { bar: 0 };
// Creating a pin is unsafe, because we need to promise that we won't use the original value directly anymore, even after the pin is dropped
let mut foo = unsafe { Pin::new_unchecked(&mut foo) };
// Reading is safe
println!("=> {} (initial)", foo.bar);
// Replacing is safe
foo.set(Foo { bar: 1 });
println!("=> {} (replaced)", foo.bar);
// Arbitrary writing is unsafe
unsafe { foo.get_unchecked_mut().bar = 2; }
println!("=> {} (written)", foo.bar);
// We can still move if we use get_unchecked_mut(), but it's also unsafe!
let old_foo = unsafe { std::mem::replace(foo.get_unchecked_mut(), Foo { bar: 3 }) };
println!("=> {} (moved)", old_foo.bar);
println!("=> {} (replacement)", foo.bar);

=> 0 (initial)
=> 1 (replaced)
=> 2 (written)
=> 2 (moved)
=> 3 (replacement)

Managing the self-reference itself is still as unsafe as ever, but by designing our API around to pin the state, we can make sure that whoever actually owns our state is forced to uphold our constraints. For example, for Future:

// std::pin::Pin is special-cased, we can't use arbitrary types as receivers (`self`) yet in stable
use std::pin::Pin;

trait PinnedFuture<Output> {
    fn poll(self: Pin<&mut Self>) -> Poll<Output>;
}

There are also some APIs for pinning things safely. Boxes own their values and their targets are never moved¹⁵, so wrapping those is fine:

impl<T> Box<T> {
    fn into_pin(self) -> Pin<Box<T>> {
        // SAFETY: `Box` owns its value and is never moved, so it will be dropped together with the `Pin`
        unsafe { Pin::new_unchecked(self) }
    }

    fn pin(value: T) -> Pin<Box<T>> {
        Self::new(value).into_pin()
    }
}

Finally, we can pin things on the stack! We don't have a special type for "owned-place-on-the-stack", and &mut T returns control to the owner once dropped, so that's also not legal. Instead, we need to use the pin! macro to ensure that the original value can never be used:

struct Foo;

let foo = std::pin::pin!(Foo);

// Equivalent to:
let mut foo = Foo;
// SAFETY: `foo` is shadowed in its own scope, so it can never be accessed directly after this point
let foo = unsafe { Pin::new_unchecked(&mut foo) };

(std's pin! does some special magic to allow it to be used as an expression, but the older futures::pin_mut! really did do this.)

Well, sometimes at least

But if we start defining Future in terms of Pin.. won't that add a whole bunch of (mental) overhead for the cases that don't require pinning? Suddenly we need to worry about whether all of our Future-s are pinned correctly. That seems like a lot of work. We could provide separate UnpinnedFuture and PinnedFuture traits, but then we have to deal with defining how the two interact. Also not great.

That's why Rust provides the Unpin marker trait:

// SAFETY: Only implement for types that can never contain references to themselves.
trait Unpin {}

It lets types opt out of pinning, letting you use Pin<&mut T> as if it was equivalent to &mut T as long as T is Unpin:

impl<T: Deref> Pin<T>
where
    T::Target: Unpin,
{
    fn new(ptr: T) -> Self {
        // SAFETY: `ptr` is unpinned
        unsafe { Self::new_unchecked(ptr) }
    }

    fn get_mut(&mut self) -> &mut T::Target where T: DerefMut {
        // SAFETY: `ptr` is unpinned
        unsafe { self.get_unchecked_mut() }
    }
}

// Convenience alias for get_mut()
impl<T: DerefMut> DerefMut for Pin<T>
where
    T::Target: Unpin,
{
    fn deref_mut(&mut self) -> &mut Self::Target {
        self.get_mut()
    }
}

We can then create and mutate pins as we please.. as long as we stick to Unpin-ned data:

struct Foo {
    bar: u64,
}
impl Unpin for Foo {}

let mut foo = Foo { bar: 0 };
Pin::new(&mut foo).bar = 1;
foo.bar = 2;

And as a final nod to convenience.. Rust actually implements Unpin by default for new types, as long as they only contain values that are also Unpin. Since that's going to exclude types that do contain self-references (*mut T is Unpin by itself), Rust provides the PhantomPinned type which does nothing except be Unpin.¹⁶ For example:

use std::marker::PhantomPinned;

struct ImplicitlyUnpin;
struct ExplicitlyNotUnpin(ImplicitlyUnpin, PhantomPinned);
struct ImplicitlyNotUnpin(ExplicitlyNotUnpin);
struct ExplicitlyUnpin(ImplicitlyNotUnpin);

impl Unpin for ExplicitlyUnpin {}

fn assert_unpin<T: Unpin>() {}
assert_unpin::<ImplicitlyUnpin>;
assert_unpin::<ExplicitlyUnpin>;
// Will fail, since these aren't unpinnable
assert_unpin::<ExplicitlyNotUnpin>;
assert_unpin::<ImplicitlyNotUnpin>;

error[E0277]: `PhantomPinned` cannot be unpinned
  --> src/main.rs:16:16
   |
16 | assert_unpin::<ExplicitlyNotUnpin>;
   |                ^^^^^^^^^^^^^^^^^^ within `ExplicitlyNotUnpin`, the trait `Unpin` is not implemented for `PhantomPinned`
   |
   = note: consider using the `pin!` macro
           consider using `Box::pin` if you need to access the pinned value outside of the current scope
note: required because it appears within the type `ExplicitlyNotUnpin`
  --> src/main.rs:6:8
   |
6  | struct ExplicitlyNotUnpin(ImplicitlyUnpin, PhantomPinned);
   |        ^^^^^^^^^^^^^^^^^^
note: required by a bound in `assert_unpin`
  --> src/main.rs:12:20
   |
12 | fn assert_unpin<T: Unpin>() {}
   |                    ^^^^^ required by this bound in `assert_unpin`

For more information about this error, try `rustc --explain E0277`.
error: could not compile `cargomxhoMm` (bin "cargomxhoMm") due to 1 previous error

A little party projection never killed nobody...

Let's say we have a pinned Future like this:

struct Timeout<F> {
    inner_future: F,
    elapsed_ticks: u64,
}

let timeout: Pin<&mut Timeout<InnerFuture>>;

The safe API on Pin only lets us replace our whole Timeout (via Pin::set), but that's not super useful for us. We need to keep our old InnerFuture, that's why we're pinning it to begin with!

To address this, we need to project our InnerFuture, temporarily splitting our struct into its individual fields while maintaining the pinning requirements.

But that raises another question; should .inner_future give a &mut InnerFuture or a Pin<&mut InnerFuture>? What about .elapsed_ticks? The short answer is.. we decide.

From Rust's perspective, either answer is valid as long as we obey the cardinal Pin rule that we cannot provide a regular &mut once we have produced a Pin for a given field.¹⁷

From our perspective, we probably want inner_future to be Pin (since it's also a Future), but elapsed_ticks doesn't have any reason to be.

Hence, we should write down a single way to project access into each field. One way¹⁸ would be to write a method for each field:

impl<F> Timeout<F> {
    fn inner_future(self: Pin<&mut Self>) -> Pin<&mut F> {
        // SAFETY: `inner_future` is pinned structurally
        unsafe {
            Pin::new_unchecked(&mut self.get_unchecked_mut().inner_future)
        }
    }

    fn elapsed_ticks(self: Pin<&mut Self>) -> &mut u64 {
        // SAFETY: `elapsed_ticks` is _not_ pinned structurally
        unsafe {
            &mut self.get_unchecked_mut().elapsed_ticks
        }
    }
}

However, this doesn't allow us to access multiple fields concurrently, since Rust doesn't have a way to express "split borrows" in function signatures at the moment:

let mut timeout: Pin<&mut Timeout<InnerFuture>> = std::pin::pin!(Timeout { inner_future: InnerFuture, elapsed_ticks: 0 });
let inner_future = timeout.as_mut().inner_future();
let elapsed_ticks = timeout.as_mut().elapsed_ticks();
inner_future.poll();
*elapsed_ticks += 1;

error[E0499]: cannot borrow `timeout` as mutable more than once at a time
  --> src/main.rs:38:21
   |
37 | let inner_future = timeout.as_mut().inner_future();
   |                    ------- first mutable borrow occurs here
38 | let elapsed_ticks = timeout.as_mut().elapsed_ticks();
   |                     ^^^^^^^ second mutable borrow occurs here
39 | inner_future.poll();
   | ------------ first borrow later used here

For more information about this error, try `rustc --explain E0499`.
error: could not compile `cargokXjM9v` (bin "cargokXjM9v") due to 1 previous error

Instead, we can build a single projection struct that projects access to all fields simultaneously:

struct TimeoutProjection<'a, F> {
    inner_future: Pin<&'a mut F>,
    elapsed_ticks: &'a mut u64,
}

impl<F> Timeout<F> {
    fn project(mut self: Pin<&mut Self>) -> TimeoutProjection<F> {
        // SAFETY: This function defines the canonical projection for each field
        unsafe {
            let this = self.get_unchecked_mut();
            TimeoutProjection {
                // SAFETY: `inner_future` is pinned structurally
                inner_future: Pin::new_unchecked(&mut this.inner_future),
                // SAFETY: `elapsed_ticks` is _not_ pinned structurally
                elapsed_ticks: &mut this.elapsed_ticks,
            }
        }
    }
}

And use it like so:

let mut timeout: Pin<&mut Timeout<InnerFuture>> = std::pin::pin!(Timeout { inner_future: InnerFuture, elapsed_ticks: 0 });
let projection = timeout.project();
let inner_future = projection.inner_future;
let elapsed_ticks = projection.elapsed_ticks;
inner_future.poll();
*elapsed_ticks += 1;

Woohoo! This is fine, since we have one method that borrows all of Timeout, and produces one TimeoutProjection that is equivalent to it. It's okay for the TimeoutProjection to borrow multiple things from the Timeout, as long as we (project()) know that those borrows are disjoint.¹⁹

But that's still a bit tedious, having to effectively write each struct thrice²⁰. Conveniently enough, There's A Crate For That²¹. Our TimeoutProjection struct could be generated by pin-project like this:

// Generates `TimeoutProjection` and `Timeout::project()` as above
#[pin_project::pin_project(project = TimeoutProjection)]
struct Timeout<F> {
    #[pin] // Projected as `Pin<&mut F>`
    inner_future: F,
    // No `#[pin]`, projected as `&mut F`
    elapsed_ticks: u64,
}

let mut timeout: Pin<&mut Timeout<InnerFuture>> = std::pin::pin!(Timeout { inner_future: InnerFuture, elapsed_ticks: 0 });
let projection = timeout.project();
let inner_future = projection.inner_future;
let elapsed_ticks = projection.elapsed_ticks;
inner_future.poll();
*elapsed_ticks += 1;

Whew. We still have to call Project²², but at least we're mostly back in familiar Rust territory again!

And finally, the same transformation works for enums as well:

#[pin_project::pin_project(project = TimeoutProjection)]
enum Timeout<F> {
    Working {
        #[pin]
        inner_future: F,
        elapsed_ticks: u64,
    },
    Expired,
}

// #[pin_project] is equivalent to:
enum ManualTimeoutProjection<'a, F> {
    Working {
        inner_future: Pin<&'a mut F>,
        elapsed_ticks: &'a mut u64,
    },
    Expired,
}
impl<F> Timeout<F> {
    fn manual_project(mut self: Pin<&mut Self>) -> ManualTimeoutProjection<F> {
        // SAFETY: This function defines the canonical projection for each field
        unsafe {
            match self.get_unchecked_mut() {
                Timeout::Working {
                    inner_future,
                    elapsed_ticks,
                } => ManualTimeoutProjection::Working {
                    // SAFETY: `inner_future` is pinned structurally
                    inner_future: Pin::new_unchecked(inner_future),
                    // SAFETY: `elapsed_ticks` is _not_ pinned structurally
                    elapsed_ticks,
                },
                Timeout::Expired => ManualTimeoutProjection::Expired,
            }
        }
    }
}

There is, however, one caveat to using pin-project: While Rust normally avoids implementing Unpin if any field is !Unpin, pin-project only considers #[pin]-ned fields. Normally this is enforced by the type system anyway (since you can't call self: Pin methods otherwise), but if you use PhantomPinned then it must always be #[pin]-ned to be effective.

Onwards, to the beginning!

Okay, now we should finally have the tools to make TrickOrTreat safe to interact with!

use std::{marker::PhantomPinned, task::ready};

struct DemandTreat {
    house: *const House,
    current_house: u16,
}
impl PinnedFuture<Result<Candy, ()>> for DemandTreat {
    fn poll(mut self: Pin<&mut Self>) -> Poll<Result<Candy, ()>> {
        if self.current_house == unsafe { (*self.house).house_number } {
            Poll::Ready(Ok(Candy))
        } else {
            self.current_house += 1;
            Poll::Pending
        }
    }
}
struct PlayTrick {
    house: *const House,
}
impl PinnedFuture<()> for PlayTrick {
    fn poll(self: Pin<&mut Self>) -> Poll<()> { todo!() }
}

#[pin_project::pin_project(project = TrickOrTreatProjection)]
enum TrickOrTreat {
    Init,
    DemandTreat {
        house: House,
        #[pin]
        demand_treat: DemandTreat,
        // SAFETY: self must be !Unpin because demand_treat references house
        #[pin]
        _pin: PhantomPinned,
    },
    PlayTrick {
        house: House,
        #[pin]
        play_trick: PlayTrick,
        // SAFETY: self must be !Unpin because play_trick references house
        #[pin]
        _pin: PhantomPinned,
    },
}
impl PinnedFuture<()> for TrickOrTreat {
    fn poll(mut self: Pin<&mut Self>) -> Poll<()> {
        loop {
            match self.as_mut().project() {
                TrickOrTreatProjection::Init => {
                    self.set(TrickOrTreat::DemandTreat {
                        house: House {
                            street: "Riksgatan".to_string(),
                            house_number: 3,
                        },
                        demand_treat: DemandTreat {
                            house: std::ptr::null(),
                            current_house: 0,
                        },
                        _pin: PhantomPinned,
                    });
                    let TrickOrTreatProjection::DemandTreat { house, mut demand_treat, .. } = self.as_mut().project() else { unreachable!() };
                    demand_treat.house = house;
                }
                TrickOrTreatProjection::DemandTreat { house, demand_treat, .. } => {
                    match ready!(demand_treat.poll()) {
                        Ok(_) => return Poll::Ready(()),
                        Err(_) => {
                            // We need to move the old house out of `self` before we replace it
                            let house = std::mem::replace(house, House {
                                street: String::new(),
                                house_number: 0,
                            });
                            self.set(TrickOrTreat::PlayTrick {
                                house,
                                play_trick: PlayTrick {
                                    // We still don't have the address of house-within-TrickOrTreat::PlayTrick
                                    house: std::ptr::null(),
                                },
                                _pin: PhantomPinned,
                            });
                            let TrickOrTreatProjection::PlayTrick { house, mut play_trick, .. } = self.as_mut().project() else { unreachable!() };
                            play_trick.house = house;
                        },
                    }
                }
                TrickOrTreatProjection::PlayTrick { play_trick, .. } => {
                    ready!(play_trick.poll());
                    return Poll::Ready(());
                },
            }
        }
    }
}

Caveats

I'm honestly still not sure about the best way to represent the self-reference "properly" in the type system.

TrickOrTreat is safe and self-contained (as long as you only construct ::Init), but DemandTreat and PlayTrick are not, since they contain unmanaged raw pointers that could end up dangling. We could use references instead, but I'm honestly not sure about whether &mut references could end up causing undefined behaviour due to aliasing. The series is ultimately not about showing what to do, but about explaining some of the magic that is usually hidden from view.

Going forward

Well.. that took a bit longer than I had meant for it to. But now we're finally through the basic building blocks of an async fn!

But an async fn alone isn't all that useful, so next up I'd like to go over how we can use those primitives to run multiple Future-s simultaneously in the same thread! Isn't that was asynchronicity was supposed to be all about, anyway?