Anouncing async_executors, a building block for executor agnostic libraries

Welcome to my very first blog post. I have been working on asynchronous Rust software for a year now and like most, I want to write applications. However, most of our code could serve several projects, so we prefer to put re-usable functionality in libraries. By far most of the code I write resides in libraries, not in binary crates.

In async code, one often needed functionality is spawning tasks. I would argue it’s one of the most fundamental components of async Rust, because without an executor, you’re simply not going to write async software. This requires a decision. Which executor can my library use? How do I convey the need for an executor in my API? Popular executor providers like tokio and async-std are frameworks. Their vision is: “We give you everything you need to build an asynchronous application in one package”.

As a library author I think it’s important to keep options open. Push decisions further down the stack. When a decision needs to be made, ideally it’s the application author that decides, not the library author. In this sense, I will not publish a library tied to one framework or another. If I need to spawn tasks, the application developer should be the one deciding which executor will be used.

This avoids a number of problems, not in the least that the dependency tree could be pulling in a bunch of functionality from several frameworks, bloating the application and compile times. Today we have 2 main competitors, but what when we will have 5? It also takes control/clarity away from the application developer about which code runs where and when.

The first time I needed to spawn in a library, I started looking into this problem and last summer I released a crate called naja_async_runtime. This crate provides a mechanism for app developers to define which executor is being used on each thread, and then all code (library or application code alike) can just call rt::spawn. A globally available spawn function which will check the TLS to find the chosen executor.

Beginning of August I was working on a major update for this library and I started to wonder. Is a global spawn function really what we want here? What do other people think? So I made posted a poll on the Rust user forum. These are the results:

Quite an interesting result. Especially since tokio (v0.1 at the time) is heavily oriented towards a global tokio::spawn, juliex was using a global spawn function and async-std which came out later that month only supports a global spawn. Some interesting discussion followed, and I read a very interesting blog post about structured concurrency.

All of this made me postpone the work on the next version of naja_async_runtime and I started to look at my code seeing if passing in generic executors wasn’t a better idea overall.

I am now leaning heavily towards the nursery approach described in the linked post. However, implementing that in an executor agnostic way requires some prior building blocks and again, when decisions are to be made I feel keeping options open is key. So the goal I set out on was to have all three models:

• global spawn fn,
• generic executors
• a nursery

Discussion has started on the tokio issue tracker recently in order to provide structured concurrency, but the aim seems to be for it to be tokio specific, so that’s no-go for anything I’d use.

I set aside the update on naja_async_runtime because both the global spawn function and the nursery can be built out of generic executors. What we really need is to define an interface for executors. We can then plug that into a TLS to make a global spawn function or into a nursery like structure (so that the nursery itself is executor agnostic).

This is where async_executors comes in. It provides the generic executors, paving the way for more advanced spawning models to be built on top of it. In the rest of this blog post I want to talk about the library and explain some of the obstacles I ran into.

Searching for an interface.

An interface is a contract. We say to the user: “You don’t know the exact type of this object, but you can count on it behaving a certain way”. Our challenge here is to find the right interface for executors and then paper over the differences of existing implementations so that we get the consistent behavior that we can formulate in a contract.

In rust the standard way of creating interfaces is through traits. They can either be used as bounds on type parameters, or if we can’t know types at compile time or we want to keep our code from becoming alphabet soup, trait objects give us even more powerful abstractions.

So what traits can help us here? Futures provides 2 traits for spawning: Spawn and LocalSpawn. As an added benefit, almost every async crate in the ecosystem already depends on futures, so we can use them without pulling in extra bloat in dependent crates. Neither async-std nor tokio implement the traits for their executors, async-std doesn’t even provide an executor type, just a global spawn function. The executor that runs on WASM, wasm-bindgen-futures does not implement the traits either.

/// The `Spawn` trait allows for pushing futures onto an executor that will
/// run them to completion.
pub trait Spawn {
    /// Spawns a future that will be run to completion.
    ///
    /// # Errors
    ///
    /// The executor may be unable to spawn tasks. Spawn errors should
    /// represent relatively rare scenarios, such as the executor
    /// having been shut down so that it is no longer able to accept
    /// tasks.
    fn spawn_obj(&self, future: FutureObj<'static, ()>) -> Result<(), SpawnError>;

    /// Determines whether the executor is able to spawn new tasks.
    ///
    /// This method will return `Ok` when the executor is *likely*
    /// (but not guaranteed) to accept a subsequent spawn attempt.
    /// Likewise, an `Err` return means that `spawn` is likely, but
    /// not guaranteed, to yield an error.
    #[inline]
    fn status(&self) -> Result<(), SpawnError> {
        Ok(())
    }
}

The first step for async_executors was to see if we can implement the futures traits for both tokio, async-std and wasm-bindgen-futures. We create wrapper types for each and implement the traits. It sounds simple enough. The spawn methods of both libraries return a JoinHandle, but when you drop it, the future keeps running in the background.

Dark clouds

However, when looking a bit closer, complications arise.

The tokio Runtime should never be dropped in async context. When using spawn, it’s possible that orphaned child tasks outlive parent tasks and since our wrapper has a tokio runtime in a Arc<Mutex>, the last place to hold one will do the drop. If that’s in async context, a thread blows up. To work around this problem, a wrapper for tokio::runtime::Handle is provided. The problem with that one is that it is only operational while the parent Runtime is still alive and it will silently drop tasks you try to spawn after the Runtime is gone.

What happens if a task panics? Tokio is the only executor which catch_unwinds the panic and transforms it into an error observable by awaiting the JoinHandle. All other executors, including wasm-bindgen-futures just let the thread running the task unwind. When catching unwinds, we need to be sure that the resulting program is unwind safe as well. We’ll sure want a clearly documented behavior here, preferably consistent too. Tokio considers that requiring the future to be Send + 'static is the same as UnwindSafe, however that isn’t really the case. It wacks some moles, but not all. A prime example is a parking_lot::Mutex, which is Send + 'static, but !UnwindSafe.

Tokio uses AssertUnwindSafe on the futures you spawn, which tells the compiler, I assert that this is unwind safe even if you think it’s not. That’s a bold statement for user supplied code they know nothing about. The reason for this is that futures often contain a &mut self. So you wouldn’t be able to spawn those anymore if they didn’t wrap it in AssertUnwindSafe. Not catching the unwind has the downside of the executor losing it’s working threads, also tearing down other completely unrelated tasks.

Unwind safety problems are quite rare in safe Rust code, which is probably why not everybody is aware of them or understands the issue. Still when that rare bug happens you will be rather sour trying to figure out what’s going wrong. In the case of tokio, it means you should make sure your futures are unwind safe, but that isn’t mentioned in the documentation.

As for async_executors, the panic can only be detected when awaiting the handle from tokio, so there isn’t much we can do about it except warning users in the documentation. Especially where we remove the Send bound for local spawning.

Overhead

Lastly we want to have an idea of the performance overhead we introduce. These traits take a FutureObj, which is a heap allocated future. Executors generally have to allocate again to have a heterogeneous queue of futures. This will lead to double boxing unless the traits were to be implemented on the executors directly rather than on wrappers. You can find benchmark details on the API of async_executors here.

A better interface?

A problem with the futures traits is that they are incompatible with structured concurrency. This makes them of questionable merit. The importance of joining tasks has found it’s way into rust, and futures even provides us with a solution: RemoteHandle. It turns out tokio and async-std don’t use it because it uses a oneshot channel to pass back the output of the future to the task awaiting the handle. As it is possible to make a more performant handle by tapping directly into the executor, both have their own JoinHandle type, which leaves us with 3 different JoinHandle types that are incompatible as far as the type system is concerned. Interoperability goes out of the door.

If we want to remedy the situation we need a contract (trait) promising that a type can spawn a task and return a JoinHandle you can await. Added benefit, you can now spawn tasks that return a value other than an empty tuple and errors can be bubbled up too. Async_executors provides the SpawnHandle and SpawnHandleLocal traits for all supported executors.

We have 3 different types, again with compatibility issues. RemoteHandle drops the future when the handle get’s dropped. Tokio and async-std detach. To solve this, we wrap the task in futures Abortable on tokio and async-std, and add a detach method to JoinHandle. This way you can choose whether to drop or detach and we get consistent behavior for all three.

If the future panics, we get:

• tokio won’t unwind any threads,
• async-std will unwind both the executor thread as well as the thread awaiting the handle
• futures RemoteHandle will unwind only the thread that awaits the handle.

To get more consistency, we unwind the thread that is awaiting the handle for tokio. The only inconsistency left is that async-std will sacrifice the executor working thread while others won’t, but I wasn’t willing to introduce another potentially dangerous catch_unwind to bring it in line, so this inconsistency remains.

None of the join handle types can work with an output type that is !Send, so for local spawning we can’t use them. I have proposed a pull request with futures to remove the Send bound from the output of RemoteHandle. It has been merged, so on the next release of futures, RemoteHandle will support a !Send output. We use it for our JoinHandle in SpawnHandleLocal implementations.

Object safety

An extra challenge is that the JoinHandle needs to be generic over the output type of the task. So either we make the SpawnHandle generic, or we make the spawn_handle method generic, but in that case our trait is no longer object safe. This is fine for a function that would just take impl SpawnHandle as a parameter but is a problem if an API needs to take an executor and store it to use it over some time. The spawn_handle method is also generic over the future type, so for an object safe version we also need to box the future, leading to double boxing.

Async_executors provides 2 versions of the trait, SpawnHandle and SpawnHandleOs<T>. The latter is object safe. There is also LocalSpawnHandle and LocalSpawnHandleOs<T>.

Benefits

There are obvious reasons to use interfaces when you write libraries, but there’s benefits for applications as well.

Performance

Ever wonder if tokio, futures or async-std gives better performance for your application? By abstracting over the executor, experimenting with different ones can be a matter of changing one line of code. Unfortunately depending on your use case, tight coupling between the executor and the reactor might spoil your fun.

Debugging

Have you run into obscure bugs where you wondered if it wasn’t an issue with the executor? Quickly switch to another one to see if it still happens.

Logging

Tracing-futures is a nice library which allows you to instrument an executor. You can automatically add a piece of structured data to anything logged from within tasks spawned by the instrumented executor.

use 
{
	tracing_subscriber :: { fmt::{ Subscriber, time::ChronoLocal } ,
	tracing_futures    :: { Instrument                           } ,
	async_executors    :: { AsyncStd                             } ,
};

Subscriber::builder()

	.with_timer( ChronoLocal::rfc3339() )
	.json()
	.with_max_level( tracing::Level::TRACE )
	.init()
;

let exec = AsyncStd::default();
let ex2  = exec.clone().instrument( tracing::info_span!( "server" ) );
let ex3  = exec.clone().instrument( tracing::info_span!( "relay"  ) );
let ex4  = exec.clone().instrument( tracing::info_span!( "client" ) );

let task server = async move
{
	// use ex2 to spawn

}.instrument( tracing::info_span!( "server" ) );

exec.spawn( server ).expect( "spawn server" );

In the example above we can pass ex2, ex3 andex4 to 3 different tasks, and anything logged from child tasks, even in dependencies using macros from the log crate, will be tagged.

This works because tracing-futures re-implements Spawn and LocalSpawn on the wrapped executor as long as the wrapped type implements them. Until now, the only thing you could pass to this were executors from the futures library that implemented those traits. With async_executors you can pass the other supported executors as well.

The new traits that async_executors provides all have blanket impls for the tracing-futures wrappers.

Featureset

The wrappers in this library can do a few things the native executors can’t:

• TokioCt can spawn !Send futures.
• JoinHandle allows both detaching and dropping the running future, which neither async-std nor tokio provides out of the box.

Conclusion

All in all I am quite satisfied with the result. It tries to do as well as possible given the constraints we are up against and it does make it possible for libraries to specify what kind of contract they need for spawning and clients can actually choose from a reasonable range of executors. If other executors come along, support for them can be added.

It’s unfortunate that so much fine print needs to be added to the contract in order to support tokio. When using async-std, the downside is that you basically can configure nothing: no well defined lifetime of an executor, no initialize code on worker threads, no decision on how many worker thread, etc.

There is some overhead, but for most asynchronous applications the cost of spawning is neglectable, where as the lack of interoperability is not. You can find benchmark details on the API of async_executors here.