Using Rustlang’s Async Tokio Runtime for CPU-Bound Tasks
Despite the term async and its association with asynchronous network I/O, this blog post argues that the Tokio runtime at the heart of the Rust async ecosystem is also a good choice for CPU-heavy jobs such as those found in analytics engines.
To take full advantage of multiple cores and async I/O, a runtime must be used, and while several alternatives are available to the Rust community, Tokio is the de facto standard. Tokio.rs describes it as: “an asynchronous runtime for the Rust programming language. It provides the building blocks needed for writing network applications.”
While this description emphasizes Tokio’s use for network communications, the runtime can be used for other purposes, as we will explore below.
It turns out that modern analytics engines invariably need to handle client requests from a network, as well as use a network to communicate with object storage systems such as AWS S3, GCP Cloud Storage and Azure Blob Storage.
Thus, any such system implemented in Rust will end up using Tokio for its network and at least part of its storage I/O (yes, I know, initially Tokio async file IO is not really async, but it is coming soon).
