Debugging Async Mysteries in Axum - Rust Guide to Handling Timeout Issues
Recently I stumbled upon a behavior of Rust app that uses axum web framework that took me some time to understand. Imagine setting up an async handler in Axum, only to find it mysteriously halting mid-execution. Here’s what happened when I encountered this timeout behavior and how to handle it effectively.
and no sign of log Finished executing info_handler at all. The code above is super simplified, actual production code is more complex (for example, it would make calls to S3-like object storage, you see in the logs it started GetObject operation, but never finishes). The answer to this behavior is in the title, timeouts!
Axum provides a flexible way of extending router using middleware well-integrated with tower. ServiceBuilder has a way to configure timeouts, and it fails request if it takes longer than a defined timeout. So far so good, right? (Further reading is going to be a bit technical, get ready!)
Failing request sounds too generic. What would happen with those async HTTP calls (S3 calls) or async filesystem operation that are happening? And to understand this better we need to keep in mind that in Rust Future must be polled in order to progress! Let's model slowness of write to disk operation and show how timeout makes the data written to a file inconsistent (we can overcome this by renaming a file in case if writes succeed, it is atomic and cheap in most filesystems). The code for Router with timeout , it sets info_handler to process GET requests to /version endpoint, sets handle_timeout_error as error handler and sets timeout for all requests to 1 second:
