Solving a ResourceT-related space leak in production
Working with Haskell is fun, until you have space leaks — especially when they occur in AWS Lambda Functions. In this blog post, we focus on how we plugged a particularly nasty space leak. In a future post, we will introduce a profiling tool for Haskell applications running on AWS Lambda.
We observed one of our Lambda Functions dying of OOM (Out Of Memory) failures, seemingly at random. The function had 1GB of memory allocated, and there seemed to be a big disconnect between the work it was doing and its memory utilization.
In a docker container with memory constraints, the free command shows the memory size of the host system instead of the container’s memory constraints.
For AWS Lambda Functions, we suspected that the situation might be similar. If there is anything special that the runtime system of a programming language needs to do to detect the correct memory limit, it might be done in those official Lambda Function runtimes — Golang and Python, for example — but not in Haskell.
If GHC RTS (Runtime System) is reading incorrect memory constraints, it may decide to defer garbage collection because it thinks there is still available memory.
