Fuzzing with memfd_create(2) and fmemopen(3)
If you’ve done a fair amount of fuzzing, you’ve likely come across targets which are deeply dependent either on file descriptors or FILEs. Rewriting the whole codebase to accept a fuzzing harness from a buffer in shared memory is awfully cumbersome, so you’re stuck with a slower fuzzing campaign than you’d hope. But then you hear about the memfd_create(2) syscall and the fmemopen(3) function.
Basically, this syscall and this function can be used to create files which only exist in memory: and they can have arbitrary data from memory written to them. This post investigates the comparison of a fuzzing campaign’s speed using them instead of traditional files on a tmpfs/ram-disk.
To create these benchmarks, we fuzz libxml2 with aflplusplus. We compile with afl-clang-lto and no special hardening flags or optimization flags
As it turns out, you cannot call fileno(3) on a FILE which has been returned using fmemopen(3). From the manpage: There is no file descriptor associated with the file stream returned by this function (i.e., fileno(3) will return an error if called on the returned stream). Therefore, we proceed slightly differently.
