The hunt for error -22
At Tweede golf we do lots of engineering work, and a while ago we worked for a client using the nRF9160 microcontroller. This chip has an LTE modem with which you can reach the internet while still not using a lot of power. It's pretty neat!
I built the majority of this client's new firmware using Rust and Embassy and aside from some low-power/sleep woes, it went well. The client took the maintenance and further development in-house and my job was done. Or so I thought.
Half a year later, we were contacted by the client because they were seeing strange things, the sort of things you don't want to see in your in-production firmware.
See, when they compiled the firmware, everything functioned fine most of the time, but sometimes a small change in any place would make it so sending data to the server always or often returned Error::NrfError(-22). For example, the problem could trigger if you read some config on line X, but not when you read the config on line X+1. It was incredibly sensitive. Changing the compiler version and optimization settings could also affect the problem.
The error was hard to replicate. It was so random that you pretty much had to stumble over it to find it. So luckily the client had made a branch out of one exact version of the codebase that exhibited the behavior.
