Who needs git when you have 1M context windows?

Lately I’ve heard a lot of stories of AI accidentally deleting entire codebases or wiping production databases. But in my case it was the other way around. I removed some working code and the LLM helped me to recover it.

I joined RevenueCat a couple of months ago to work on LTV predictions. My first few projects were straightforward: fix small things, ship some low-hanging fruit. After getting familiar with the code, I decided to extend the LTV model and add some cool machine learning. So I dove in. Spent a couple of days wrangling data, cleaning it, understanding what was going on, and the usual standard pre-training stuff. I was in “research mode”, so all my code lived in notebooks and ugly scripts. But after enough trial and error, I managed to improve the main metric by 5%. I was hyped. Told the team. Felt great.

Then came the cleanup. I refactored all the sketchy code into a clean Python package, added tests, formatted everything nicely, added type hints, and got it ready for production. Just before opening the PR, I ran the pipeline again to double-check everything… and the results had dropped by 2%.

Leave a Comment