🪴 Dmitrii's personal blog
See the result here. You can see 2804 predictions HN users made, evaluated by LLM with categories, filters and search. Try typing “Linux” in a search bar! Source code is on GitHub
I stumbled upon a 2010 thread of predictions for the upcoming decade and had fun reading through it. I thought maybe I can find people that were good at predicting the future, and check what they think about the next decade. I decided to use LLMs to evaluate these predictions. LLMs, with their extensive knowledge of recent history, should be capable of assessing the accuracy of most predictions.
Initially, I aimed to analyze all predictions made by Hacker News users. I utilized a ClickHouse dataset, employing regular expressions and similar techniques for data filtering. Claude helped me write SQL queries, but the results still contained significant noise. To streamline the process, I narrowed the focus to 12 specific Ask HN prediction threads, with the idea to extend it to all HN comments later. Lesson: ClickHouse is very nice and can be a go-to (after SQLite).
I used the LMSYS Leaderboard to find an open-source model that fits in my Mac M1 Max for local execution. The goal was to obtain structured JSON output. While APIs like Google’s Gemini Flash were cheap, they required multiple queries to achieve the desired output format, because they didn’t force JSON output but only provided suggestions. Ultimately, I selected the Nous Research Hermes-2-Theta-Llama-3-70B-GGUF model. The GGUF weights were readily available on HuggingFace, allowing me to configure my JSON schema and run llama.cpp with an additional grammar output restriction. This approach worked, but a processing time of approximately 4 minutes per comment, so I just let it run over night. Lesson: local LLMs are pretty good already, and pretty simple in a setup. Hope Ollama will get a proper Grammar support too though.
