Search code, repositories, users, issues, pull requests...
DailyBench is an automated, daily evaluation suite to track model performance over time, monitor for regression during peak load periods, and detect quality changes across flagship LLM APIs.
DailyBench is a lightweight tool evaluation suite built on a fork of HELMLite that runs standardized benchmarks against LLM APIs and tracks performance over time. This helps detect when providers make undisclosed changes to their models. DailyBench runs at a random time within every 6-hour window, 4 times a day. The results are aggregated and published to the public dashboard.
We attempt to make model responses as deterministic as possible by forking HELMLite and setting all recommended parameters for each provider (seed, temperature, top_p, etc.). However, we cannot guarantee that the model responses will be exactly the same across runs and accept that there will be some variance; instead, we aim to detect if regressions or changes in model quality are happening, especially if they are happening in clear, repeated patterns.
Recent pseudo-evidence suggests that some LLM providers quantize their models during peak US hours and make other undisclosed changes. Users deserve to know they're getting consistent quality from the APIs they pay for.
