Claude’s Quality is Dropping – Here’s Why
Claude's performance has taken a noticeable hit lately. I've been using it daily for several months, and the degradation is pretty clear. Let me break down what's happening and why I think it's occurring.
That last point is particularly concerning. Claude was known for handling long documents better than any other AI - it was one of its standout features. Now it seems to ignore large chunks of input text or fails to properly synthesize information from longer documents. What used to be Claude's specialty has become notably worse.
Anthropic's CEO Dario Amodei recently addressed this on Lex Fridman's podcast. He insisted they never switch to lower-quality models during high load. But what's interesting is what he didn't address - there are many other ways to reduce compute load without technically switching models.
This matches what we're seeing - same model, worse outputs. It's particularly noticeable with long inputs, where the model seems to struggle maintaining context and attention across the full text. It's a way to handle more users without technically lying about not switching models.
%20tasks%2C%20the%20demand%20for%20these%20state-of-the-art%20models%20increases%20commensurately.%20However%2C%20their%20deployment%20is%20costly%20both%20in%20terms%20of%20processing%20power%20and%20time.%20In%20this%20paper%2C%20we%20investigate%20the%20feasibility%20of%20creating%20smaller%2C%20highly-performing%20specialized%20algorithms%20by%20way%20of%20fine-tuning.%20To%20do%20this%2C%20we%20first%20present%20a%20large%20pre-trained%20model%20with%2020%20unique%20scenarios%20that%20combine%20different%20social%20contexts%20with%20games%20of%20varying%20social%20dilemmas%2C%20record%20its%20answers%2C%20and%20use%20them%20for%20Q%26A%20fine-tuning%20on%20a%20smaller%20model%20of%20the%20same%20family.%20Our%20focus%20is%20on%20in-context%20game-theoretic%20decision-making%2C%20the%20same%20domain%20within%20which%20human%20interaction%20occurs%20and%20that%20requires%20both%20a%20theory%20of%20mind%20(or%20a%20semblance%20thereof)%20and%20an%20understanding%20of%20social%20dynamics.%20The%20smaller%20model%20is%20therefore%20trained%20not%20just%20on%20the%20answers%20provided%2C%20but%20also%20on%20the%20motivations%20provided%20by%20the%20larger%20model%2C%20which%20should%20contain%20advice%20and%20guidelines%20to%20navigate%20both%20strategic%20dilemmas%20and%20social%20cues.%20We%20find%20that%20the%20fine-tuned%20smaller%20language%20model%20consistently%20bridged%20the%20gap%20in%20performance%20between%20the%20smaller%20pre-trained%20version%20of%20the%20model%20and%20its%20larger%20relative%20and%20that%20its%20improvements%20extended%20in%20areas%20and%20contexts%20beyond%20the%20ones%20provided%20in%20the%20training%20examples%2C%20including%20on%20out-of-sample%20scenarios%20that%20include%20completely%20different%20game%20structures.%20On%20average%20for%20all%20games%2C%20through%20fine-tuning%2C%20the%20smaller%20model%20showed%20a%2046%25%20improvement%20measured%20as%20alignment%20towards%20the%20behavior%20of%20the%20larger%20model%2C%20with%20100%25%20representing%20indistinguishable%20behavior.%20When%20presented%20with%20out-of-sample%20social%20contexts%20and%20games%2C%20the%20fine-tuned%20model%20still%20displays%20remarkable%20levels%20of%20alignment%2C%20reaching%20an%20improvement%20of%2018%25%20and%2028%25%20respectively.)
