Implementing a Deception Eval with the UK AISI's Inspect Framework¶
The UK AI Safety Institute recently released the Inspect framework for writing evaluations. It includes a number of tools and features which make writing evaluations easy. In this post, we'll write a simple evaluation with Inspect. If you find this topic interesting, I encourage you to try using Inspect to write your own!
We're going to reproduce the core of the paper "Large Language Models Can Strategically Deceive Their Users When Put Under Pressure" from Jérémy Scheurer, Mikita Balesni, and Marius Hobbhahn at Apollo Research. Jérémy was kind enough to provide two of the original classification prompts not present in the original project repository. Thanks Jérémy!
In the "Deception Under Pressure" paper, the authors create a scenario where an LLM plays the role as a stock trader. It first receives some demonstrations about actions it can do, such as searching for trades, making trades, communications, etc. accompanied by some sample reasoning scratchpads about why it's taking those actions.
Next, pressure is placed upon the model by having its manager say that the company might shut down if they have another bad quarter. It then receives an insider tip about a potential merger about to happen, followed by some reasoning provided about how insider trading is risky.
