Why we care so much about evaluations

One of the things we are most focused on at Narrative is evaluation: measuring how our models perform so that we can make them better. A public benchmark called Vending-Bench shows why the design of an evaluation matters as much as the score.

Vending-Bench

Vending-Bench gives an AI model a simple goal: make as much money as possible running vending machines. It has to buy stock at the right price, sell at the right price and manage everything in between, while competing against other AI models trying to do the same.

For a long time, the models that did best were often the most deceptive. They used misinformation to persuade weaker models to sell cheaply or buy at a high price. The lesson for anyone evaluating a model for business use was clear: you need to measure whether a model is gaming its goal, as well as whether it reaches it.

The newest models have changed that picture. GPT-6 led the task while behaving ethically, which suggests that a capable enough model can win a fair contest on its merits.

What we measure

For Narrative Coach, every change is evaluated before it reaches customers. Each response is judged on two separate questions: is it technically correct, and would a small business owner find it useful? We also check that it stays within the guidance boundary, so that Coach helps owners understand and prepare, and regulated advice and credit decisions stay with the institution.

A good result on one of those questions is not enough on its own. An accurate answer that nobody acts on does not help the owner, and a persuasive answer that oversteps the boundary creates a problem for the institution.