A System for Testing New AI Models

Nathaniel Whittemore and Nufar Gaspar walk through a repeatable system for benchmarking AI models against your own work tasks, rather than trusting generic leaderboards. They show how to pick use cases, run blind comparisons, and weigh quality against cost, latency, and policy constraints using tools like OpenRouter and Gemini.

Key points

• Choose 4-6 recurring tasks that reflect real work, include one wish-list task, and define what “bad” looks like.
• Run each task in a fresh chat, hide model names, and score outputs blind on both a 1-5 scale and a short qualitative note.
• Use your own judgment first; AI judges can help, but may be biased or poorly aligned with your preferences.
• Compare not just quality, but also cost, latency, and whether company policy or tool constraints block switching.
• Repeat critical tests several times because the same model can produce different results on the same prompt.
• Open-source models can be “good enough” for many table-stakes tasks; frontier models still matter for harder workflows.