Why list prices mislead: OpenRouter Head of Insights on how to choose the right model

Show notes

Companies now use an average of 12 models per month, yet the biggest hidden variable isn't which model you pick—it's the inference provider serving it and whether your cache hit rate stays above 85%. Peter Walker from OpenRouter explains how the same open-weight model can have wildly different cost and performance depending on the provider's caching policy, latency tail, and whether a stealth provider is reselling your prompts. The conversation covers why agentic workloads broke cache economics in February 2026, how ZDR policies vary even among "zero data retention" claims, and why the frontier labs are now competing across price and intelligence spectrums simultaneously. For engineers choosing production models, this is a practical map of the dimensions that actually move the bill.

Key takeaways

  • The inference provider layer—not just the model layer—now has 200+ competitors, and provider choice directly determines effective cost, latency, and data sovereignty.
  • Cache hit rate is the dominant cost lever for agentic workloads: a cache miss can cost 50-100x more per token, and median agentic cache rates have climbed from ~50% to ~85% in 2026.
  • Most organisations use a 'hub and spoke' model architecture with one 'daily driver' taking the majority of tokens and spend, plus specialised models for departments like customer support or sales.
  • Zero data retention claims vary in trustworthiness by jurisdiction and provider size; some stealth models on OpenRouter are free specifically because they are not ZDR.
  • The gap between frontier and open-weight models is widening in raw capability but narrowing in perceived value, as daily tasks saturate below frontier intelligence.
  • Reasoning-only models and jagged intelligence are driving demand for fast decision models like Jev and OpenAI's decision model for latency-critical paths.

Chapters

  • 00:00 The New Complexity
  • 03:29 Provider Layer Competition
  • 05:31 Choosing a Provider
  • 08:04 Data Retention Realities
  • 12:01 Model Portfolios
  • 14:45 Latency and Reasoning
  • 17:51 Cache Economics
  • 22:57 Cost-Performance Sweet Spot
  • 25:46 What Comes Next
  • 28:35 Open Weight vs Frontier
  • 31:24 Specialisation and Product
  • 33:36 Where to Follow Peter

Links and tools mentioned

About Peter Walker

Peter Walker is Head of Insights at OpenRouter, where he analyses global model usage patterns across the inference layer.

This episode is presented by SparkLaunch and Dam Secure.

Credits

New comment

Your name or nickname, will be shown publicly
At least 10 characters long
By submitting your comment you agree that the content of the field "Name or nickname" will be stored and shown publicly next to your comment. Using your real name is optional.