Routing between models saves money, and the exit plan is what you are really buying

In July 2023 Lingjiao Chen, Matei Zaharia and James Zou posted a paper with a title that was also a warning: "How is ChatGPT's behavior changing over time?" They ran the March and June versions of GPT-4 through a handful of tasks. On one of them, deciding whether a number is prime, accuracy fell from 84% to 51%. The paper reported other shifts too, including more formatting errors in generated code and a lower willingness to answer opinion questions. The authors concluded that the same hosted model can change substantially in a few months, so applications built on it need continuous monitoring.

Within days Arvind Narayanan and Sayash Kapoor published a reply. The prime-number test had used 500 numbers, all of them prime, so the right answer to every question was "yes." A model that always says "prime" scores perfectly, and the March model mostly did say "prime" while the June model mostly said "composite." When the critics added 500 composite numbers, all the models did poorly. Their argument was that the paper had measured a change in behavior, shaped by fine-tuning, and not a loss of capability, which comes from pre-training and moves slowly.

Both sides are right about something that matters to anyone shipping a product. The capability may be intact, and your application can still break, because the thing an API serves is a moving target. Behavior changes under a stable model name, tokenizers differ across providers, retired models disappear on a published schedule, and prices move. A design that assumes one provider forever is betting on none of these happening. The alternative is a design with a small internal interface in front of the model, a set of evaluation cases that define acceptable behavior, and at least one tested alternative. Routing, in the narrow sense of sending different requests to different models, comes along almost as a by-product, and it can also pay for itself.

What the router papers show

The cost case comes from two lines of research. FrugalGPT, by Chen, Zaharia and Zou (2023), proposed cascades: try a cheap model first, check whether the answer is reliable using a learned scorer, and escalate to a more expensive model only when it is not. They reported matching GPT-4's performance on the tasks they tested with up to 98% lower cost, or improving accuracy over GPT-4 by 4% at the same cost. RouteLLM, a 2024 paper from researchers around the LMSYS group (Isaac Ong, Ion Stoica and Joseph Gonzalez among the authors), trained routers on human preference data to choose between a strong and a weak model for each query. In the project's write-up, routing cut cost against GPT-4 alone by over 85% on MT-Bench, 45% on MMLU and 35% on GSM8K while retaining 95% of GPT-4's quality.

The spread in those numbers is the interesting part. The saving is bounded by how much of the traffic is easy. My reading is that the benchmarks differ in how many questions need the larger model: open-ended MT-Bench prompts leave a weaker model room to score well, while MMLU and GSM8K have right and wrong answers that it misses more often. Neither paper is a promise about your workload, and both measured model pairs from 2023 and 2024 whose price and quality gaps have moved since. What transfers is the shape of the argument.

The arithmetic is worth doing for your own case. Suppose a small model costs 5% of a large one per request, and 70% of your requests are ones it handles correctly. In a cascade, every request tries the small model first, which costs 0.05 per request, and the 30% that escalate pay 1.00 on top. The expected cost is 0.05 + 0.3 = 0.35 of the all-large baseline, a 65% saving. A router that picks a model up front, without the wasted first attempt, would cost 0.7 × 0.05 + 0.3 × 1.00 = 0.335. Neither can save more than the easy share, which here is 70%, and both suffer when the escalation signal is wrong. A hard query sent to the small model yields a bad answer. An easy query sent to the large model wastes money, which is the smaller error. Cascades also add latency on the escalated requests, since the user waits for the small model's attempt before the large model starts. If the small model answers in 1.5 seconds and the large one in 6, the 70% of requests that stay small finish four times faster, and the 30% that escalate take 7.5 seconds instead of 6. The average falls to 3.3 seconds while the 95th percentile gets worse, and which of those numbers matters depends on the product.

The router is an ML problem with your data

The papers train their routers on public preference data. A production router trained that way knows what chat users in general preferred and nothing about whether your invoices were extracted correctly. The labels that tell you which requests a small model can handle are your own evaluation cases, scored per model: for each case, did the small model pass? With a few hundred of those, a simple classifier or even a rule (short input, known template, a confident extraction) can split traffic usefully, and you can measure how often it errs.

Without the cases, routing is guesswork with a dashboard. A team that reaches for routing because the invoice is large, and has no such labels, may watch quality complaints rise and conclude that routing does not work, when what failed was the absence of a measurement. The same labeled set that gates releases is what makes a router safe to deploy, and it will tell you early if the saving comes with a quality loss you cannot accept.

Prompts do not port

Switching providers looks easy in a diagram: change the endpoint, change the key. In practice each model family has its own habits. Instructions that one model follows literally another reads loosely. System prompts, tool-calling schemas, structured-output modes and stop behavior differ. Tokenizers differ, so the same document costs a different number of tokens on each vendor, which makes price per million tokens an unreliable basis for comparison. The only comparison that holds is cost per successfully completed task, measured on your cases.

The sound approach is to treat prompts as artifacts that belong to a model, versioned and tested together. A thin internal interface takes messages, tool definitions and a desired output schema, and adapters translate for each provider. Features that exist only at one vendor can be used, but in a place where you know the cost: behind the adapter, with a documented fallback behavior. Prompt caching, batch endpoints and fine-tuning are all vendor-specific, and all worth using. Each one adds to the migration bill, so the bill should be written down.

Even careful abstraction leaks. A fallback model will do the task a little differently, and the suite shows by how much. A fallback that scores six points below the primary on your cases is still a fallback, provided someone has decided that six points is survivable during an outage and has written the decision down.

Pin the version and watch for drift

The 2023 episode has a practical lesson about how to consume a hosted model. Where a provider offers dated snapshots, pin to one, since an alias that points to "the latest" lets the vendor change your application's behavior on its own schedule. Then run the evaluation suite against the live endpoint every night, with the pass rate and its interval plotted over time. A snapshot that was supposed to be frozen and drifts anyway shows up as a step in the line. A planned upgrade shows up as a step you chose.

The nightly run costs little. Two hundred cases at a few thousand tokens each is a few hundred thousand tokens a day, which at mid-tier list prices is a dollar or two a day. What it buys is the ability to say, when a customer reports odd answers, whether the model moved on Thursday or the problem is somewhere else.

Concentration risk is already an audit question

Resilience is a topic that regulators in some sectors have put in writing. The EU's Digital Operational Resilience Act, which covers financial entities, requires in Article 28(8) that firms keep exit strategies for ICT services that support critical or important functions. The strategies must be documented and tested, they must account for failures at the provider, and they must allow an exit without disrupting business or reducing the quality of service to clients. The regulation is not about AI, but a bank that uses a hosted model inside a critical process is using an ICT service, and an auditor can ask to see the plan and the test.

A plan that cannot be tested is a sentence in a document. For a model dependency, the test is concrete: point the application at the fallback in a staging environment, run the evaluation suite, and record the quality gap, the cost difference, the added latency and the manual steps it took. Any firm can run that exercise, regulated or not. One option reduces risk without touching the prompts: reach the same model through a different channel. Several frontier models are sold directly by their developer and also through the large cloud platforms. Anthropic's pricing page, for instance, notes that some retired models remain available on Amazon Bedrock and Google Cloud. A team that has its primary traffic on the developer's API and a standby route on a cloud marketplace has hedged against an outage on one side while keeping the behavior it tested. It has not hedged against the model itself being retired everywhere, or against a quality problem in the model, which is what a second model family is for. The two hedges address different failures, and a plan should say which one covers which.

The risks it covers are ordinary ones. A provider can have an outage, retire a model on a schedule (Anthropic's pricing documentation lists several models as retired as of October 2026), change a price, change its terms, or alter behavior silently, as Chen and colleagues observed. The question for each is how long it takes to be running on something else.

The case for staying with one

There is a respectable opposing view: pick a single provider, learn it deeply, and spend the engineering time on the product. Depth has real returns. A team that commits to one vendor can design around its caching rules, its batch discounts, its fine-tuning options and its tool-use conventions, and avoids the lowest-common-denominator trap of writing to the features all providers share. Maintaining two sets of prompts, two sets of adapters and two sets of evaluation baselines is real work that does not ship features.

I agree with most of that, and I do not think it argues for a single vendor with no exit. It argues for proportion. One primary provider carries the traffic. One tested fallback exists, exercised on a schedule and kept to the minimum needed to survive an outage or a deprecation. A router appears only when a measurement shows it will pay. The break-even is simple to state. If your monthly model spend is $40,000 and a router would plausibly save half of it on the easy traffic, that is $20,000 a month. If building and maintaining the router, its evaluation cases and its monitoring takes the equivalent of a third of an engineer at a fully loaded $15,000 a month, the saving is about four times the cost. At $4,000 a month in spend, the same arithmetic points the other way. Those figures are placeholders, and the method is to put your own into the formula before building anything. There is also a benefit the formula leaves out. A team that can show, with its own cases, that another provider's model passes nearly as many of them at a fraction of the price negotiates from a different position than a team that merely prefers its current vendor.

Run the drill

A written strategy for leaving a provider is not an exit plan until someone has rehearsed it. The rehearsal takes an afternoon. Route a copy of staging traffic to the fallback, run the full evaluation suite against it, and write down what broke: the prompts that need rewriting, the tools whose schemas differ, the output formats that no longer parse, the latency percentile that moved. Do it once a quarter, since the models on both sides change in that time.

The first run will probably go badly, which is the point of doing it on a Tuesday afternoon and not during an outage. Whatever the drill breaks becomes the migration backlog, and the length of that backlog is an honest measure of your dependence.

Sources

1. Chen, L., Zaharia, M., Zou, J. (2023). How is ChatGPT's behavior changing over time? arXiv:2307.09009.
2. Narayanan, A., Kapoor, S. (2023). Is GPT-4 getting worse over time? AI Snake Oil / Normal Tech. normaltech.ai/p/is-gpt-4-getting-worse-over-time.
3. Chen, L., Zaharia, M., Zou, J. (2023). FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv:2305.05176.
4. Ong, I., Almahairi, A., Wu, V., et al. (2024). RouteLLM: Learning to Route LLMs with Preference Data. arXiv:2406.18665; and LMSYS (1 July 2024), RouteLLM blog post, lmsys.org/blog/2024-07-01-routellm.
5. Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA), Article 28(8), Official Journal L 333, 27.12.2022.
6. Anthropic. Pricing, Claude developer documentation (retired-model listings as of October 2026). platform.claude.com/docs/en/about-claude/pricing.