Using the strongest model for every request is not always the best investment

FrugalGPT organized scenarios with two-digit price differences among multiple commercial LLMs and examined cost-reduction strategies such as prompt adaptation, approximation, and cascading. The learned cascade achieved up to 98% cost reduction while matching the best single model on experimented natural language tasks, or exceeded GPT-4 by 4 accuracy points at the same cost.[1]

The key point is not the maximum value but the design principle that not all requests require the same capability; classification, structured extraction, and simple formatting may suffice with small models, whereas ambiguous judgments or multi-constraint adjustments may require high-performance models. Instead of treating business tasks at a single average difficulty, estimate the required quality and failure impact for each input.

Routing units should be based on business groups with identical inputs and failure impacts, not department names. For each group, pass both small and large models through the same samples and compare accuracy with post-review confirmed results. By moving only those groups where the estimated quality difference and its uncertainty fall within a predeclared tolerance and later error detection has been verified to the low-cost default path, you can avoid the error of directly transferring public benchmark savings to your own context.[1]

Hybrid LLM proposed a router that sends requests to either a small or large model based on question difficulty and desired quality. Experiments showed a reduction of up to 40% in large model calls without compromising answer quality.[2] A single-call design is easier to implement in terms of reducing latency and duplicate costs compared to a cascade that sequentially tests multiple models.

On the other hand, a cascade reviews the answer and confidence of the low-cost model and escalates only when necessary. While it offers the advantage of reusing the initial answer, incorrect confidence estimation may halt difficult cases, and repeated calls increase latency. The choice between them depends on the possibility of automatic answer scoring, acceptable latency, retry costs, and the ability to detect failures afterward.

Routers and cascades should be compared under the same quality floor. Place single-routing misassignments, step-escalation additional wait times, duplicate calls, and initial answer reuse on the same ledger. If low-confidence cases are frequent and cascade escalation becomes the norm, this indicates route complexity rather than savings design, necessitating a decision to revert to a single safer model.[2]

RouteLLM learned to route between strong and weak models from human preference data, showing results of reducing costs to less than half without compromising quality on public benchmarks. It also reported generalization to model pairs not included in the training.[3]

Nevertheless, preferences learned may not necessarily reflect your company's accuracy, tone, or legal requirements. If difficulty distributions shift due to new products, peak seasons, input languages, or prompt changes, previous thresholds will become misaligned. Creating router correctness using only another LLM's scoring poses the risk of optimizing for the scorer's preferences, necessitating manually corrected business samples.

Explanations for quality degradation include not only input distribution changes but also scorer preferences and candidate model version changes. Run fixed manually corrected samples and recent real input samples in parallel; if only the former degrades, suspect the model side, and if only the latter degrades, suspect distribution changes. For categories where misrouting cannot be explained by either, stop automatic routing and re-correct.[3]

The three studies show that routing can shift the quality-cost frontier under specific task, candidate, and price conditions, but public results cannot be directly applied to your company's expectations.[1] [2] [3] Internally, pass the same requests through both candidate models and save not only the results of the chosen path but also the results and evaluation versions of the unchosen paths.

Cases where the small model failed and the large model succeeded but were sent to the small model are dangerous misroutes verifiable by comparison with the unchosen path. Cases where both failed are treated as cases unsolvable by the router alone, requiring separate investigation of preprocessing, candidate sets, and evaluation conditions. Cases where both passed but were sent to the large model are considered over-assigned candidates after verifying price and latency conditions. Do not collapse these three categories into a single rate.

Quality-side re-correction targets only business groups where dangerous misroutes have increased, candidate model versions have changed, or ground truth criteria have changed. Price changes do not constitute a deterioration of router quality, so do not automatically change quality labels or thresholds. If multiple causes exist, isolate them one by one while keeping quality comparisons fixed, and attach cause-specific test versions.

When fees or billing units change, the cost curve for the same path changes, but this alone does not change the evidence of answer quality. Separate price versions from quality evaluation versions and apply old and new prices to fixed routing results. When redoing quality tests, report price differences and quality differences in separate columns, align baseline dates, and retain judgments from the old price version.

The economic ledger records price versions, model calls, escalations, duplicate calls, retries, reviews, and wait times per case. Perform recalculation scenarios separately for replacing only prices and for changing the routes themselves. If cases violating the quality floor emerge in the latter, reject the route proposal rather than compensating with savings, and save the rejection reason, scenario name, and premises.

Approvals are determined by comparing total costs in the economic ledger among routes allowed in the quality ledger. Even if the safer model becomes more expensive due to price revisions, do not loosen thresholds without quality justification. Conversely, even if quality improvement is confirmed, do not finalize savings if the price version is outdated. Only the point where both versions and approvers align is considered a valid judgment, including expiration dates, next review conditions, deadlines, and judgment numbers.

• QUALITY: Fix the results of both candidates and ground truth criteria independently of price
• MISROUTE: Count dangerous misroutes, both-failed cases, and over-assignments separately
• PRICE: Recalculate billing versions, calls, escalations, retries, and reviews
• APPROVE: Approve quality and price versions separately and specify valid combinations

In comparative experiments, pass the same inputs through small, large, and routed paths. Record accuracy, major errors, confirmation time, retries, latency, input/output tokens, and router costs. Separate cases misrouted to small models from those over-assigned to large models, using safer thresholds in areas where the former's loss is significant.

In operations, send low-confidence, out-of-scope language, high-impact category, and unverifiable answers to higher models or humans. If model prices or versions change, recalculate past cost curves. Success conditions include not only savings compared to the large model but also maintaining the quality floor and retaining value when including total processing time and correction costs. Since input composition changes by department or season, monitor misrouting rates by segment in addition to overall averages.

Divide operational responsibility among owners of business quality, cost/latency, and router changes. If the quality floor set by the quality owner is violated, revert to the safer path even if cost savings remain. Recovery requires re-measuring the quality floor, confirmation time, and retries with samples including misrouted real cases, and procedures must ensure thresholds are not loosened solely due to price revisions.

• Quality Constraints: Fix minimum acceptable accuracy and verifiability, and a maximum for major errors per business group first
• Cost Scope: Include router, retry, review, and wait times in addition to model fees
• Escalation Rules: Send low-confidence, out-of-scope inputs, high-impact, and unverifiable cases to large models or humans
• Rollback: Revert categories violating the quality floor to the safer single path without waiting for cause determination