Every few months, a new model tops the leaderboard. Teams scramble to integrate it. Executives forward the press release. And the engineering team is left explaining why switching models in production is not like updating an app on your phone.

At GRAL, model selection is treated as an infrastructure decision — not a hype cycle. The model you choose affects your latency budget, your cost structure, your compliance posture, and your ability to iterate for years. Getting it wrong is expensive. Getting it right requires ignoring most of what the industry tells you to care about.

Benchmarks Measure the Wrong Things

Academic benchmarks test models on tasks that rarely resemble enterprise workloads. A model that scores 92% on a general reasoning benchmark might score 60% on your specific document classification task — because your documents have domain jargon, inconsistent formatting, and edge cases that no benchmark covers.

The worse problem is that benchmarks measure peak performance under ideal conditions. Enterprise systems need consistent performance under variable conditions. A model that scores 95% on average but drops to 40% on 5% of inputs is more dangerous than a model that scores 85% consistently. That 5% failure rate will hit your most complex, highest-value transactions — the ones where the model's output matters most.

GRAL evaluates models on production-representative data before any selection decision. Not the vendor's demo dataset. Not a curated test set. Real data, with all its messiness. The model that performs best on clean inputs is rarely the model that performs best on what your systems actually produce.

The Real Selection Criteria

GRAL's model selection framework starts with constraints, not capabilities. Before evaluating any model's accuracy, we establish the boundaries it must operate within:

Latency budget. If your system needs sub-200ms responses — common in real-time decision systems, voice AI, and transaction processing — that constraint eliminates most large language models immediately. No amount of accuracy justifies a response time that makes the system unusable. GRAL defines latency requirements at the P99 level, not the average, because your users experience the tail, not the mean.

Cost per inference at scale. A model that costs $0.01 per inference looks cheap in a pilot. At 10 million inferences per month, it is a $100,000 monthly line item. GRAL models cost projections at production scale from day one, including the cost of the infrastructure required to serve the model — GPU instances, memory, networking, redundancy.

Data residency. In regulated industries, data cannot leave certain jurisdictions. This rules out most cloud-hosted API models unless the provider offers region-locked deployment. GRAL builds for on-premise and edge deployment precisely because data residency is non-negotiable for many enterprise use cases.

Fine-tuning flexibility. A model you cannot fine-tune is a model you cannot adapt as your business changes. Proprietary API models offer limited or no fine-tuning. Open-source models offer full control. This trade-off matters more than most teams realize at selection time — it determines whether your AI system improves with your data or stays frozen at the vendor's last update.

Vendor lock-in risk. If your entire system depends on a single model provider's API, you are one pricing change or deprecation notice away from a crisis. GRAL designs systems with model abstraction layers — the ability to swap models without rewriting the application. This is not a theoretical concern. Model providers have deprecated APIs, changed pricing by 10x, and altered rate limits with weeks of notice.

Open-Source vs. Proprietary

This is not an ideological question. It is an engineering decision with clear trade-offs.

Proprietary models (accessed via API) are the right choice when:

  • You need state-of-the-art general reasoning and your task does not require domain-specific fine-tuning
  • Your inference volume is low enough that per-call pricing is cheaper than running your own infrastructure
  • Your data is not subject to residency or confidentiality constraints that prohibit third-party processing
  • Time to deployment matters more than long-term cost optimization

Open-source models (self-hosted) are the right choice when:

  • You need to fine-tune on proprietary data — which is most enterprise use cases
  • Your inference volume makes per-call API pricing untenable
  • Data must stay within your infrastructure
  • You need deterministic behavior and full control over model versioning
  • You are building a system that must run for years without depending on a vendor's roadmap

Most GRAL systems use open-source models as the foundation, with proprietary models as a fallback or for specific subtasks where general reasoning quality matters more than domain specificity. This is a pragmatic architecture, not an ideological one.

The Model Zoo Problem

Enterprises that adopt AI without a clear model strategy end up with a model zoo: a different model for every use case, each requiring its own infrastructure, monitoring, and maintenance. One team uses GPT-4 via API. Another fine-tuned a Llama variant. A third is running a custom BERT model from 2023. Nobody can tell you the total cost of AI inference across the organization.

This happens because teams optimize locally. Each team picks the best model for their specific task without considering the organizational cost of model proliferation. The result is duplicated infrastructure, inconsistent quality standards, and a maintenance burden that grows linearly with every new model.

GRAL avoids this by standardizing on a small number of model families that cover the organization's needs. A typical GRAL architecture uses two or three model tiers:

  • A lightweight model for high-volume, low-complexity tasks — classification, routing, extraction. Fast, cheap, runs everywhere.
  • A mid-tier model for tasks requiring domain understanding — summarization, analysis, structured reasoning. Fine-tuned on domain data.
  • A large model (often proprietary API) reserved for complex, low-volume tasks where quality justifies the cost — nuanced generation, multi-step reasoning, ambiguous inputs.

Every task maps to a tier. New use cases fit into the existing architecture rather than spawning new infrastructure. The discipline here is saying no to the shiny new model when your existing tier handles the task adequately.

Total Cost of Ownership Over 3-5 Years

The initial cost of deploying a model is a fraction of the total cost of running it. GRAL's TCO framework accounts for:

Infrastructure costs. GPU instances, storage, networking, redundancy. These scale with inference volume and model size. A model that requires 80GB of VRAM costs fundamentally more to serve than one that fits in 16GB — even if both produce similar quality outputs for your task.

Maintenance costs. Model monitoring, retraining pipelines, data labeling, performance regression testing. Every model in production needs ongoing attention. GRAL budgets 20-30% of the initial deployment cost annually for maintenance — and that estimate has proven conservative.

Opportunity costs. Engineering time spent maintaining a model that should have been replaced. Teams that chose the wrong model early often spend more time working around its limitations than they would have spent migrating to a better fit.

Migration costs. If you need to switch models — because the vendor changed pricing, because your requirements evolved, because a better option emerged — the cost depends entirely on how tightly coupled your application is to the specific model. Systems designed with model abstraction can swap models in days. Systems built directly on a model's API can take months.

GRAL has seen organizations where the annual cost of running a poorly chosen model exceeds the cost of the entire AI project's initial build. The selection decision echoes for years.

The Decision Framework

GRAL's model selection process follows a specific sequence:

Step 1: Define constraints. Latency, cost ceiling, data residency, compliance requirements. These are non-negotiable filters. Any model that fails a constraint is eliminated, regardless of its accuracy.

Step 2: Evaluate on production data. Not benchmarks. Not demo datasets. Actual data from the target system, including edge cases and failure modes. GRAL runs blind evaluations — the team evaluating model outputs does not know which model produced which output.

Step 3: Test operational characteristics. How does the model behave under load? What happens when inputs are malformed? How does latency change at the P99 level as throughput increases? What is the cold start time? GRAL tests the operational envelope, not just the accuracy envelope.

Step 4: Model the cost at scale. Project inference volumes for 12 months. Calculate infrastructure costs. Include monitoring, maintenance, and retraining. Compare the all-in cost across candidates.

Step 5: Assess the exit strategy. If this model needs to be replaced in 18 months, what does the migration look like? How coupled is the application to model-specific behavior? GRAL designs for replaceability from the start.

This process takes longer than picking the top model from a leaderboard. It produces systems that still work — and still make economic sense — years later.

What This Means in Practice

GRAL's model selection framework often produces counterintuitive results. The best model on paper is rarely the best model for the system. A smaller, faster, cheaper model that meets all constraints and can be fine-tuned on domain data will outperform a larger, more expensive model that was not designed for your specific workload.

The discipline is resisting the pull of capability in favor of fit. Enterprise AI is not about having the most powerful model. It is about having the right model — running reliably, at acceptable cost, within your constraints, for years.

That is a less exciting pitch than "we use the latest frontier model." It is also the approach that keeps systems in production instead of in pilot purgatory.