Engineering
Frontier models in the enterprise: how to choose, evaluate and switch without rebuilding
Enterprises should not pick a frontier model the way they pick a database vendor. Models change every few months. What works instead is selection driven by evaluation on your own cases, an architecture that isolates the model behind tested interfaces, and an operating rhythm in which upgrades are regression-tested and rolled out like any other change.
- Author
- Luka Kokot, Founder and Chief Executive, Applicat AI
- Published
- Updated
- Reading time
- 7 min read
The problem with choosing a model
Every few months an AI lab releases a model that is better, cheaper or faster than the one before, and often all three. Public benchmarks measure general ability. They say nothing about your claims letters or your product catalogue. An enterprise that standardises on one model in January and hard-wires it into ten systems will spend the following year either falling behind or rebuilding.
Principle one: evaluate on your own cases
Build an evaluation set from real cases: documents, requests, questions and expected outcomes drawn from your operation, with the awkward ones included. Score candidate models on accuracy, groundedness, safety behaviour, latency and cost at expected volumes. The result is a ranking that means something for your business, and it is reusable every time a new model arrives.
Principle two: isolate the model
Treat the model as a component behind an interface. Prompts, tool definitions, retrieval and output schemas live in your system, versioned and tested. The model is called through a gateway that can route to Anthropic, OpenAI, Google or an open-weight model running in your tenancy. Swapping models then becomes a configuration change, validated by the evaluation suite before it goes anywhere near production.
Principle three: run upgrades as operations
- A new model is released.
- It is run against the evaluation suite in a staging environment.
- Results are compared with the current model on accuracy, safety, latency and cost.
- If it wins, it is rolled out to a share of traffic, monitored, then promoted.
- The change is recorded in the system's governance log.
This is the same discipline that infrastructure teams apply to patches. It turns the fast-moving frontier from a risk into a source of routine improvement: the same system gets better every quarter without a rebuild.
Where open-weight models fit
Open-weight models run inside your own environment, which matters when data cannot leave a jurisdiction, when latency must be very low, or when volumes make per-token pricing uncompetitive. They are usually behind the frontier on general capability but can match or beat it on narrow tasks after fine-tuning. A mature enterprise portfolio keeps at least one open-weight option evaluated and ready.
Questions to ask any AI provider
- Which models did you evaluate for this use case, on what cases, and can we see the scores?
- How long would it take to switch models, and what would break?
- What is your process when a new model is released?
- Where does our data go for each model you propose?
- What does inference cost at our expected volumes, and how is that monitored?