akashyap.ai

Building LLM-Agnostic SaaS

There are a lot of capable LLMs: OpenAI, Anthropic, Gemini, Kimi, Meta, and many others. Most of them can generate good outputs and call tools reliably.

But they don't behave the same way.

A prompt that works well with one model may fail with another. One model might be great at tool calling but bad at structured output. Another might follow complex instructions well but struggle with a specific type of reasoning.

So, how do you build a SaaS that can move between models without breaking the task it was built to perform?

The goal isn't to make every model behave identically. That's probably impossible. The goal is to get different models to meet the same product requirements.

Here's how I'd do it.

1. Start with a strong model

Start with a frontier model such as GPT-5.6 Sol or Opus. Get the complete workflow working reliably before worrying about alternatives.

Build the prompts, tools, schemas, and application logic. Test the product manually and make sure it consistently produces what the customer needs.

At this point, you're not optimizing for cost or portability. You're establishing that the workflow can work.

2. Record successful workflows

Have a human use the product while logging the complete workflow:

The human should then select the workflows that correctly represent what the product needs to do. They should also define the conditions and expected results.

This is important. An LLM can help write the evals, but humans still need to define what success means.

3. Use a coding agent to write two levels of evals

Give the selected workflows, expected results, prompts, tools, and application code to a strong coding agent such as Codex, Claude Code, or Cursor.

Ask it to build two levels of evals.

Component evals

These test individual parts of the workflow:

Only test behaviour that the product actually depends on. If two independent tools can be called in any order, don't fail a model just because it uses a different order.

End-to-end evals

These test whether the complete workflow produced what the customer needed.

Two models may take different routes and still produce an equally good result. Component evals tell you where something broke. End-to-end evals tell you whether the product still works.

You need both.

4. Shortlist alternative models

Now look for possible replacements.

Use public benchmarks to find models that are likely to be good at your task. If your product builds websites, look at frontend and coding benchmarks. If it solves mathematical problems, look at math benchmarks.

Benchmarks should only help you decide which models to test. They should not decide which model you ship.

Public benchmarks don't contain your prompts, tools, schemas, context, or customer workflows. Your own evals do.

5. Run the evals and tune each model

Run every shortlisted model against the same eval suite using your existing prompts. Record the results before changing anything.

Then start adapting the prompts for each model. One model might need more rules or context. Another might need shorter instructions, different examples, or a different prompt structure.

These changes don't have to be purely additive. You should be able to add, replace, restructure, or remove instructions for a specific model.

Store prompts in versioned Markdown files. Keep the shared task instructions separate from model-specific changes. This makes the prompts easy to review, compare, and maintain.

Run the evals again after every meaningful change. You're not trying to make the new model imitate the original model. You're trying to get it to meet the same product requirements.

6. Decide whether the performance is good enough

Similar performance does not mean identical output.

Some requirements should be non-negotiable. The model may need to produce valid schemas, call critical tools correctly, follow safety rules, and avoid specific failures.

Other requirements can use acceptable thresholds: accuracy, usefulness, completeness, latency, or cost.

The model is ready when it passes the non-negotiable requirements and the complete workflow still produces what the customer needs.

7. Ship and keep updating the evals

Once you're happy with the performance, ship.

As customers use the product, you'll discover new workflows and failures. Add the useful ones to the eval suite. Over time, the evals become the specification for what your product must continue doing, regardless of which model is underneath it.

That's what building LLM-agnostic SaaS actually means.

It isn't about hiding multiple providers behind one API. It's about making the product's behaviour measurable, then proving that another model can deliver it.