Can I use feature flags for AI and LLM apps?

Yes. The model name, system prompt, temperature, and token limit live in a single JSON flag, so you change the model or tune the prompt from the dashboard without a deploy. A targeting rule routes specific tenants to a specific model, a percentage rollout migrates a new model in gradually, and the fallback config acts as a fail-safe kill switch. The flag owns the configuration and the routing decision; your eval and observability tools own whether the output is good and what it costs.

How do I build an LLM model kill switch?

Put the model config behind a flag and give the evaluation a known-good fallback. When a model starts erroring, hallucinating, or running slow or expensive, change the flag value in the dashboard, or ramp its rollout back to zero, and every connected SDK picks up the change over the stream within a second or two, fleet-wide, with no redeploy. Because reads are served from a local cache, the switch keeps working even if the control plane is briefly unreachable, and the fallback config you passed is the ultimate backstop.

How do I roll out a new model gradually, like gpt-4 to claude?

Stack a percentage rollout on the model-config flag to move traffic from the current model to the new one in steps, for example 10 then 50 then 100 percent. Sticky bucketing keeps each user on one model as you raise the number, so a single conversation does not switch models between turns, and the cohort holds together across every replica. Watch your evals and cost between steps, and raise the percentage only when the signal looks healthy. Rolling back is the same in reverse, and it takes effect in seconds.

Can I use a different model or prompt per customer?

Yes. Add a targeting rule on an attribute you pass at evaluation, such as the tenant id or plan, and serve that segment a specific model or prompt. Rules run top to bottom and the first match wins, so an override for a named enterprise customer can sit above the broad rule everyone else falls through. Pinning a tenant to a model, or moving them off it, is a dashboard change that every SDK reflects on its next evaluation, with no code change and no deploy.

Should I store the prompt and temperature in a feature flag?

Store the live selection there. One JSON flag can hold the model name, system prompt, temperature, and max_tokens as a single value, so the whole configuration changes together and ships to every instance at once. Keep the authoring and version history of your prompts in your repo or a dedicated prompt tool, and keep provider API keys in your secrets manager rather than the flag. The flag is where you choose which configuration is live, not where you archive every version or store secrets.

What is the difference between a feature flag and an AI eval tool?

A feature flag decides which model, prompt, and settings a request uses and who gets them. An eval or observability tool measures the other half: whether the responses are good, whether a new model regressed, and what each call costs in tokens and latency. The two compose. The flag delivers and routes the model inside your application, and the eval tool tells you whether the change was an improvement. Many AI features need the flag from day one; add the eval tooling when you want the migration driven by data rather than by whether the model threw an error.

Feature flags for AI and LLM apps

Put model choice, the system prompt, and generation settings behind a flag. Swap models with no deploy, migrate gradually, route per tenant, and kill a bad model in seconds. The flag owns the config and routing; your eval and observability tools own whether the output is good.

Last updated: July 8, 2026

An AI feature carries a lot of config: which model to call, the system prompt, the temperature, the token limit, and which customers get which. A feature flag is a clean place to keep that config and change it live. You ship one JSON flag holding the model settings, read it on the request path, and change the model, the prompt, or who gets them from the dashboard rather than a deploy. A percentage rollout migrates a new model in gradually and keeps each user on one model, a targeting rule routes specific tenants, and because the SDK serves a cached copy of the config, the model kill switch keeps working even if the control plane has a blip. What a flag does not do is judge whether the new model is better or track its cost. That stays with your eval and observability stack. The flag owns the config and routing decision; those tools own the quality and cost. For the mechanics, see how rollout strategies and targeting rules work.

Get started free Read the rollout docs

Why put model choice behind a flag?

Four reasons teams reach for a flag when a model, a prompt, or a provider is likely to change.

1
Swap models without a deploy
The model name and its settings live in config, not in a shipped constant. Move to a new model, change providers, or edit the prompt in the dashboard, and every instance reads the new value on its next evaluation. The code that builds the request never changes.
2
Migrate gradually, keep it sticky
Move from one model to another with a percentage rollout instead of a hard cutover. Sticky bucketing keeps each user on a single model as you ramp, so a conversation does not switch models between turns while you watch evals and cost climb the steps.
3
Kill a bad model in seconds
A new model that hallucinates, slows down, or runs up cost is a config change away from gone. Flip the flag or ramp the rollout to zero and every SDK reverts to the known-good model within a second or two, fleet-wide, with no rebuild and no scramble to revert a deploy.
4
Route per tenant or cohort
Pin an enterprise customer to a specific model, give one cohort a different prompt, or hold a tenant on the old model while everyone else migrates. A targeting rule handles it, so per-customer routing is configuration rather than a fork in your code.

What an AI feature needs, and which half a flag covers

An AI feature has a config side and a quality side. Feature flags own the config side completely. The quality side stays with the eval and observability tools you run around your models.

Part	Where it comes from	What it covers
Which model to call	The feature flag	A JSON flag holds the model name, so switching from one model to another, or rolling a new one out gradually, is a config change in the dashboard rather than a code deploy.
The prompt and generation settings	The feature flag	The system prompt, temperature, and max_tokens ride in the same JSON flag value, so you tune them and ship the change to every instance without touching application code.
Who gets which model	The feature flag	A targeting rule routes a specific tenant, plan, or cohort to a particular model or prompt, and a percentage rollout moves everyone else over in steps. Both are dashboard settings.
Falling back when the model fails	Your app, with the flag	Your code catches a model error and retries with the flag's fallback config, and the SDK already serves a cached copy of that config if it cannot reach Featureflip. The flag supplies the safe default; your code decides when to use it.
Whether the new model is better	Your eval stack	Scoring output quality, catching regressions, and comparing models across a test set is the job of an eval or LLM-observability tool. The flag ships the model; it does not grade the responses.
Token cost and latency	Your observability stack	Tracking spend per model and response latency belongs to your metrics and tracing tools. The flag can move traffic between models; your observability tells you what that move cost.

That division is the whole mental model. Holding a model name, prompt, and settings in a flag is a form of remote config, changed from a dashboard rather than a deploy, and one JSON flag can carry several values at once the way a multivariate flag serves more than on and off. The flag guarantees a fast, consistent answer to which model and prompt this request uses. Whether the output is good, and what it cost, are questions your eval and observability tools answer.

How to put a model behind a flag

Six steps. The first four are the setup; the last two are the rollout and routing you change without shipping.

Step	What you do	Detail
1	Create the model-config flag	Add a JSON flag whose value holds the model name, system prompt, temperature, and token limit. One flag carries the whole configuration for the feature.
2	Set the safe default	The fallback you pass to the evaluation is a known-good, reliable model config. It is what the SDK serves if it cannot reach Featureflip, so an outage falls back to a model you trust rather than an experimental one.
3	Read it on the request path	Call jsonVariation with the user context right before you build the model request. Evaluation is in-memory and sub-millisecond, so reading the config adds no network hop ahead of the model call.
4	Add the fallback path	Wrap the model call so that if it errors, times out, or returns something unusable, your code retries with the fallback config. Pair that with the flag as a manual kill switch you can flip from the dashboard.
5	Migrate a new model gradually	Stack a percentage rollout to move traffic from the current model to a new one in steps, 10 then 50 then 100. Sticky bucketing keeps each user on one model, so a single conversation does not switch models between turns.
6	Route specific tenants	Add a targeting rule above the rollout to pin an enterprise customer or a cohort to a particular model or prompt, independent of the gradual migration everyone else is on.

Steps five and six are ordinary targeting rules and a percentage rollout, the same building blocks a progressive release uses, pointed at model choice instead of a UI feature. Order the rules so a named-tenant override sits above the broad rollout, since the first match wins.

How Featureflip handles the config side

The evaluation and rule mechanics that turn a model config into a fast, consistent decision, so the only thing your other tools own is the quality and the cost.

JSON flag values for model config. A single jsonVariation flag carries the model name, system prompt, temperature, and token limit, so the whole configuration changes together in one place. See the SDK overview for the variation methods.
Per-tenant and per-cohort routing. A targeting rule serves a specific model or prompt to a plan, tenant, or segment. Rules run top to bottom and the first match wins, so an override sits above the broad rule. See the targeting docs for operators and ordering.
Sticky gradual migration. A percentage rollout moves traffic to a new model in steps, and deterministic bucketing keeps each user on one model as you ramp, across every replica. See rollout strategies.
Sub-millisecond local evaluation. The config read runs against an in-memory copy, so choosing the model adds no network hop before the model call, where every added millisecond of latency is felt.
Fail-safe cached config. Reads are served from a local cache, so the model kill switch and the current config keep working even if the control plane is briefly unreachable. The default you pass is the ultimate backstop.
Real-time changes. Change the model, edit the prompt, or ramp the rollout in the dashboard and every connected SDK picks it up over a Server-Sent Events stream within a second or two, fleet-wide, with no redeploy.

What it looks like in your app

The application reads the model config from the flag, passes the user context, and uses whatever the flag returns to build the request. The last argument is the fallback, a known-good config returned if the SDK cannot reach Featureflip, so the safe model doubles as the outage default:

const context = {
  user_id: 'user-456',
  tenant_id: 'org-123',
  plan: user.plan,
};

// The fallback is a known-good model config: served if the SDK
// can't reach Featureflip, so an outage never breaks generation.
const modelConfig = client.jsonVariation('llm-model-config', context, {
  model: 'gpt-4o-mini',
  system_prompt: 'You are a helpful support assistant.',
  temperature: 0.7,
  max_tokens: 1024,
});

const reply = await llm.chat({
  model: modelConfig.model,
  temperature: modelConfig.temperature,
  max_tokens: modelConfig.max_tokens,
  messages: [
    { role: 'system', content: modelConfig.system_prompt },
    { role: 'user', content: userMessage },
  ],
});

To make the model a kill switch, wrap the call so a failure falls back to the safe config, and flip the flag from the dashboard the moment a model misbehaves. Both paths return generation the user can trust:

// Flip 'llm-model-config' in the dashboard to swap the whole fleet
// to a known-good model in seconds, with no redeploy. In code, a
// failed call also falls back to the safe default config.
try {
  return await callModel(modelConfig, userMessage);
} catch (err) {
  const safe = {
    model: 'gpt-4o-mini',
    system_prompt: 'You are a helpful support assistant.',
    temperature: 0.3,
    max_tokens: 1024,
  };
  return await callModel(safe, userMessage);
}

The rollout and routing live entirely in the dashboard. Ramp llm-model-config from the current model to a new one, or pin a tenant to a specific model with a targeting rule, and every SDK reflects it on its next evaluation without a rebuild. Because the fallback is a model you trust, an outage fails safe to known-good generation rather than an experimental model. The same jsonVariation call works in every language the platform supports, from Python and Go to C#, Java, and Node. For a step-by-step Python walkthrough, see rolling out a new LLM model behind a flag, or pick a quickstart from the SDK overview.

Model config vs model quality

A flag covers the config side. An eval or observability tool covers the quality side. Knowing which one you need keeps you from asking a flag to do an eval platform's job.

Dimension	Model config (a flag)	Model quality (eval tools)
Question it answers	Which model, prompt, and settings does this request use, and who gets them?	Is the output any good, and what does it cost to produce?
What it owns	The config values and the routing and rollout decision	Eval scores, output quality, token spend, and latency
Source of truth	Reads the model config you set in the dashboard	Owns the metrics, traces, and eval results
How it changes	Flip a value or ramp a rollout in the dashboard	Numbers flow in from your eval and observability tooling
Right tool	A feature flag	An eval, tracing, or LLM-observability tool

They compose. The flag is the switch that decides which model and prompt run inside your application, and the eval tool owns the scores that tell you whether the change helped. If all you need is to change models, route per tenant, and be able to pull a bad one, a flag is the whole answer. If you also need to measure quality and cost, pair the flag with an eval or observability tool and let the flag deliver the model in front of it.

Common mistakes to avoid

The patterns that turn a clean setup into a mess. Most of them come from asking the flag to own something an eval, prompt, or secrets tool should.

Treating the flag as a prompt studio

A flag holds the config that is live right now and switches it. It is not a place to author, version, and diff every prompt you have ever written. Keep prompt history in your repo or a dedicated prompt tool, and let the flag pick which version is serving.

Recomputing the model on every request

If the model is chosen fresh on each call without sticky bucketing, a user can jump between models mid-conversation, which makes the experience inconsistent and the results hard to compare. Bucket on the user or conversation id so a user stays on one model until you change the rule.

Making the experimental model the default

The fallback config passed to the evaluation is what the SDK serves during an outage. If that default is the new, unproven model, a control-plane blip sends every request to it. Default to the model you trust and let a matched rule opt users into the new one.

Expecting the flag to tell you the model is better

A flag ships a model; it does not measure the output. Rolling a new model to 100% because it did not error is not the same as knowing it is better. Drive the migration with an eval harness or observability, and use the flag to act on what they show.

Putting API keys in the flag value

Model names, prompts, and numeric settings belong in the flag. Provider API keys and other secrets do not, especially since client SDKs evaluate config in the browser. Keep credentials in your secrets manager and keep only the model configuration in the flag.

When a flag is the wrong tool

A flag makes model choice and routing easy, but it is not an AI platform. A few needs call for something more:

You need to author, version, and compare prompts. Treating a prompt as a first-class artifact with history and diffs is a prompt-management tool or plain version control. The flag selects which prompt is live; it is not the editor or the change log.
You need to measure output quality or run evals. Scoring responses, catching regressions, and comparing models across a test set is an eval or observability platform's job. The flag delivers the model; it does not judge it.
You want a statistically valid comparison of two models. If the goal is to prove one model beats another with significance rather than to migrate safely, that is an experiment. See A/B testing with feature flags, where the flag delivers a sticky split and your analytics call the winner.
You need a model gateway or key management. Routing around provider outages, enforcing quotas, and holding API keys are gateway and secrets-manager concerns. A flag chooses the model; it does not broker the connection or store the credentials.

Frequently asked questions

Can I use feature flags for AI and LLM apps?: Yes. The model name, system prompt, temperature, and token limit live in a single JSON flag, so you change the model or tune the prompt from the dashboard without a deploy. A targeting rule routes specific tenants to a specific model, a percentage rollout migrates a new model in gradually, and the fallback config acts as a fail-safe kill switch. The flag owns the configuration and the routing decision; your eval and observability tools own whether the output is good and what it costs.
How do I build an LLM model kill switch?: Put the model config behind a flag and give the evaluation a known-good fallback. When a model starts erroring, hallucinating, or running slow or expensive, change the flag value in the dashboard, or ramp its rollout back to zero, and every connected SDK picks up the change over the stream within a second or two, fleet-wide, with no redeploy. Because reads are served from a local cache, the switch keeps working even if the control plane is briefly unreachable, and the fallback config you passed is the ultimate backstop.
How do I roll out a new model gradually, like gpt-4 to claude?: Stack a percentage rollout on the model-config flag to move traffic from the current model to the new one in steps, for example 10 then 50 then 100 percent. Sticky bucketing keeps each user on one model as you raise the number, so a single conversation does not switch models between turns, and the cohort holds together across every replica. Watch your evals and cost between steps, and raise the percentage only when the signal looks healthy. Rolling back is the same in reverse, and it takes effect in seconds.
Can I use a different model or prompt per customer?: Yes. Add a targeting rule on an attribute you pass at evaluation, such as the tenant id or plan, and serve that segment a specific model or prompt. Rules run top to bottom and the first match wins, so an override for a named enterprise customer can sit above the broad rule everyone else falls through. Pinning a tenant to a model, or moving them off it, is a dashboard change that every SDK reflects on its next evaluation, with no code change and no deploy.
Should I store the prompt and temperature in a feature flag?: Store the live selection there. One JSON flag can hold the model name, system prompt, temperature, and max_tokens as a single value, so the whole configuration changes together and ships to every instance at once. Keep the authoring and version history of your prompts in your repo or a dedicated prompt tool, and keep provider API keys in your secrets manager rather than the flag. The flag is where you choose which configuration is live, not where you archive every version or store secrets.
What is the difference between a feature flag and an AI eval tool?: A feature flag decides which model, prompt, and settings a request uses and who gets them. An eval or observability tool measures the other half: whether the responses are good, whether a new model regressed, and what each call costs in tokens and latency. The two compose. The flag delivers and routes the model inside your application, and the eval tool tells you whether the change was an improvement. Many AI features need the flag from day one; add the eval tooling when you want the migration driven by data rather than by whether the model threw an error.

Put your AI models behind a flag

Free Solo plan covers 10 flags and 2 environments. No credit card, no demo call: create a JSON flag and read your model config from it.

Get started free See pricing

Feature flags for AI and LLM apps

Why put model choice behind a flag?

Swap models without a deploy

Migrate gradually, keep it sticky

Kill a bad model in seconds

Route per tenant or cohort