The flag flips in a second. Getting a person to it is the slow part: a pager, a groggy acknowledgment, a laptop, a VPN, a search for the right flag in the right environment, and a moment of doubt about whether this is the flag that broke things. On a good night that takes five minutes, and at 3 a.m. it can take twenty.
An automated kill switch takes the person out of that path. When the alert fires, it calls the flag directly, and the feature turns off before anyone has found their glasses. Most of this post is about picking that alert.
If you’re new to the pattern itself (what a kill switch is, what to wrap, how to pick a safe default), start with the kill switch use case. Everything here assumes the switch already exists and asks who should be allowed to flip it.
1. What an automated kill switch is
It’s a feature flag with a second caller. People still toggle it from the dashboard, and your monitoring tool can toggle it as well, through a URL that does exactly one thing: turn one flag off in one environment.
In Featureflip that URL is a flag trigger. You create it on the flag’s Triggers tab, choose an environment and the action (turn on or turn off), and paste the URL into an alert’s contact point. When the alert fires, it POSTs to the URL. The request body is ignored, so whatever payload Grafana or Datadog sends by default is fine.
What happens next is identical to someone clicking the toggle. Connected SDKs get the new state over their stream within seconds, with no deploy and no restart. Webhooks fire. The audit log records the change with the trigger’s name as the actor, so nobody has to guess later whether a person or a machine did it.
2. Which alerts should be allowed to flip a flag
CPU at 90%, a full disk or a lagging database replica say nothing about any one feature, and turning a feature off won’t fix them. Wire one of those to a kill switch and the feature vanishes for customers whenever some unrelated batch job runs hot.
Before you connect an alert to a flag, check it against these.
It measures the code behind the flag. The error rate of the new checkout endpoint, the failure count of the new payment provider’s client, the p99 latency of the recommendations call you just shipped. If the metric can only go red because of code the flag controls, turning the flag off is the correct response and you know it in advance. If working that out takes judgment, leave the decision to a person.
Turning the flag off makes it go green. People skip this check more than the others. A flag around a new code path passes. A flag around a data migration often fails it, because turning it off leaves half-written rows behind. Ask what the off state does to requests already in flight and to data the new path wrote. If the answer is “falls back to the old behavior, which still works,” automate it.
It fires on a rate. One failed request in a minute is noise. Alert on an error rate above a baseline, held for a few minutes, with a minimum request count so a quiet hour at 4 a.m. doesn’t trip it on two failures out of five. A threshold you’d be comfortable having a computer act on without asking is stricter than one that just wakes someone up, so tune it with that in mind.
If you’ve added evaluation details to your traces with an inspector recipe for OpenTelemetry, you can go one step further and check whether the failing requests were the ones on the new variant. An alert built that way is much harder to trip by accident, and the postmortem starts with evidence instead of “errors went up around the time we raised the rollout.”
3. Automate off, keep on human
A trigger can turn a flag on as well as off, and it’s tempting to wire both ends of an alert: off when it fires, on when it resolves. Resist that.
When an alert resolves, the symptom has stopped, and in this setup it stopped because the flag turned off. Turning the feature back on automatically brings the broken code back, the alert fires again, the flag turns off again, and the feature blinks on and off every few minutes for the rest of the night. Customers get a checkout that works for four minutes and then breaks again.
Keep the asymmetry. A machine can turn the feature off, because off is the safe, known state you designed the default around. Turning it back on needs someone who has read the logs, and it should happen gradually, starting with a small percentage while you watch the same alert that tripped the switch. We’d go further and skip turn-on triggers in production entirely unless you have a specific reason for one.
Both Grafana and Datadog send a second notification when an alert resolves, and both will call your trigger URL again unless you stop them. For a turn-off trigger that’s harmless: the second call returns unchanged, sends no webhook, and only leaves a “Fired” entry in the flag’s history. In Datadog you can keep the recovery from calling it at all by wrapping the webhook handle in {{#is_alert}}…{{/is_alert}} in the monitor message.
4. Wiring one up
The setup takes about five minutes per flag. With Grafana it goes like this, and the flag triggers guide has the Datadog version.
- Create the trigger. Open the flag, go to the Triggers tab, pick Production, name it after the alert (
checkout-error-rate, say), and choose Turn flag off. Copy the secret now. Featureflip shows it once. - Add a contact point in Grafana. Under Alerting, add a Webhook contact point. Set the URL to
https://api.featureflip.io/api/triggers, the method to POST, and put the secret in the Authorization header credentials with theBearerscheme. Keeping the secret in a header keeps it out of the URL, and out of any proxy log the URL passes through. - Route the alert. Add a notification policy that sends your checkout error-rate rule to that contact point, alongside the pager.
- Fire it on purpose. In a staging environment with its own trigger, call it by hand and watch the flag turn off:
curl -X POST "https://api.featureflip.io/api/triggers" \ -H "Authorization: Bearer $CHECKOUT_KILL_SECRET"The response tells you what happened:
{ "result": "applied", "flag": "checkout-v2", "environment": "staging", "enabled": false }Fire it from staging on a quiet afternoon, so a typo in the contact point turns up then and not during an incident. Leave the pager on that alert too, since someone still has to find the cause.
5. Closing the loop: show the flip where the alert is
The engineer who gets paged opens the dashboard the alert came from. If the flag change isn’t on that dashboard, they spend the first few minutes of the incident working out why the error graph suddenly dropped back to normal, and whether something else is wrong now.
So send the flag change back to the same place. Featureflip’s outbound webhooks have built-in destinations for this:
- Grafana gets an annotation for each flag change, tagged with the flag, environment and project keys. Add an annotation query filtered on the
featurefliptag and the flip shows up as a vertical line on the error-rate panel, right where the graph turns. - Datadog gets a change event, recorded as a
feature_flagchange named after the flag key, which is what Datadog’s Change Tracking reads. The toggle lands on the same timeline as your deploys and the monitor that fired. - Slack gets a message in the incident channel saying which flag changed, in which environment, and who changed it. For a trigger, “who” is the trigger’s name.
Each of these takes a minute to set up, and between them, nobody on the incident has to ask “did someone touch a flag?” The audit log answers the same question afterward: the actor reads Trigger "checkout-error-rate" and the change reason says it was fired by that trigger.
6. What can go wrong
Plan for these before you rely on it.
The URL is a credential. Anyone holding the trigger URL or its secret can turn that flag off in production. Store it as a secret in your monitoring tool, never in a repository, a runbook page or a chat message. If it leaks, rotate the trigger. The old URL stops working immediately.
Triggers are served by the Management API. The Evaluation API your SDKs read from is a separate service. If the Management API is unreachable, your SDKs keep evaluating flags from their cached configuration, but the alert can’t flip anything until it’s back. The pager is your fallback for that case, which is one more reason to keep it on.
Plans change. Triggers are on the Team plan and above. If a plan lapses, turn-off triggers keep working on purpose, because a kill switch shouldn’t stop working over a billing problem. Turn-on triggers stop until the plan is restored.
Critical environments still apply. If production is a critical environment that requires a reason for every change, a trigger firing still goes through, with the reason filled in for you. Creating or rotating the trigger there needs a reason like any other write.
A flag the code no longer reads does nothing. If the guarded code shipped without the flag check, or someone removed the check while the flag stayed, the trigger turns off a flag nobody evaluates. The flag’s Analytics tab shows whether it’s being read at all, so check it when you set the trigger up.
7. What it doesn’t replace
An automated kill switch shortens the worst minutes of an incident and stops there. Someone still has to find out why the code failed, repair any data the failing path wrote, and bring the feature back gradually. It also covers only the failures you predicted well enough to put a flag around.
The shorter version
- An automated kill switch is a feature flag your monitoring tool can turn off, through a trigger URL the alert calls.
- Only wire it to alerts that measure the code behind the flag, where turning the flag off is the known fix, on a rate with a minimum volume.
- Automate turning off. Keep turning back on with a person, gradually.
- Send the flag change back to Grafana, Datadog or Slack so the paged engineer sees it next to the alert.
- Treat the URL as a secret, rehearse it in staging, and keep the pager on the same alert.
Frequently asked questions
What is an automated kill switch?
It’s a feature flag that a monitoring alert can turn off without a person in the loop. The alert calls a secret URL tied to one flag in one environment, and the flag turns off the same way it would from the dashboard: connected SDKs update within seconds, webhooks fire, and the change is recorded in the audit log under the trigger’s name.
Should an alert turn a feature back on when it resolves?
No. An alert usually resolves because the flag turned off, so turning it back on brings the broken code back and the feature starts flapping. Let the alert turn the feature off, and have a person turn it back on after reading the logs, ideally through a small percentage rollout first.
Which monitoring tools can call a flag trigger?
Anything that can send an HTTP POST. Grafana alerting does it through a Webhook contact point, and Datadog through its Webhooks integration and an @webhook handle in the monitor message. A shell script or runbook can use curl. The request body is ignored, so no custom payload is needed.
What happens if Featureflip is down when the alert fires?
Your SDKs keep evaluating flags from their cached configuration, because evaluation runs on a separate service from the one that serves triggers. The trigger can’t flip the flag until the Management API is reachable again, which is why the same alert should still page a person.
Is a trigger URL safe to share?
Treat it like an API key. Anyone with the URL or its secret can turn that flag off in that environment. Store it as a secret in your monitoring tool, prefer sending it in the Authorization header over putting it in the URL, and rotate the trigger if it leaks.
Featureflip’s flag triggers are on the Team plan and above, and webhooks to Slack, Datadog and Grafana work on every plan, Free included. Start free, wrap the feature you’re about to ship, and connect its kill switch to the alert that watches it.