Incident Report: Control D DNS Outage on October 11, 2026
On October 11, 2026, Control D’s DNS service for paid plans was down for about 54 minutes. Here is what caused it and what we are changing.
On October 11, 2026, from 16:40 to 17:35 UTC (12:40 to 1:35 p.m EDT), Control D's DNS service for paying customers stopped answering queries at every location. All Personal and Organization accounts were affected, on every protocol and every device. Almost everything online starts with a DNS lookup, so for most of you this looked like the internet had gone down. We are sorry.
Our free DNS service stayed online throughout. No query was answered incorrectly, and no customer settings or data were changed or exposed. This was our own error.
An update to our API had changed one stored setting to a format our DNS software could not read, and an error-handling flaw made our DNS software stop instead of skipping the unreadable value. That setting is shared configuration, delivered to every DNS server for paid plans at the same moment, so all of them stopped at once. Our tests did not cover this case, and our alerting and communication with you were slower than they should have been.
Incident details
| Field | Details |
|---|---|
| Incident date | October 11, 2026. |
| Incident window | 16:40 to 17:35 UTC (about 54 minutes). |
| Affected functionality | DNS resolution for all paid plans (Personal and Organization accounts), at every location, on every protocol including DNS-over-HTTPS and DNS-over-TLS, and through every app, router, device and integration. |
| Unaffected functionality | Our free DNS service. Customer settings, profiles, and data were not changed or exposed. |
| Cause | Our own error. |
Customer impact
- DNS queries from paying customers went unanswered everywhere from 16:42. Service began returning at 17:31 and was fully restored at 17:35.
- In households, every device appeared to lose internet access, including phones on mobile data that use a persistent Control D DNS profile.
- Organizations using Control D as their primary DNS lost name resolution across their endpoints.
- Our DNS software returned no answer rather than a wrong one, so no query was allowed, blocked, or redirected against your settings.
What happened
- On September 16 we released an API update that added validation for an optional per-domain override in services. Services are the ready-made entries in a profile for things like streaming platforms, each backed by a list of domains that we maintain. As part of the update, the API began saving the override as an integer. Our DNS software expects it as a string.
- The new code only ran when a domain was saved with an explicit override, such as "allow" or "block". Ordinary additions to a service did not touch it, and services were edited in the following weeks without incident.
- At 16:40 UTC on October 11, a member of our team made a routine change in our admin tools to resolve a customer request, adding four domains to one service with an explicit "allow" override. It was the first save of that kind in the 25 days since the September 16 update. The API stored the four overrides as numbers and told every DNS server for paid plans to reload that service.
- A service's domain list is shared configuration, delivered to every DNS server for paid plans at the same moment, and none of them could read the new values. Because of an error-handling flaw, our DNS software stopped instead of logging and skipping the unreadable values. As our locations went offline, a safeguard that moves traffic to healthy locations kicked in, but every location had the same problem, so there was nowhere healthy to send it. By 16:42, no queries from paying customers were being answered anywhere.
- Our DNS software restarted automatically. It was designed not to start if it could not read all of its configuration, so that it would never serve you with a half-loaded filtering policy. Every restart therefore failed on the same stored values until we corrected them.
Root cause
Contributing conditions
- Test coverage: Our tests did not check that data saved by our API stayed in the format our DNS software reads.
- No check before publishing: Configuration written by our API was not checked against our DNS software's own reader before it was published.
- Configuration changes were not staged: Updates to our DNS software itself are rolled out gradually. Configuration changes are not, so this one reached every DNS server for paid plans at the same moment.
- Alerting: Our first automated alerts, at 16:46, reported individual servers and locations failing. None of them said that DNS for paying customers was down.
Detection and response
| UTC | Event |
|---|---|
| 16:40 | Routine change to one service saved in our admin tools. Every DNS server for paid plans stops. |
| 16:42 | No DNS queries from paying customers are being answered at any location. |
| 16:46–16:47 | First automated alerts: individual servers failing and locations going offline. |
| 16:51 | DNS failures across multiple locations reported internally; on-call paged. |
| 16:55 | On-call acknowledges and begins triage. |
| 17:05–17:16 | Outage acknowledged to partners, our support team, and publicly on Reddit and Discord. |
| 17:16 | Cause traced to the 16:40 change and the September API update. |
| 17:30 | Affected service saved again with the four overrides removed. |
| 17:31–17:35 | DNS service begins returning at 17:31 and is fully restored at every location by 17:35. |
| 17:35–17:46 | All-clear sent to partners and by email to customers who had contacted us. |
| 17:58 | Edits to services in our admin tools frozen until both fixes were in place. |
| 19:06–19:40 | DNS software update released and rolled out, first to one server and then to all others. |
We were too slow to tell you what was happening. Control D had no incident status page, our network status page still showed most locations as healthy, and many of you learned on Reddit or Discord that the problem was ours. We are fixing this with a status page and new rules for incident updates, listed below.
Resolution and recovery
At 17:30 UTC we saved the affected service again with the four overrides removed, so no number was stored. Each DNS server's next automatic restart succeeded, and service was fully restored at every location by 17:35. A check of the stored configuration found the bad values in those four entries only, all on that one service.
We froze edits to services in our admin tools until two fixes were in place. The first, an update to our DNS software, narrows the startup safeguard described above: a service entry it cannot read is now set aside and logged, during reloads and at startup, instead of stopping the software. The second, an API fix, stores overrides as text again.
Corrective and preventive actions
| Area | Action | Customer outcome | Status |
|---|---|---|---|
| DNS software resilience | Set aside and log any service entry that cannot be read, during reloads and at startup, instead of stopping the software. Read the override correctly whether it is stored as text or as a number. | An unreadable service entry can no longer stop DNS for every customer. | Done. |
| API data format | Store per-domain overrides as text again, keeping the new validation. | Saving a service with an override no longer produces a value our DNS software cannot read. | Done. |
| DNS software resilience | Audit every type of configuration our DNS software loads. A further update based on that audit makes our DNS software more resistant to other kinds of unreadable configuration. | Fewer ways for a bad piece of configuration to interrupt DNS. | Running on a small number of servers since October 15; full rollout to follow. |
| Testing | Add a test that saves service changes through the real code path and checks the exact stored format, and regression tests with malformed input for every type of configuration our DNS software reads. Run against the code that was live on October 11, these tests reproduce the failure. | A format change like this one, or bad input of this kind, is caught before release. | Written; being added to our automated test pipelines. |
| Change control | Document the data formats our DNS software depends on and require them to stay backward compatible. Changes to those formats will require review by the engineers responsible for our DNS software. | Format changes are checked by the people who know what our DNS software expects. | Documentation updated; review requirement being put in place. |
| Configuration validation | Check configuration written by our API against our DNS software's own reader before it is published. | A value our DNS software cannot read is rejected before it reaches any server. | Planned |
| Staged configuration rollout | Roll out configuration changes gradually, the way software updates already are, so a change reaches a small set of servers before all of them. | A bad change reaches a small set of servers first, so it can be caught before it reaches all of them. | In design. |
| Alerting | Page on-call when answered DNS queries for paying customers drop to zero in any region for more than two minutes, when DNS servers report down, and when our DNS software crashes in more than one location. Crash alerts will page at top priority and open an incident call immediately. | Any of these would have paged us within two to four minutes on October 11. | Crash alerts being added to on-call paging; other alerts planned. |
| Status page | Launch a hosted status page that runs independently of our own infrastructure, with subscriptions and automatic posting to our community channels. A top-priority page will open a status page incident automatically. | One place to see an outage and get updates from us, even when our own systems are down. | Being set up. |
| Incident communications | Post an "investigating" notice within 15 minutes of a top-priority page, update every 30 minutes until resolved, and post an explicit all-clear. Our support team and support assistant receive the same message. | You hear about an incident from us quickly, and support can tell you what is happening. | Planned. |
Closing
This outage was our mistake, and we were slow to tell you about it. Many of you rely on Control D to protect your household or your organization, and for nearly an hour we let you down. We are sorry.
The specific error that caused the outage is fixed and released, and our DNS software now sets aside an unreadable service entry instead of stopping. We have kicked off a larger review of how the control plane and the resolvers share state, with the goal of explicit, machine-checked contracts between them. Later this month, we will embark upon a re-architecture of the entire core DNS system.