1 server is down on 29 Sep 2026 14:48:28 (UTC+01:00)
Completed on 29 Sep 2026 15:23:55 (UTC+01:00)
Postmortem
The directly observed outage was 3 minutes 38 seconds. The SSH control panel was unavailable for about 4 minutes 38 seconds.
This was probably due to the upstream ISP remotely restarting the router for an update or similar, which had that effect.
Postmortem Summary
RO2 did not crash or reboot. At 13:43:04 UTC its only active uplink, lost its access-point association and DHCP lease. The host and containers continued running, but it had no usable default route, so the server was unavailable. DHCP returned at 13:46:42 UTC and remote SSH to the server was restored at 13:47:42 UTC.
The directly observed outage was 3 minutes 38 seconds. The SSH control panel was unavailable for about 4 minutes 38 seconds.What caused it
All logs on RO2 all record the same event: upstream disconnected, carrier/DHCP disappeared, then the access point was rejoined. There was no shutdown, kernel panic, out-of-memory event, suspend, or Docker restart in the incident window.This was probably due to the upstream ISP remotely restarting the router for an update or similar, which had that effect.
Separate DNS issue corrected
After the initial outage, RO2's IP has changed, so ro2.server.zombidev.me returned an old address that was linked to an unrelated Apache/Synology page. The stale DNS route was corrected, and our custom automation software was patched to automatically update DDNS more reliably.Conclusion
We have come to the conclusion that this outage was not one that we could prevent, as it was caused by our upstream ISP without any notice.
Update on 29 Sep 2026 15:09:52 (UTC+01:00)
We are still looking into the incident.
We are still looking into the incident.
Resolved on 29 Sep 2026 15:09:19 (UTC+01:00)
We have found that the external IP has changed after the series of events that happened before. The server is now externally accessible once again. We are still looking into the incident.
We have found that the external IP has changed after the series of events that happened before. The server is now externally accessible once again. We are still looking into the incident.
In progress on 29 Sep 2026 15:08:07 (UTC+01:00)
Internally networking has recovered after applying a patch but externally networking has been redirected to a random Synology VPN plus page. We are looking into the incident.
Internally networking has recovered after applying a patch but externally networking has been redirected to a random Synology VPN plus page. We are looking into the incident.
In progress on 29 Sep 2026 15:01:22 (UTC+01:00)
RO2 is intermittently going down and then back up. Treat this as the node being fully down. We are looking what is happening remotely.
RO2 is intermittently going down and then back up. Treat this as the node being fully down. We are looking what is happening remotely.
Resolved on 29 Sep 2026 14:53:36 (UTC+01:00)
RO2 is back up. We are still looking into why the incident happened.
RO2 is back up. We are still looking into why the incident happened.
Investigating on 29 Sep 2026 14:48:28 (UTC+01:00)
We are aware RO2 is down and the location admin has been notified. We are awaiting for their response.
We are aware RO2 is down and the location admin has been notified. We are awaiting for their response.