GBS Status
Server is under investigation on 30 Sep 2026 18:10:22 (UTC+01:00)
Investigating on 4 Oct 2026 15:06:20 (UTC+01:00)
While working on reintroducing RO1, the SATA connection has started failing again. While working with the location admin we have come to the probable diagnosis that it is not easily fixable. We are still working on it.
While working on reintroducing RO1, the SATA connection has started failing again. While working with the location admin we have come to the probable diagnosis that it is not easily fixable. We are still working on it.
Monitoring on 4 Oct 2026 14:44:51 (UTC+01:00)
After a deep cleaning of the physical server and temporary replacement of the drives, RO1's drives are in normal operation again. We have put back the original drives after the admin had manually checked them in an external closure and after fixing filesystem errors.
RO1 is now considered ready to be reintroduced in the node pool, and we are actively working on that.
After a deep cleaning of the physical server and temporary replacement of the drives, RO1's drives are in normal operation again. We have put back the original drives after the admin had manually checked them in an external closure and after fixing filesystem errors.
RO1 is now considered ready to be reintroduced in the node pool, and we are actively working on that.
Identified on 4 Oct 2026 13:09:44 (UTC+01:00)
We have worked with the location admin into tracking down the issue, which is either SSD failure or SATA link failure, both which are physical issues. The location admin is now investigating and fixing the physical issue, so it is out of our hands.
We have worked with the location admin into tracking down the issue, which is either SSD failure or SATA link failure, both which are physical issues. The location admin is now investigating and fixing the physical issue, so it is out of our hands.
Scheduled on 2 Oct 2026 10:59:04 (UTC+01:00)
Some services have been migrated off RO1. We are unable to migrate other services, however the rest are mainly internal so they will not have an impact. The location admin has responded and they will look into the issue later today.
Some services have been migrated off RO1. We are unable to migrate other services, however the rest are mainly internal so they will not have an impact. The location admin has responded and they will look into the issue later today.
Update on 1 Oct 2026 21:03:55 (UTC+01:00)
We are migrating still core services which were being hosted by RO1 onto other nodes such as UK1. The location admin is currently not responsive.
We are migrating still core services which were being hosted by RO1 onto other nodes such as UK1. The location admin is currently not responsive.
Investigating on 1 Oct 2026 13:21:13 (UTC+01:00)
The location admin is still unavailable, so physical investigation of the drive links isn't possible yet. The drive link has failed completely so we were forced to fully shut down the node until fixed. In the meantime we are migrating core services which were being hosted by RO1 onto other nodes such as UK1.
The location admin is still unavailable, so physical investigation of the drive links isn't possible yet. The drive link has failed completely so we were forced to fully shut down the node until fixed. In the meantime we are migrating core services which were being hosted by RO1 onto other nodes such as UK1.
Identified on 1 Oct 2026 11:07:50 (UTC+01:00)
In the process of bringing RO1 back online the drive connection has fallen and I/O errors are preventing us from restarting the services. We will get in contact with the location admin ASAP to analyse and fix the connection.
In the process of bringing RO1 back online the drive connection has fallen and I/O errors are preventing us from restarting the services. We will get in contact with the location admin ASAP to analyse and fix the connection.
In progress on 1 Oct 2026 11:03:21 (UTC+01:00)
We have determined that with the fixes we implemented, RO1 can be taken out of maintenance.
We are finishing up work on RO1, and we are estimating for it to be done in max a couple hours.
We have determined that with the fixes we implemented, RO1 can be taken out of maintenance.
We are finishing up work on RO1, and we are estimating for it to be done in max a couple hours.
Identified on 1 Oct 2026 09:11:48 (UTC+01:00)
RO1's disks now are reporting as clean. We are doing a final assessment to see if we can take it out of maintenance for now.
RO1's disks now are reporting as clean. We are doing a final assessment to see if we can take it out of maintenance for now.
Update on 1 Oct 2026 08:10:26 (UTC+01:00)
One of RO1's drives are suspected to be failing.
The backup from RO1 to UK1 is complete. We are now looking into fixing the problem remotely.
One of RO1's drives are suspected to be failing.
The backup from RO1 to UK1 is complete. We are now looking into fixing the problem remotely.
Update on 30 Sep 2026 21:52:22 (UTC+01:00)
One of RO1's drives are suspected to be failing.
We are currently taking a full snapshot of RO1's drives onto UK1. It is currently 50% done. This is sadly taking a lot due to there being a lot of data with not a lot of bandwidth.
One of RO1's drives are suspected to be failing.
We are currently taking a full snapshot of RO1's drives onto UK1. It is currently 50% done. This is sadly taking a lot due to there being a lot of data with not a lot of bandwidth.
Update on 30 Sep 2026 18:17:23 (UTC+01:00)
One of RO1's drives are suspected to be failing.
We are currently taking a full snapshot of RO1's drives onto UK1.
We are looking into the incident.
One of RO1's drives are suspected to be failing.
We are currently taking a full snapshot of RO1's drives onto UK1.
We are looking into the incident.
Investigating on 30 Sep 2026 18:10:22 (UTC+01:00)
RO1 is reporting failures relating to its primary disk. SMART test passed but details are worrying.
We are looking into the issue.
RO1 is reporting failures relating to its primary disk. SMART test passed but details are worrying.
We are looking into the issue.
RO2 is down. on 4 Oct 2026 13:08:02 (UTC+01:00)
In progress
We are aware of the downtime of RO2. This does not affect any of our public or core services.
The downtime is due to a physical networking issue, and the location admin is working on fixing it by the end of the day.
We are aware of the downtime of RO2. This does not affect any of our public or core services.
The downtime is due to a physical networking issue, and the location admin is working on fixing it by the end of the day.
Reworking UB systems on 3 Oct 2026 11:53:51 (UTC+01:00)
Completed on 3 Oct 2026 13:01:55 (UTC+01:00)
We have finished rerouting our internal UB services and we have introduced the new UB service for testing.
We have finished rerouting our internal UB services and we have introduced the new UB service for testing.
In progress on 3 Oct 2026 11:53:51 (UTC+01:00)
We are working our internal UB routing systems in order to accommodate a new UB system. Users of any one of these UB systems may be affected.
We are working our internal UB routing systems in order to accommodate a new UB system. Users of any one of these UB systems may be affected.
Services down on 2 Oct 2026 18:53:36 (UTC+01:00)
Resolved on 2 Oct 2026 19:48:33 (UTC+01:00)
All services have been restarted and now they have recovered.
All services have been restarted and now they have recovered.
In progress on 2 Oct 2026 19:29:32 (UTC+01:00)
We have fixed the resource exhaustion issue, however due to that many of our services are now in a bugged state. We are restarting those services so that they are in a proper working state, before we consider this issue finished.
We have fixed the resource exhaustion issue, however due to that many of our services are now in a bugged state. We are restarting those services so that they are in a proper working state, before we consider this issue finished.
In progress on 2 Oct 2026 19:19:26 (UTC+01:00)
The resource exhaustion was due to many playwright containers taking up all the RAM. We have removed those containers and we are actively looking for the creator of those containers as they keep respawning.
The resource exhaustion was due to many playwright containers taking up all the RAM. We have removed those containers and we are actively looking for the creator of those containers as they keep respawning.
Identified on 2 Oct 2026 19:13:22 (UTC+01:00)
It seems NL1 is unstable due to resource exhaustion. We are fixing the issue.
It seems NL1 is unstable due to resource exhaustion. We are fixing the issue.
Update on 2 Oct 2026 19:10:08 (UTC+01:00)
In the process of recovering the services our SSH connection to the server has failed. We are looking into recovering the connection to the server.
In the process of recovering the services our SSH connection to the server has failed. We are looking into recovering the connection to the server.
Investigating on 2 Oct 2026 18:53:36 (UTC+01:00)
We are looking into an issue with numerous services suddenly going down.
We are looking into an issue with numerous services suddenly going down.
1 server is down on 29 Sep 2026 14:48:28 (UTC+01:00)
Completed on 29 Sep 2026 15:23:55 (UTC+01:00)
Postmortem
The directly observed outage was 3 minutes 38 seconds. The SSH control panel was unavailable for about 4 minutes 38 seconds.
This was probably due to the upstream ISP remotely restarting the router for an update or similar, which had that effect.
Postmortem Summary
RO2 did not crash or reboot. At 13:43:04 UTC its only active uplink, lost its access-point association and DHCP lease. The host and containers continued running, but it had no usable default route, so the server was unavailable. DHCP returned at 13:46:42 UTC and remote SSH to the server was restored at 13:47:42 UTC.
The directly observed outage was 3 minutes 38 seconds. The SSH control panel was unavailable for about 4 minutes 38 seconds.What caused it
All logs on RO2 all record the same event: upstream disconnected, carrier/DHCP disappeared, then the access point was rejoined. There was no shutdown, kernel panic, out-of-memory event, suspend, or Docker restart in the incident window.This was probably due to the upstream ISP remotely restarting the router for an update or similar, which had that effect.
Separate DNS issue corrected
After the initial outage, RO2's IP has changed, so ro2.server.zombidev.me returned an old address that was linked to an unrelated Apache/Synology page. The stale DNS route was corrected, and our custom automation software was patched to automatically update DDNS more reliably.Conclusion
We have come to the conclusion that this outage was not one that we could prevent, as it was caused by our upstream ISP without any notice.
Update on 29 Sep 2026 15:09:52 (UTC+01:00)
We are still looking into the incident.
We are still looking into the incident.
Resolved on 29 Sep 2026 15:09:19 (UTC+01:00)
We have found that the external IP has changed after the series of events that happened before. The server is now externally accessible once again. We are still looking into the incident.
We have found that the external IP has changed after the series of events that happened before. The server is now externally accessible once again. We are still looking into the incident.
In progress on 29 Sep 2026 15:08:07 (UTC+01:00)
Internally networking has recovered after applying a patch but externally networking has been redirected to a random Synology VPN plus page. We are looking into the incident.
Internally networking has recovered after applying a patch but externally networking has been redirected to a random Synology VPN plus page. We are looking into the incident.
In progress on 29 Sep 2026 15:01:22 (UTC+01:00)
RO2 is intermittently going down and then back up. Treat this as the node being fully down. We are looking what is happening remotely.
RO2 is intermittently going down and then back up. Treat this as the node being fully down. We are looking what is happening remotely.
Resolved on 29 Sep 2026 14:53:36 (UTC+01:00)
RO2 is back up. We are still looking into why the incident happened.
RO2 is back up. We are still looking into why the incident happened.
Investigating on 29 Sep 2026 14:48:28 (UTC+01:00)
We are aware RO2 is down and the location admin has been notified. We are awaiting for their response.
We are aware RO2 is down and the location admin has been notified. We are awaiting for their response.