WORK IN PROGRESS
We are soon going to be working on an internal restructure of our nodes and services to further improve reliability and make serviceability easier, while making sure everything is up to date and ready to go.
Server is under investigation on 30 Sep 2026 17:10:22 (UTC+00:00)
Investigating on 4 Oct 2026 14:06:20 (UTC+00:00)
While working on reintroducing RO1, the SATA connection has started failing again. While working with the location admin we have come to the probable diagnosis that it is not easily fixable. We are still working on it.
While working on reintroducing RO1, the SATA connection has started failing again. While working with the location admin we have come to the probable diagnosis that it is not easily fixable. We are still working on it.
Monitoring on 4 Oct 2026 13:44:51 (UTC+00:00)
After a deep cleaning of the physical server and temporary replacement of the drives, RO1's drives are in normal operation again. We have put back the original drives after the admin had manually checked them in an external closure and after fixing filesystem errors.
RO1 is now considered ready to be reintroduced in the node pool, and we are actively working on that.
After a deep cleaning of the physical server and temporary replacement of the drives, RO1's drives are in normal operation again. We have put back the original drives after the admin had manually checked them in an external closure and after fixing filesystem errors.
RO1 is now considered ready to be reintroduced in the node pool, and we are actively working on that.
Identified on 4 Oct 2026 12:09:44 (UTC+00:00)
We have worked with the location admin into tracking down the issue, which is either SSD failure or SATA link failure, both which are physical issues. The location admin is now investigating and fixing the physical issue, so it is out of our hands.
We have worked with the location admin into tracking down the issue, which is either SSD failure or SATA link failure, both which are physical issues. The location admin is now investigating and fixing the physical issue, so it is out of our hands.
Scheduled on 2 Oct 2026 09:59:04 (UTC+00:00)
Some services have been migrated off RO1. We are unable to migrate other services, however the rest are mainly internal so they will not have an impact. The location admin has responded and they will look into the issue later today.
Some services have been migrated off RO1. We are unable to migrate other services, however the rest are mainly internal so they will not have an impact. The location admin has responded and they will look into the issue later today.
Update on 1 Oct 2026 20:03:55 (UTC+00:00)
We are migrating still core services which were being hosted by RO1 onto other nodes such as UK1. The location admin is currently not responsive.
We are migrating still core services which were being hosted by RO1 onto other nodes such as UK1. The location admin is currently not responsive.
Investigating on 1 Oct 2026 12:21:13 (UTC+00:00)
The location admin is still unavailable, so physical investigation of the drive links isn't possible yet. The drive link has failed completely so we were forced to fully shut down the node until fixed. In the meantime we are migrating core services which were being hosted by RO1 onto other nodes such as UK1.
The location admin is still unavailable, so physical investigation of the drive links isn't possible yet. The drive link has failed completely so we were forced to fully shut down the node until fixed. In the meantime we are migrating core services which were being hosted by RO1 onto other nodes such as UK1.
Identified on 1 Oct 2026 10:07:50 (UTC+00:00)
In the process of bringing RO1 back online the drive connection has fallen and I/O errors are preventing us from restarting the services. We will get in contact with the location admin ASAP to analyse and fix the connection.
In the process of bringing RO1 back online the drive connection has fallen and I/O errors are preventing us from restarting the services. We will get in contact with the location admin ASAP to analyse and fix the connection.
In progress on 1 Oct 2026 10:03:21 (UTC+00:00)
We have determined that with the fixes we implemented, RO1 can be taken out of maintenance.
We are finishing up work on RO1, and we are estimating for it to be done in max a couple hours.
We have determined that with the fixes we implemented, RO1 can be taken out of maintenance.
We are finishing up work on RO1, and we are estimating for it to be done in max a couple hours.
Identified on 1 Oct 2026 08:11:48 (UTC+00:00)
RO1's disks now are reporting as clean. We are doing a final assessment to see if we can take it out of maintenance for now.
RO1's disks now are reporting as clean. We are doing a final assessment to see if we can take it out of maintenance for now.
Update on 1 Oct 2026 07:10:26 (UTC+00:00)
One of RO1's drives are suspected to be failing.
The backup from RO1 to UK1 is complete. We are now looking into fixing the problem remotely.
One of RO1's drives are suspected to be failing.
The backup from RO1 to UK1 is complete. We are now looking into fixing the problem remotely.
Update on 30 Sep 2026 20:52:22 (UTC+00:00)
One of RO1's drives are suspected to be failing.
We are currently taking a full snapshot of RO1's drives onto UK1. It is currently 50% done. This is sadly taking a lot due to there being a lot of data with not a lot of bandwidth.
One of RO1's drives are suspected to be failing.
We are currently taking a full snapshot of RO1's drives onto UK1. It is currently 50% done. This is sadly taking a lot due to there being a lot of data with not a lot of bandwidth.
Update on 30 Sep 2026 17:17:23 (UTC+00:00)
One of RO1's drives are suspected to be failing.
We are currently taking a full snapshot of RO1's drives onto UK1.
We are looking into the incident.
One of RO1's drives are suspected to be failing.
We are currently taking a full snapshot of RO1's drives onto UK1.
We are looking into the incident.
Investigating on 30 Sep 2026 17:10:22 (UTC+00:00)
RO1 is reporting failures relating to its primary disk. SMART test passed but details are worrying.
We are looking into the issue.
RO1 is reporting failures relating to its primary disk. SMART test passed but details are worrying.
We are looking into the issue.
RO2 is down. on 4 Oct 2026 12:08:02 (UTC+00:00)
In progress
We are aware of the downtime of RO2. This does not affect any of our public or core services.
The downtime is due to a physical networking issue, and the location admin is working on fixing it by the end of the day.
We are aware of the downtime of RO2. This does not affect any of our public or core services.
The downtime is due to a physical networking issue, and the location admin is working on fixing it by the end of the day.
Reworking UB systems on 3 Oct 2026 10:53:51 (UTC+00:00)
Completed on 3 Oct 2026 12:01:55 (UTC+00:00)
We have finished rerouting our internal UB services and we have introduced the new UB service for testing.
We have finished rerouting our internal UB services and we have introduced the new UB service for testing.
In progress on 3 Oct 2026 10:53:51 (UTC+00:00)
We are working our internal UB routing systems in order to accommodate a new UB system. Users of any one of these UB systems may be affected.
We are working our internal UB routing systems in order to accommodate a new UB system. Users of any one of these UB systems may be affected.
Services down on 2 Oct 2026 17:53:36 (UTC+00:00)
Resolved on 2 Oct 2026 18:48:33 (UTC+00:00)
All services have been restarted and now they have recovered.
All services have been restarted and now they have recovered.
In progress on 2 Oct 2026 18:29:32 (UTC+00:00)
We have fixed the resource exhaustion issue, however due to that many of our services are now in a bugged state. We are restarting those services so that they are in a proper working state, before we consider this issue finished.
We have fixed the resource exhaustion issue, however due to that many of our services are now in a bugged state. We are restarting those services so that they are in a proper working state, before we consider this issue finished.
In progress on 2 Oct 2026 18:19:26 (UTC+00:00)
The resource exhaustion was due to many playwright containers taking up all the RAM. We have removed those containers and we are actively looking for the creator of those containers as they keep respawning.
The resource exhaustion was due to many playwright containers taking up all the RAM. We have removed those containers and we are actively looking for the creator of those containers as they keep respawning.
Identified on 2 Oct 2026 18:13:22 (UTC+00:00)
It seems NL1 is unstable due to resource exhaustion. We are fixing the issue.
It seems NL1 is unstable due to resource exhaustion. We are fixing the issue.
Update on 2 Oct 2026 18:10:08 (UTC+00:00)
In the process of recovering the services our SSH connection to the server has failed. We are looking into recovering the connection to the server.
In the process of recovering the services our SSH connection to the server has failed. We are looking into recovering the connection to the server.
Investigating on 2 Oct 2026 17:53:36 (UTC+00:00)
We are looking into an issue with numerous services suddenly going down.
We are looking into an issue with numerous services suddenly going down.