Server is under investigation on 30 Sep 2026 18:10:22 (UTC+01:00)
Investigating on 4 Oct 2026 15:06:20 (UTC+01:00)
While working on reintroducing RO1, the SATA connection has started failing again. While working with the location admin we have come to the probable diagnosis that it is not easily fixable. We are still working on it.
While working on reintroducing RO1, the SATA connection has started failing again. While working with the location admin we have come to the probable diagnosis that it is not easily fixable. We are still working on it.
Monitoring on 4 Oct 2026 14:44:51 (UTC+01:00)
After a deep cleaning of the physical server and temporary replacement of the drives, RO1's drives are in normal operation again. We have put back the original drives after the admin had manually checked them in an external closure and after fixing filesystem errors.
RO1 is now considered ready to be reintroduced in the node pool, and we are actively working on that.
After a deep cleaning of the physical server and temporary replacement of the drives, RO1's drives are in normal operation again. We have put back the original drives after the admin had manually checked them in an external closure and after fixing filesystem errors.
RO1 is now considered ready to be reintroduced in the node pool, and we are actively working on that.
Identified on 4 Oct 2026 13:09:44 (UTC+01:00)
We have worked with the location admin into tracking down the issue, which is either SSD failure or SATA link failure, both which are physical issues. The location admin is now investigating and fixing the physical issue, so it is out of our hands.
We have worked with the location admin into tracking down the issue, which is either SSD failure or SATA link failure, both which are physical issues. The location admin is now investigating and fixing the physical issue, so it is out of our hands.
Scheduled on 2 Oct 2026 10:59:04 (UTC+01:00)
Some services have been migrated off RO1. We are unable to migrate other services, however the rest are mainly internal so they will not have an impact. The location admin has responded and they will look into the issue later today.
Some services have been migrated off RO1. We are unable to migrate other services, however the rest are mainly internal so they will not have an impact. The location admin has responded and they will look into the issue later today.
Update on 1 Oct 2026 21:03:55 (UTC+01:00)
We are migrating still core services which were being hosted by RO1 onto other nodes such as UK1. The location admin is currently not responsive.
We are migrating still core services which were being hosted by RO1 onto other nodes such as UK1. The location admin is currently not responsive.
Investigating on 1 Oct 2026 13:21:13 (UTC+01:00)
The location admin is still unavailable, so physical investigation of the drive links isn't possible yet. The drive link has failed completely so we were forced to fully shut down the node until fixed. In the meantime we are migrating core services which were being hosted by RO1 onto other nodes such as UK1.
The location admin is still unavailable, so physical investigation of the drive links isn't possible yet. The drive link has failed completely so we were forced to fully shut down the node until fixed. In the meantime we are migrating core services which were being hosted by RO1 onto other nodes such as UK1.
Identified on 1 Oct 2026 11:07:50 (UTC+01:00)
In the process of bringing RO1 back online the drive connection has fallen and I/O errors are preventing us from restarting the services. We will get in contact with the location admin ASAP to analyse and fix the connection.
In the process of bringing RO1 back online the drive connection has fallen and I/O errors are preventing us from restarting the services. We will get in contact with the location admin ASAP to analyse and fix the connection.
In progress on 1 Oct 2026 11:03:21 (UTC+01:00)
We have determined that with the fixes we implemented, RO1 can be taken out of maintenance.
We are finishing up work on RO1, and we are estimating for it to be done in max a couple hours.
We have determined that with the fixes we implemented, RO1 can be taken out of maintenance.
We are finishing up work on RO1, and we are estimating for it to be done in max a couple hours.
Identified on 1 Oct 2026 09:11:48 (UTC+01:00)
RO1's disks now are reporting as clean. We are doing a final assessment to see if we can take it out of maintenance for now.
RO1's disks now are reporting as clean. We are doing a final assessment to see if we can take it out of maintenance for now.
Update on 1 Oct 2026 08:10:26 (UTC+01:00)
One of RO1's drives are suspected to be failing.
The backup from RO1 to UK1 is complete. We are now looking into fixing the problem remotely.
One of RO1's drives are suspected to be failing.
The backup from RO1 to UK1 is complete. We are now looking into fixing the problem remotely.
Update on 30 Sep 2026 21:52:22 (UTC+01:00)
One of RO1's drives are suspected to be failing.
We are currently taking a full snapshot of RO1's drives onto UK1. It is currently 50% done. This is sadly taking a lot due to there being a lot of data with not a lot of bandwidth.
One of RO1's drives are suspected to be failing.
We are currently taking a full snapshot of RO1's drives onto UK1. It is currently 50% done. This is sadly taking a lot due to there being a lot of data with not a lot of bandwidth.
Update on 30 Sep 2026 18:17:23 (UTC+01:00)
One of RO1's drives are suspected to be failing.
We are currently taking a full snapshot of RO1's drives onto UK1.
We are looking into the incident.
One of RO1's drives are suspected to be failing.
We are currently taking a full snapshot of RO1's drives onto UK1.
We are looking into the incident.
Investigating on 30 Sep 2026 18:10:22 (UTC+01:00)
RO1 is reporting failures relating to its primary disk. SMART test passed but details are worrying.
We are looking into the issue.
RO1 is reporting failures relating to its primary disk. SMART test passed but details are worrying.
We are looking into the issue.