Services Restored After Nearly 14-Hour Infrastructure Outage

All external and internal CSLRP services are now fully operational again following an infrastructure outage that lasted nearly 14 hours.

We understand that this was a significant period of downtime, and we would like to explain what happened, why restoring the services took longer than expected, and how the issue was ultimately resolved.

What Happened?

The outage began when the internet service provider unexpectedly assigned and linked a new IP address to the server connection.

Because the server and its services were still configured to use the previous network information, external access was immediately lost. This affected the main CSLRP server, development services, websites, SSH access, and several other systems hosted at the same location.

The newly assigned IP address could only be identified from within the local network. This meant that the issue could not be fully diagnosed or corrected remotely.

Why Did It Take Nearly 14 Hours?

Resolving the outage required someone to physically attend the server location and access the local network.

Unfortunately, an available technician could not reach the location until the following day. Until physical access was available, we were unable to confirm the new network details or apply the required configuration changes.

Once the technician arrived, the cause was identified and the necessary network configuration was updated. Connectivity gradually returned, although several internal services initially experienced additional licensing and configuration errors as a result of the IP address change.

These remaining issues were investigated and corrected before the incident was marked as fully resolved.

All Services Restored

All public-facing services are now operating normally, and the remaining internal systems have also been restored.

Previously planned server updates were postponed during the incident so that all attention could remain focused on restoring the existing infrastructure safely. These updates will instead be completed at a later date.

We will continue reviewing our network configuration and recovery procedures to reduce the impact of similar incidents in the future.

Thank You for Your Patience

We apologise for the extended downtime and any inconvenience it may have caused.

Thank you to everyone for your patience and understanding while the issue was being investigated, and thank you to the technician who attended the location and helped bring our infrastructure back online.

— The Cross State Line RP Team