Cobalt Technology - Notice history

This incident has been resolved. A GFCI outlet had tripped during the storm and took down power to the tower. This is not the ideal setup for powering an always-on communications system, so we are working on a new solution to put a self-healing breaker in the circuit instead. This will be done during a future maintenance window in the middle of the night.

This incident has been resolved.

This incident has been resolved. Earlier today we implemented a change in our core Dallas datacenter network that was intended to increase redundancy in the network, but it inadvertently injected duplicate routes in the network and caused traffic to get disrupted. We worked throughout the afternoon and evening to resolve the issues and we believe we have restored stability to the network. This issue came as a result of our overarching plan to bring a higher level of redundancy and resiliency to our network to protect ourselves from future outages. We will continue to implement carrier-grade practices in our network to increase stability as we grow.

We have implemented a fix that appears to have stabilized the network. We are going to keep an eye on it to see if the problem creeps back in. We are leaving the incident open for now and categorizing it as Degraded Performance until we are confident the issue has been resolved.

Connectivity in our Dallas datacenter network appears to be cutting in and out. We are in talks with our network engineers to implement a fix.

This incident has been resolved. We climbed the tower and replaced the faulty equipment with a spare we had on hand.

Our crew is about to climb the tower to replace the faulty equipment. Estimated restoration time: 9 PM

This incident has been resolved. All repairs on the tower are complete. If anyone is still out, please check that everything is plugged into the router properly in your house. If you still have issues, please contact us for troubleshooting on an individual basis.

Repairs are being made and customers are beginning to come back up. We will continue our work until everyone is back up.

The out-of-town tower crew arrived on site a few minutes ago and is about to begin their work. Estimated restoration time is 5 PM.

Tower crews are busy around the state because of the storm. We found a crew that can come out and replace the damaged equipment. Current estimated restoration time: 5 PM Today.

Equipment on our tower took a direct lightning strike during the storm last night. We are working with our tower climbing crew to get them out to replace the equipment.

This incident has been resolved. Power has been restored.

The outage has been resolved. Another fiber cut happened in the DFW area and our failover system did not take affect immediately like it was supposed to. We have remedied the situation and the entire network is back up and running and full capacity.

The source of the issue has been located and resolved. One of our many upstream providers lost connection to our datacenter early this morning. Our system was running through other upstream providers, but a lot of traffic was still trying to navigate through the upstream connection that was having issues.
The connection has been restored and traffic is flowing to all destinations now.