Cobalt Technology - Notice history

The fiber cut has been repaired and speeds are back to normal.

The upstream fiber provider has given an update. The repair crew has bored the new hole and are about to pull conduit and fiber through it. The location that was damaged affected multiple major Tier 1 carriers at the same time, so crews are working diligently to repair the lines.

Our network is outside of peak usage times right now, so customers should not se degraded speeds until overall usage begins to pick up around 6 PM. We are expecting connection to be back to full capacity by then, but we cannot know for sure since we are relying on a national ISP to repair the damage.

These fiber cuts have become exceedingly frequent in the last 6 months so we are working on an internal plan to add capacity to alleviate load during times like these.

This incident has been resolved. The brand new primary link has been installed and all traffic has been moved over to it. This should fix the speed issues. The new primary link is not affected by rain, so there should be no degradation during storms.

The tower regained connectivity after a few minutes.

This incident has been resolved. A fiber module in the main link on the tower was faulty. We replaced the module and the main link is back at full speed.

Later this next week we will be installing a brand new main link to the Holliday tower that will greatly increase capacity and stability. We have ide tified the weak spots in our network design that keep allowing outages to happen and we are working to fix them.

We brought the last bit of tower equipment back online around 11 PM last night. All tower equipment on Holliday and Lakeside City is operational, so if anyone is still without service please call our office at 940-463-0040 or message us on Facebook to get personalized tech support.

We had multiple out-of-town crews working tirelessly since Saturday night to restore service. We overhauled our grounding and lightning arrestor systems on the Holliday site and replaced a lot of damaged equipment. We added a 3rd redundant link to the tower to help mitigate fiber cuts, and we will also be completing a capacity upgrade on the Holliday tower within the next couple of weeks.

The last 3 days have been turbulent, but we turned a bad situation into a chance to upgrade and plan for the future! We greatly appreciate everyone's patience, understanding, and loyalty to our locally-owned company during this process.

We implemented a fix and are currently monitoring the result. Connection has been restored to nearly all Holliday customers. One Access Point pointing to the Northeast from our tower is still having issues, but everyone else should be back online.

We will continue to work on restoring the last Access Point and then monitor the network for stability.

We took this time to drastically increase lightning surge protection to all of our equipment. We will be implementing a capacity upgrade this evening for the Holliday tower to ensure there are no bandwidth constraints.

Our tower crew has concluded their work and now the remaining work is in the cabinet on the ground. We got partial service back online a few hours ago but were unable to keep it up for long. We are continuing to work on the configurations of the equipment on the ground. Since the previous ETAs have been missed, we will refrain from providing an exact ETA at this time, but our Panhandle crew is almost here to provide assistance in the configurations of the ground equipment.

Our crews began working at 5:45 AM. The first order of business is to get the last vital piece of equipment online that feeds internet into the site. Once that is brought online, the majority of customers will be back up and we can then work on individual cases of lightning damage.
We are expecting initial connection to be restored by 9 AM. At this time we know the Access Points pointing toward FM 2224 and Quail Ridge need to be replaced, but that process should not take more than an hour or two after internet is brought back up to the tower.
We greatly appreciate your understanding.

The majority of the work is complete, but one vital piece of equipment is still offline. Our DFW tower crew is coming back out very early tomorrow morning along with another crew from the Panhandle. It is unsafe to be on a tower when lightning is anywhere nearby, so we cannot resume work until the storm subsides.

Our Holliday tower had multiple redundant connections, but this storm has taught us that sometimes the backups need backups. We are taking this time to increase the robustness of our network, upgrade our lightning arrestor systems, and ensure our network is resilient against these sorts of outages in the future.

We apologize for the inconvenience this outage has caused and we deeply thank you for your loyalty to our locally-owned business. Rest assured this repair is our top priority and it will be all-hands-on-deck very early in the morning.

The repair crew has been on site since before dawn. Most of the work is finished but we had to call it for safety reasons due to the new storm coming in. We will continue to work on restoring service as quickly and safely as possible. At this point we have to wait for the storm to pass and get a safe distance away before resuming work.

Getting this tower back online is our number 1 priority. We understand the inconvenience this outage is causing to all of our customers in Holliday and we sincerely apologize for the issues.

The lightning damage is more extensive than we first expected. An out-of-town tower crew has been called and will be arriving early this morning to assist in the replacement of equipment at the top of the tower.
Expected restoration time is this afternoon (Sunday, May 7th)

Based on weather reports it looks like repairs can begin around 3 AM. We will continue to monitor storm conditions and adjust restoration times accordingly.

An employee is on site at the Holliday tower and has determined that it took a direct lightning strike that destroyed a few vital pieces of equipment. This tower has multiple connections coming in for redundancy, and at this time they all appear to be down. We have spares on hand, but cannot climb the tower to replace the units until the storm passes and it is safe to do so.

This incident has been resolved. A GFCI outlet had tripped during the storm and took down power to the tower. This is not the ideal setup for powering an always-on communications system, so we are working on a new solution to put a self-healing breaker in the circuit instead. This will be done during a future maintenance window in the middle of the night.

This incident has been resolved.

This incident has been resolved. Earlier today we implemented a change in our core Dallas datacenter network that was intended to increase redundancy in the network, but it inadvertently injected duplicate routes in the network and caused traffic to get disrupted. We worked throughout the afternoon and evening to resolve the issues and we believe we have restored stability to the network. This issue came as a result of our overarching plan to bring a higher level of redundancy and resiliency to our network to protect ourselves from future outages. We will continue to implement carrier-grade practices in our network to increase stability as we grow.

We have implemented a fix that appears to have stabilized the network. We are going to keep an eye on it to see if the problem creeps back in. We are leaving the incident open for now and categorizing it as Degraded Performance until we are confident the issue has been resolved.

Connectivity in our Dallas datacenter network appears to be cutting in and out. We are in talks with our network engineers to implement a fix.

This incident has been resolved. We climbed the tower and replaced the faulty equipment with a spare we had on hand.

Our crew is about to climb the tower to replace the faulty equipment. Estimated restoration time: 9 PM

This incident has been resolved. All repairs on the tower are complete. If anyone is still out, please check that everything is plugged into the router properly in your house. If you still have issues, please contact us for troubleshooting on an individual basis.

Repairs are being made and customers are beginning to come back up. We will continue our work until everyone is back up.

The out-of-town tower crew arrived on site a few minutes ago and is about to begin their work. Estimated restoration time is 5 PM.

Tower crews are busy around the state because of the storm. We found a crew that can come out and replace the damaged equipment. Current estimated restoration time: 5 PM Today.

Equipment on our tower took a direct lightning strike during the storm last night. We are working with our tower climbing crew to get them out to replace the equipment.

This incident has been resolved. Power has been restored.