Cobalt Technology - Storm Damage to Holliday Tower – Incident details

Storm Damage to Holliday Tower

Resolved
Major outage
Started over 1 year agoLasted 2 days

Affected

Holliday Tower

Major outage from 12:11 AM to 4:30 AM

Updates
  • Resolved
    Resolved

    We brought the last bit of tower equipment back online around 11 PM last night. All tower equipment on Holliday and Lakeside City is operational, so if anyone is still without service please call our office at 940-463-0040 or message us on Facebook to get personalized tech support.

    We had multiple out-of-town crews working tirelessly since Saturday night to restore service. We overhauled our grounding and lightning arrestor systems on the Holliday site and replaced a lot of damaged equipment. We added a 3rd redundant link to the tower to help mitigate fiber cuts, and we will also be completing a capacity upgrade on the Holliday tower within the next couple of weeks.

    The last 3 days have been turbulent, but we turned a bad situation into a chance to upgrade and plan for the future! We greatly appreciate everyone's patience, understanding, and loyalty to our locally-owned company during this process.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result. Connection has been restored to nearly all Holliday customers. One Access Point pointing to the Northeast from our tower is still having issues, but everyone else should be back online.

    We will continue to work on restoring the last Access Point and then monitor the network for stability.

    We took this time to drastically increase lightning surge protection to all of our equipment. We will be implementing a capacity upgrade this evening for the Holliday tower to ensure there are no bandwidth constraints.

  • Update
    Update

    Our tower crew has concluded their work and now the remaining work is in the cabinet on the ground. We got partial service back online a few hours ago but were unable to keep it up for long. We are continuing to work on the configurations of the equipment on the ground. Since the previous ETAs have been missed, we will refrain from providing an exact ETA at this time, but our Panhandle crew is almost here to provide assistance in the configurations of the ground equipment.

  • Update
    Update

    Our crews began working at 5:45 AM. The first order of business is to get the last vital piece of equipment online that feeds internet into the site. Once that is brought online, the majority of customers will be back up and we can then work on individual cases of lightning damage.
    We are expecting initial connection to be restored by 9 AM. At this time we know the Access Points pointing toward FM 2224 and Quail Ridge need to be replaced, but that process should not take more than an hour or two after internet is brought back up to the tower.
    We greatly appreciate your understanding.

  • Update
    Update

    The majority of the work is complete, but one vital piece of equipment is still offline. Our DFW tower crew is coming back out very early tomorrow morning along with another crew from the Panhandle. It is unsafe to be on a tower when lightning is anywhere nearby, so we cannot resume work until the storm subsides.

    Our Holliday tower had multiple redundant connections, but this storm has taught us that sometimes the backups need backups. We are taking this time to increase the robustness of our network, upgrade our lightning arrestor systems, and ensure our network is resilient against these sorts of outages in the future.

    We apologize for the inconvenience this outage has caused and we deeply thank you for your loyalty to our locally-owned business. Rest assured this repair is our top priority and it will be all-hands-on-deck very early in the morning.

  • Update
    Update

    The repair crew has been on site since before dawn. Most of the work is finished but we had to call it for safety reasons due to the new storm coming in. We will continue to work on restoring service as quickly and safely as possible. At this point we have to wait for the storm to pass and get a safe distance away before resuming work.

    Getting this tower back online is our number 1 priority. We understand the inconvenience this outage is causing to all of our customers in Holliday and we sincerely apologize for the issues.

  • Update
    Update

    The lightning damage is more extensive than we first expected. An out-of-town tower crew has been called and will be arriving early this morning to assist in the replacement of equipment at the top of the tower.
    Expected restoration time is this afternoon (Sunday, May 7th)

  • Update
    Update

    Based on weather reports it looks like repairs can begin around 3 AM. We will continue to monitor storm conditions and adjust restoration times accordingly.

  • Identified
    Identified

    An employee is on site at the Holliday tower and has determined that it took a direct lightning strike that destroyed a few vital pieces of equipment. This tower has multiple connections coming in for redundancy, and at this time they all appear to be down. We have spares on hand, but cannot climb the tower to replace the units until the storm passes and it is safe to do so.

  • Investigating
    Investigating

    We are investigating the cause of the loss of connection at our Holliday Tower. We will provide an update as soon as possible.