Cobalt Technology - Notice history

Wichita Falls (Western Rural - Arena Rd. Tower) - Operational

100% - uptime
May 2023 · 100.0%Jun · 100.0%Jul · 100.0%
May 2023
Jun 2023
Jul 2023

Wichita Falls (Stadium Tower) - Operational

100% - uptime
May 2023 · 100.0%Jun · 100.0%Jul · 100.0%
May 2023
Jun 2023
Jul 2023

Holliday Tower - Operational

98% - uptime
May 2023 · 92.97%Jun · 100.0%Jul · 100.0%
May 2023
Jun 2023
Jul 2023

Iowa Park ( City Limits) - Operational

97% - uptime
May 2023 · 99.98%Jun · 100.0%Jul · 91.57%
May 2023
Jun 2023
Jul 2023

Iowa Park (Rural-South) - Operational

100% - uptime
May 2023 · 99.98%Jun · 100.0%Jul · 100.0%
May 2023
Jun 2023
Jul 2023

Lakeside City - Operational

100% - uptime
May 2023 · 100.0%Jun · 100.0%Jul · 100.0%
May 2023
Jun 2023
Jul 2023

Datacenter - Operational

100% - uptime
May 2023 · 100.0%Jun · 100.0%Jul · 100.0%
May 2023
Jun 2023
Jul 2023

Phone System - Operational

100% - uptime
May 2023 · 100.0%Jun · 100.0%Jul · 100.0%
May 2023
Jun 2023
Jul 2023

Customer Account Portal - Operational

100% - uptime
May 2023 · 100.0%Jun · 100.0%Jul · 100.0%
May 2023
Jun 2023
Jul 2023

Website - Operational

100% - uptime
May 2023 · 100.0%Jun · 100.0%Jul · 100.0%
May 2023
Jun 2023
Jul 2023

Hwy 79 South Area (Rural) - Operational

100% - uptime
May 2023 · 100.0%Jun · 100.0%Jul · 100.0%
May 2023
Jun 2023
Jul 2023

Notice history

Jul 2023

Iowa Park City Limits Outage
  • Resolved
    Resolved

    This incident has been resolved. All equipment on the Iowa Park City Limits tower has been repaired. If you are still offline, please call or text at 940-463-0040 for individual support.

  • Update
    Update

    All-but 11 customers have been brought back online. We require special equipment to get the last 11 customers back online and expect those 11 to be brought back online Tuesday afternoon.

    We apologize for the inconvenience. If you are one of the 11 affected, we are willing to work with you on your bill for this month. Please contact our office.

  • Update
    Update

    Our crew is on site and beginning work. We hope to have everyone back online within a few hours.

  • Update
    Update

    We have brought roughly half of our Iowa Park City Limits customers back online. We ran into issues with some of our replacement equipment malfunctioning, so we are traveling to get more tonight and will have our crew back at the tower first thing in the morning to finish the repairs.

    We apologize for the inconvenience. Please know we are working as fast as possible to resolve the issues.

  • Update
    Update

    We have replaced a lot of the damaged equipment and we are starting to see some customers come back online. We are continuing to work on the rest.

  • Update
    Update

    Our tower crew is en route from DFW. We expect the site to back up and running today.

  • Identified
    Identified

    Lightning hit our equipment at the top of the tower. We are currently working with our tower crew to schedule a safe time for them to come out and repair the damage once the storm has passed. They are ready to head out today once the weather clears up.

  • Investigating
    Investigating

    We are currently investigating this incident. We will provide updates as soon as we can.

May 2023

Upstream Fiber Cut Causing Degraded Speeds for Wichita Falls, Iowa Park, Lakeside City, and Memorial Stadium areas.
  • Resolved
    Resolved

    The fiber cut has been repaired and speeds are back to normal.

  • Update
    Update

    The upstream fiber provider has given an update. The repair crew has bored the new hole and are about to pull conduit and fiber through it. The location that was damaged affected multiple major Tier 1 carriers at the same time, so crews are working diligently to repair the lines.

    Our network is outside of peak usage times right now, so customers should not se degraded speeds until overall usage begins to pick up around 6 PM. We are expecting connection to be back to full capacity by then, but we cannot know for sure since we are relying on a national ISP to repair the damage.

    These fiber cuts have become exceedingly frequent in the last 6 months so we are working on an internal plan to add capacity to alleviate load during times like these.

  • Identified
    Identified

    One of our upstream fiber providers had a fiber cut between Wichita Falls and DFW. The Wichita Falls, Iowa Park, Lakeside City, and Memorial Stadium networks are currently running on the backup service until the fiber is repaired.

    Holliday is at full speeds and is not affected by the fiber cut.

    Some people have expressed concern about the frequency of fiber cuts, so here is a brief explanation of why it keeps happening.
    All internet connectivity to the greater-Wichita Falls area comes in through fiber optics lines from Lubbock, DFW, or Oklahoma City. Running fiber 200+ miles is extremely expensive, so it is done by the large Tier 1 companies such as AT&T and others. Smaller providers then rent space on the fiber optic lines to get connection to the big city networks.
    There is a great deal of construction going on between Wichita Falls and DFW, and a backhoe or boring crew hitting a line anywhere between the cities will cause a complete outage to anyone down the line, just like hitting a water main will. Many of the crews doing work in DFW are not following proper procedures to call in underground utility locates and are hitting many different utilities underground, including fiber optic lines.

    We protect against this by paying for more than 1 fiber optic line that takes diverse geographical routes to DFW so that the likelihood of a total outage is greatly reduced. That is why the network is able to continue running during a fiber cut.

    We will provide updates as the upstream provider provides updates. It will most likely be repaired during the night.

Storm Damage to Holliday Tower
  • Resolved
    Resolved

    We brought the last bit of tower equipment back online around 11 PM last night. All tower equipment on Holliday and Lakeside City is operational, so if anyone is still without service please call our office at 940-463-0040 or message us on Facebook to get personalized tech support.

    We had multiple out-of-town crews working tirelessly since Saturday night to restore service. We overhauled our grounding and lightning arrestor systems on the Holliday site and replaced a lot of damaged equipment. We added a 3rd redundant link to the tower to help mitigate fiber cuts, and we will also be completing a capacity upgrade on the Holliday tower within the next couple of weeks.

    The last 3 days have been turbulent, but we turned a bad situation into a chance to upgrade and plan for the future! We greatly appreciate everyone's patience, understanding, and loyalty to our locally-owned company during this process.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result. Connection has been restored to nearly all Holliday customers. One Access Point pointing to the Northeast from our tower is still having issues, but everyone else should be back online.

    We will continue to work on restoring the last Access Point and then monitor the network for stability.

    We took this time to drastically increase lightning surge protection to all of our equipment. We will be implementing a capacity upgrade this evening for the Holliday tower to ensure there are no bandwidth constraints.

  • Update
    Update

    Our tower crew has concluded their work and now the remaining work is in the cabinet on the ground. We got partial service back online a few hours ago but were unable to keep it up for long. We are continuing to work on the configurations of the equipment on the ground. Since the previous ETAs have been missed, we will refrain from providing an exact ETA at this time, but our Panhandle crew is almost here to provide assistance in the configurations of the ground equipment.

  • Update
    Update

    Our crews began working at 5:45 AM. The first order of business is to get the last vital piece of equipment online that feeds internet into the site. Once that is brought online, the majority of customers will be back up and we can then work on individual cases of lightning damage.
    We are expecting initial connection to be restored by 9 AM. At this time we know the Access Points pointing toward FM 2224 and Quail Ridge need to be replaced, but that process should not take more than an hour or two after internet is brought back up to the tower.
    We greatly appreciate your understanding.

  • Update
    Update

    The majority of the work is complete, but one vital piece of equipment is still offline. Our DFW tower crew is coming back out very early tomorrow morning along with another crew from the Panhandle. It is unsafe to be on a tower when lightning is anywhere nearby, so we cannot resume work until the storm subsides.

    Our Holliday tower had multiple redundant connections, but this storm has taught us that sometimes the backups need backups. We are taking this time to increase the robustness of our network, upgrade our lightning arrestor systems, and ensure our network is resilient against these sorts of outages in the future.

    We apologize for the inconvenience this outage has caused and we deeply thank you for your loyalty to our locally-owned business. Rest assured this repair is our top priority and it will be all-hands-on-deck very early in the morning.

  • Update
    Update

    The repair crew has been on site since before dawn. Most of the work is finished but we had to call it for safety reasons due to the new storm coming in. We will continue to work on restoring service as quickly and safely as possible. At this point we have to wait for the storm to pass and get a safe distance away before resuming work.

    Getting this tower back online is our number 1 priority. We understand the inconvenience this outage is causing to all of our customers in Holliday and we sincerely apologize for the issues.

  • Update
    Update

    The lightning damage is more extensive than we first expected. An out-of-town tower crew has been called and will be arriving early this morning to assist in the replacement of equipment at the top of the tower.
    Expected restoration time is this afternoon (Sunday, May 7th)

  • Update
    Update

    Based on weather reports it looks like repairs can begin around 3 AM. We will continue to monitor storm conditions and adjust restoration times accordingly.

  • Identified
    Identified

    An employee is on site at the Holliday tower and has determined that it took a direct lightning strike that destroyed a few vital pieces of equipment. This tower has multiple connections coming in for redundancy, and at this time they all appear to be down. We have spares on hand, but cannot climb the tower to replace the units until the storm passes and it is safe to do so.

  • Investigating
    Investigating

    We are investigating the cause of the loss of connection at our Holliday Tower. We will provide an update as soon as possible.

May 2023 to Jul 2023

Next