OLIVE TOWNSHIP, Mich. (WOOD) — Katie Coenen was supervising the floor at the Ottawa County Central Dispatch Authority Tuesday afternoon when a call came in. On the other end of the line was another 911 operator from a neighboring county.

“Allegan called us and said a couple of our phone lines were ringing down to them. Generally, that does only happen when it’s too busy here. It wasn’t super busy at the time,” Coenen said.

Another Ottawa dispatcher tried 911. That went to Allegan as well.

The system was down. As many as five calls didn’t get through before the problem was discovered.

Fortunately, Ottawa uses a third-party provider outside of the normal 911 system that gives dispatchers callback numbers even when the main system fails.

“We have a great crew here. They adapted very well. I was giving them phone numbers to call back,” Coenen said. “Even when we’re taking 911 call, it changes every couple of seconds. So you’re adapting to the 911 calls. You’re just doing it one the technology side.”

It was a scene that played out in dispatch centers across the state Tuesday afternoon. Several dispatch centers were hit with the outage, including the ones for Calhoun, Kalamazoo and Newaygo counties and the city of Grand Rapids. The outage was traced to Marquette-based Peninsula Fiber Network, the telecom that provides voice, data and other services under state contract to 136 dispatch centers in Michigan. There is a redundancy system, but the so-called data storm caused by the original failure knocked out the back-up.

In Ottawa County, the problem was fixed within about an hour. Other 911 systems were back to normal by 5 p.m. Tuesday.

Peninsula’s system provides enhanced data sharing and connectivity to 911 centers statewide.

“We are lightyears ahead of other states when it comes to that type of thing,” Ottawa County Central Dispatch Authority Executive Director Pete McWatters said.

But when it goes down, there can be a ripple effect.

“That definitely is where the problem has taken place the couple of different times with equipment that has caused what they call a data storm, which the impact, this time, I understand, is pretty statewide,” McWatters said.

Peninsula’s general manager sent News 8 a statement saying it was “conducting a technical evaluation of the cause of the network event and will make that report known upon completion to the Michigan counties that were impacted.”

McWatters said state organizations, like the Michigan Communication Directors Association and the Michigan 911 Committee, are also investigating what happened.

“It’s a significant issue that needs to be addressed and corrected,” McWatters said. “But I’m also confident that that’s going to take place.”

In a news release Friday, Peninsula Fiber Network said it’s “investing $6 million to expedite the redesign of its network and install new optical transport network equipment that will improve the resilience of its 911 system.”

It expects to complete work over the next several months.

PFN released the following statement Friday morning:

“Peninsula Fiber Network experienced a system error on Jan. 10 that impacted service to 911 dispatch centers across the state. An error occurred in the company’s optical transport network resulting in database corruption. This caused a hardware/software mismatch resulting in some 911 calls to not be completed as expected or calls that lacked critical caller and address information.

“PFN technicians discovered a disruption in the company’s optical transport service at roughly 3:15 pm on Jan. 10th. The redundancy mechanism that supports the network then malfunctioned causing statewide disruptions. PFN staff was able to set up a call bridge with 911 dispatch centers across the state within 20 minutes to troubleshoot the problem and ensure calls were rerouted while the equipment was being serviced.

“We take our mission-critical role in delivering consistent and reliable service to Michigan’s 911 operators very seriously and build multiple redundancies into our network. We also work with dispatch center managers to ensure processes are in place to reroute calls between centers when unexpected issues occur,” said Peninsula Fiber Network General Manager Scott Randall. “Despite our best preparations, some calls were still disrupted and for that we are deeply sorry. We are now making several upgrades to the whole system to eliminate the possibility of additional network errors in the future.”

“PFN is investing $6 million to expedite the redesign of its network and install new optical transport network equipment that will improve the resilience of its 911 system. This work will be completed over the next several months. A final completion date has not been determined at this time.”

Peninsula Fiber Network