Fiber cut along North Shore exposes vulnerabilities in tech-dependent world

Jacob Muus first noticed something was amiss late Friday morning when a customer tried to use a gift card.

"And I was kind of stumped by why it wasn't going through,” recalled Muus, a co-owner of Johnson's Foods grocery store in Grand Marais. “I tried using different cash registers and seeing if it would work on those, but it didn't."

Muus quickly learned the store had lost its internet service. So he made the decision to stop accepting credit cards, because while his system is set up to store card information, he had no way to confirm whether a customer's account was in good standing. 

Some customers wrote checks, but since ATMs in town also didn't work, many tourists "were in a tough bind if they didn't have enough cash on hand."

Farther up the North Shore of Lake Superior, the Grand Portage Band of Lake Superior Chippewa also could only accept cash and checks at its casino, convenience store and trading post for a six-hour stretch during one of the final Fridays of the busy summer tourism season. 

“We were pretty much dead in the water here,” said chairman Robert Deschampe, who added that the band is working to tally how much business it potentially lost.

The local hospital was unable to send radiology images to physicians in the Twin Cities for review, said Cook County North Shore Hospital administrator Kimber Wraalstad.

And while Cook County’s 911 service remained functional, few people could call it, because cell phone service had also been cut, along with voice over internet providers for the homes and businesses that use that technology for phone service. 

That became an issue at Johnson’s Foods, Muus said, when some people got into a heated argument in the parking lot. 

“And so they wanted me to call the sheriff and I said, ‘Well, the landlines are down. The cell phones are down, [the] internet's down.’” He told them there was nothing he could do. “Just a lot of headaches that day.”

‘A scary thing’

Those headaches began at 11:19 a.m. on Friday morning, when a company installing an underground cable about three miles north of Silver Bay accidentally cut the main fiber optic trunk line that runs from Duluth up the North Shore. 

That line is owned by the Northeast Service Cooperative, or NESC, a nonprofit public corporation established by the state legislature that operates a 1,200-mile fiber optic network across northeastern Minnesota.

Jon Loeffen, who oversees that network for NESC, said it’s still unclear who cut the line. What is clear, he said, is that whoever it was broke the law by failing to properly notify existing utilities through Gopher State One Call that they were planning to dig in the area.

“That was a terrible place for us to get hit because it was a main artery to our network,” Loeffen said. 

And that fiber optic line doesn’t just provide internet service. It provides fiber connectivity between cell towers and carriers like AT&T and Verizon. So when the line was cut Friday, it also disrupted cell phone service. 

“If there had been a storm, or some kind of a large crash, and people weren't able to report right away … I mean, time is survival in a lot of those situations,” said Grand Marais Mayor Jay DeCoux. “So it really is a scary thing.”

DeCoux, who also manages emergency communications for Cook County, said he and other local officials are putting pressure on NESC and other entities to ensure something like this doesn’t happen again. 

“Because knowing the background of how the system works, this didn't have to happen,” DeCoux said. “We got really lucky.”

For years there was only a single fiber optic line that linked the North Shore to the Duluth area. Any disruption to that line would often lead to widespread outages. 

A few years ago NESC installed a second line to create a more diverse system with redundancy built into it, so that if one line went down, traffic could be diverted to the other line almost instantaneously. 

But this incident exposed holes in the system. The cut to the fiber line a few miles north of Silver Bay occurred “in an area that did not have that redundancy built in on that particular segment,” said Loeffen. 

It took about six hours to switch most internet and cell phone service over to that redundant line. But it took AT&T nearly 24 hours to restore its service, DeCoux said.

NESC is now putting together a plan to install additional equipment and reconfigure its network links up the North Shore, “so that hopefully in the future, if something like that occurs again, there won't be an outage for some of those services because they're going to be on a protected, diverse path,” Loeffen said.

The goal, he said, is to complete that work in less than a month.

Crash kits

Cook County, on the far northeastern tip of Minnesota’s Arrowhead region, is used to utility disruptions. 

911 service goes down often enough that the county has established a work-around with its nine volunteer fire departments, which act as hubs where people can request needed services. 

Wildfires and windstorms on the Gunflint Trail have led to power and phone outages that sometimes stretch for weeks.

That’s why Mike Prom, CEO of Voyageur Brewing Company in Grand Marais, has created what he calls a “crash kit” that employees train on twice a year in case of internet or power outages. 

It includes instructions to help manually calculate different tax rates, and old-fashioned guest checks with carbon copy receipts. 

Prom said during last week’s outage a lot of customers didn’t want to provide their phone numbers in case there was a problem with their credit cards. But he said most people were ecstatic that they remained open. 

“Maybe we're a little more tolerant, because of where we live, and you know that's kind of why we live where we live, is just the wilderness aspect of being here.”

“It's one of the vulnerabilities we have being located where we are,” said Sara McManus, member services manager at Arrowhead Cooperative, which provides broadband internet service to 3,500 customers throughout Cook County. 

“Folks up here are incredibly patient and incredibly tolerant. And we're just so grateful for their grace as we work through this.”