Computer failure may have wiped Mukilteo’s data

MUKILTEO — Planning documents, city financial information, police records, City Council meeting minutes and more were lost when Mukilteo’s computer system crashed last month, officials said.

A little more than 30 percent of the data has been recovered, and it’s uncertain if the rest will ever follow, city officials said.

Most of the lost information is stored on paper, city administrator Joe Hannan said, but city staffers spent 100 hours re-entering the data for December 2011 alone.

The city has already paid two companies about $25,000 to recover the data and might have to pay $20,000 more, depending on how much can be restored, Hannan said.

It’s also faced with spending about $180,000 on a new system. Expanding the city’s information technology staff, which could cost more still, also will be up for discussion.

“I’m interested in moving forward with a more advanced system,” Councilwoman Jennifer Gregerson said.

The event that triggered the meltdown apparently came last July, when the cooling system in the room containing the city’s file servers failed and the room overheated, Hannan said. The hard drives didn’t actually fail until nine months later, however, on April 4.

A few weeks before the crash, the city’s network engineer — one of only two information technology staff members — left for another job, Hannan said, and has yet to be replaced.

Only the information technology director remained, and he was working in the field on a fiber-optic line at the time, Hannan said.

By the time the problem was discovered, it was too late to do prevent the loss of the file servers, he said.

The system had two backups: an immediate, digital copying system and a system that transferred the data to tapes. The tape system broke down in December and wasn’t repaired earlier because officials were relying on the digital copying system, Hannan said.

That system, however, also failed during the crash.

Also, the overall system was not considered advanced enough for a municipal government, Hannan said. For that, Hannan said, he would take responsibility.

“The hard drives we used were not considered as stable or reliable as other systems,” he said.

Upgrading, however, had to be balanced with the cost, Hannan said.

“It’s a choice you make with a lot of items,” he said. “We never thought all these systems would fail.”

Now, the city will have to spend money on a better system anyway, he said.

Gregerson said requests for better equipment and more staff have come from the administration but never were acted on by the City Council.

“For a long time we’ve understaffed and underfunded the IT department,” she said. “I’m disturbed that the decision to cut corners and try to do it cheaper, that those choices were made.”

After the incident, the 32 hard drives were sent to an outside company that, after three weeks and a $5,000 payment from the city, said it could not recover the information.

The hard drives were then sent to another company that has so far recovered more than 30 percent of the total lost, officials said.

This has cost $20,000 and could cost up to another $20,000, Hannan said. He said the city will find out more Tuesday.

Not all the city’s data was lost — some of the tape backups were available, Hannan said. And staff re-entered the missing financial data from late last year to meet a state reporting deadline this month, Hannan said.

The city is waiting to see how much of the other information can be recovered before proceeding, he said.

Police had done their own backup through February of this year, but all the data from March was lost, Gregerson said.

If a home invasion robbery that took place April 30 had happened in March, “we might have lost all of our evidence and, potentially, criminals could go free,” Gregerson said.

The city has a temporary system in place and the tape backup has been repaired, Hannan said.

In addition to a new system, the city also is exploring an agreement for a joint backup system with Lynnwood and Edmonds, Hannan said.

As for whether the staff will be increased, Hannan said that will be up to the council.

The city is scheduled to consider the matter at its meeting June 11.

Bill Sheets: 425-339-3439; sheets@heraldnet.com.