Outsourced 787 electronics hurt Boeing
The dissolution of Boeing Commercial Electronics offers a case study in how Boeing dealt away in-house expertise.
Itsuo Inouye / Associated Press
Boeing Commercial Airplanes vice president Mike Sinnett with a model of the newly designed 787 battery in Tokyo on March 15.
A National Transportation Safety Board investigator works inside a Boeing 787 Dreamliner airplane under investigation at Boston's Logan Airport on Jan. 8.
Japan Transport Safety Board
The distorted lithium-ion battery (left) and an undamaged auxiliary battery of the All Nippon Airways Boeing 787 that made an emergency landing on Jan. 16 in Takamatsu, Japan.
The All Nippon Airways 787 airliner at Takamatsu airport after it made an emergency landing on Jan. 16.
The company created the unit in the early 1980s because all the systems on a modern jet -- including the electrical, hydraulics, engine, fuel, cabin air and flight-control systems -- are managed by electronics.
"It was a strategic move to control the electronics itself," said Dwight Schaeffer, a former senior manager at Boeing Commercial Electronics.
Yet as Boeing launched the 787 Dreamliner program in 2003, management dispersed all those Everett engineers, outsourced their work, then sold off the Texas plant.
Part of a broad handoff of control to airplane-systems suppliers, the move was intended to cut Boeing's costs.
On jets before the 787, BCE integrated components from many different suppliers so they worked together properly. And if suppliers got in trouble, BCE stepped in and got the job done.
"Now they don't have that capability," said Jerry Packard, another former BCE manager. "That's all lost."
In contrast to Boeing's well-known move to let "global partners" design and manufacture the 787's wings, tail and fuselage, the way it handed design control to 787 systems partners, including management of subcontractors, received little attention at the time.
After this year's costly three-month grounding of the plane from January's battery problems, that approach is getting new scrutiny.
Longtime industry analyst Richard Aboulafia worries it may bring the 787 more grief in future.
"Without complete oversight of the subsystems, they might be finding systems glitches for years," said Aboulafia.
In the aftermath of a rash of 787 systems problems -- in its electrical power-distribution panels and generators as well as its battery system -- the dissolution of Boeing Commercial Electronics offers a case study in how Boeing dealt away in-house expertise and relinquished control over systems suppliers.
BCE designed and built electronic boxes and circuit cards that controlled a multitude of crucial systems on all Boeing planes before the 787.
As a senior manager at BCE, Schaeffer managed the budgets for about 230 employees and ran research and development, business development, project management and product support.
He said BCE's role as an integrator of different subsystems gave it a clear overview of how a jet's systems were coming together.
"No supplier would trust another," said Schaeffer. "But the suppliers would trust us not to give away their secrets."
On the 777 program, for example, Honeywell supplied a system to detect when the weight of the plane was on the landing gear; Allied Signal supplied a smoke-detection system for the cargo hold; Hamilton Standard (which later became Hamilton Sundstrand) supplied an electrical anti-ice system; Fenwal Controls supplied a system to detect leaks in air ducts; Walter Kidde supplied a fire-detection system.
These all fed into and were controlled by a single BCE-designed electronics box, as were other systems designed and built by BCE itself -- including the hydraulics monitoring and the passenger cabin's environmental controls.
To make it all work together, BCE controlled both the physical format of this vital communications nexus, as well as the electrical and software standards for each system.
Another key BCE role was bailing out suppliers that got into trouble.
Lead engineer Ed D'Souza recalls the example of Vickers, a supplier based in Grand Rapids, Mich., of the system for loading and unloading cargo for the 767 freighter.
On previous contracts, Vickers had built only the hardware.
"Failing miserably" at designing the electronic controls, the company appealed to Boeing for help, D'Souza said.
"They only wanted to do the mechanical stuff. They were more than happy to turn the electronics over to us," he said. "We essentially brought the job back in-house and did the entire design and delivery ourselves."
Schaeffer, Packard and D'Souza, now all retired, trace some of Boeing's 787 problems to a loss of control of systems design and the disbanding of BCE.
It's uncommon for retired Boeing managers to speak out publicly against company policies, but they agreed to do so in hope that Boeing will reverse direction for future jet programs.
As Dreamliners entered scheduled service in numbers last year, problems surfaced initially with the 787's electrical system.
At least four times, electrical arcing in circuit boards inside the power panels caused airline-service disruptions, including a Dec. 4 flight diversion by United Airlines.
On Dec. 17, an electrical generator on a United 787 failed after a flight attendant reported a loud bang under the floor.
According to a Federal Aviation Administration incident report, mechanics found a flash mark behind a power panel.
The Wall Street Journal recently cited an internal Boeing report that revealed a total of 350 Dreamliner service disruptions before January. That figure is comparable to the incidence of glitches in the early days of the 777 program, but a person with knowledge of the data said a greater proportion of the 787 problems were electrical.
Then in January, problems with the battery system -- a battery fire on the ground in Boston and a smoldering battery in flight in Japan -- grounded the entire Dreamliner fleet.
Were these "teething problems" or signs of an underlying systems vulnerability?
Boeing insists that the way it outsourced 787 systems was not significantly different from what it's done in the past.
Instead of relying on multiple suppliers sending in pieces Boeing integrated, the jet-maker had its major systems partners "design, build and integrate subsystems," said spokesman Larry Wilson.
"We streamlined our approach," Wilson said.
Mike Sinnett, Boeing senior vice president and 787 chief project engineer, in April told a National Transportation Safety Board investigative hearing into the battery failures that the company maintained both tight oversight and overall control over its systems partners.
Compared with the radically new role for 787 airframe suppliers, Boeing's relationship with 787 systems suppliers was "more traditional," he said.
Some suppliers saw the changes as more radical.
Clay Jones, chief executive of Rockwell Collins, which supplies the 787 cockpit avionics suite, said last year that on the Dreamliner, "Boeing fairly dramatically changed its attitude of how to work with suppliers."
He said Boeing elevated the role of suppliers like his company to that of "true partners."
Some of those suppliers would have liked to work with BCE, said Schaeffer.
Before Boeing selected its systems suppliers for the jet, the Hamilton Sundstrand division of United Technologies approached Schaeffer's group and asked to partner with BCE in making its bid for the electrical systems.
"They were weak on software, and we had helped them out before," Schaeffer said.
But with BCE already on the chopping block, Boeing's leadership nixed that idea.
Boeing ultimately chose Hamilton Sundstrand to design the 787's electrical system and to integrate subsystems such as the power-distribution panels.
The panels, which have proved so troublesome on the 787, were designed in-house on jets before the 787, said Schaeffer.
For the 787 battery system, Boeing selected Thales of France to do the design and integrate subsystems built by subcontractors, including battery-maker GS Yuasa of Japan, and Securaplane of Tucson, Ariz., which made the charging system.
"That would never have happened in the old days," Schaeffer said. The battery-component suppliers "would have been contracted directly with Boeing."
At the NTSB investigative hearing, testimony revealed that information from the subcontractors -- such as the analysis and testing of certain battery failures -- for the most part flowed indirectly to Boeing, through Thales.
"I've no problem with outsourcing the build portions," meaning the making of the hardware, said Schaeffer.
But he said he believes Boeing ought to have kept the design in-house and been responsible for integrating the various components from suppliers into a working system.
"When they outsourced that to the people that make the equipment, like Thales, that's what really broke the back," Schaeffer said.
The 787 is the first Boeing jet with all its electronic components sourced from outside suppliers.
Schaeffer said eliminating BCE from the supply chain meant many subsystem suppliers were "out of sight, out of mind," no longer working directly with Boeing.
"Just by getting rid of us, Boeing outsourced its systems-design responsibilities more on the 787 than on other airplanes," he said.
At the start of the 787 program, Boeing's leadership in Chicago insisted on the need to reduce the development cost by bringing in "risk-sharing partners" who would take on -- and help pay for -- more of the development work.
Hans Weber, a leading aviation-engineering consultant, calls this the "fundamental weakness in the 787 program from the start."
This was "ultimately a financial invention. It's not done for technical reasons," Weber said.
Internal documents obtained by The Seattle Times in the early days of the program showed Boeing planned to spend just $5 billion to develop the 787, a figure Weber calls "unrealistically low" and that later ballooned to at least triple that as production problems mounted.
Once design work had been outsourced, Weber said, an inevitable consequence was that in-house design expertise became surplus and "had to go."
"We know of the problems on the structures side. We are learning more and more on the systems side," said Weber.
"These are problems that could very well be traced to a lack of sufficient oversight and inadequate transfer of corporate-engineering expertise."
Boeing has upgraded both the battery systems and the power panels on the Dreamliners, and airlines around the world expect to resume their full 787 services by June.
If more systems problems arise for Boeing in future, one mitigating factor is that fixing them is generally much cheaper than making major structural changes to the airframe.
Correcting systems faults may require only swapping out an electronics box or making software changes.
More than ever, though, Boeing must look outside the company for that expertise.
Our new comment system is not supported in IE 7. Please upgrade your browser here.