Space

ALL SYSTEMS GO

HOW TROUBLESHOOTERS RESCUED THE ‘FROZEN’ ARM

Danylo Hawaleshka July 16 2001
Space

ALL SYSTEMS GO

HOW TROUBLESHOOTERS RESCUED THE ‘FROZEN’ ARM

Danylo Hawaleshka July 16 2001

ALL SYSTEMS GO

Space

The moment when John Lymer’s pager went off on May 17 is etched in his memory. The chief engineer for Canadarm2, Canada’s vital contribution to the international space program, was at work at the Brampton, Ont., headquarters of MacDonald Dettwiler Space and Advanced Robotics Ltd., where the sophisticated robotic arm was born. It had been just over two weeks since Canadian astronaut Chris Hadfield and his crewmates aboard the shutde Endeavour successfully mounted Canadarm2 to the side of the International Space Station, and it was working fine. Until that fateful day. The station’s crew had Canadarm2 outstretched, testing every joint by rotating them at top speed. Without warning, the robotic limb froze. NASA’s response: call Lymer. And in Brampton, Lymer’s first thought on hearing about the

failure: “Oh no, it’s only been two weeks.” With equipment that complicated, failures are to be expected over time. That it happened so soon in the arm’s 20-year lifespan, however, left Lymer shaking his head in disbelief. The stakes were enormous— the entire multibilliondollar space station project depends on the arm working properly. With Canada’s technological pride and joy on the fritz, NASA had to postpone the next two shuttle missions. For Lymer and his team, it was imperative they find the problem—fast—and fix it. As the disastrous news spread, 20 engineers and software experts at Canadian-based MD Robotics (formerly Spar Space Robotics) began working 15-hour days, seven days a

week, on staggered shifts around the clock. Lymer sped to NASA’s Mission Control outside Houston.

First, the troubleshooters traced the problem to the arm’s backup control system and a 78-pin computer chip—the size of a matchbook—in the shoulder. A builtin test circuit appeared to conclude that the data the chip was sending to the arm’s main computer were faulty, so it was instructing the computer intermittendy to shut everything down. Next, the team sent software modifications to the arm via NASA’s radio linkup with the space station. On June 6, it instructed the arm to save and transmit the data the next time the elusive failure occurred. “The computer, instead of just shutting down,” says Lymer, “would burp everything it knows about itself.”

To the technicians’ great relief, further software modifications confirmed that the chip was simply interpreting data incorrecdy, that in fact there was nothing wrong with the arm. That ruled out the worst-case scenario: an expensive mission to replace a faulty joint or computer. By June 7, they had the arm operating again. On June 26—40 days after the failure— a final version of a software patch instructing the arm’s system to ignore the false alarms was in place. Three days later, NASA gave the next shuttle mission, by then a month late, the green light for a launch this week. It will carry an airlock to be used by station crew members making space walks. All eyes will be on Canadarm2 as it faces its first real test: lifting the airlock into position.

Keeping in touch with the repair effort, Hadfield says he quickly became confident the failure wasn’t mechanical, and that a software change would fix things. “If that’s our only problem in a system as complicated as this,” he said, “we’ve done superb engineering and development to be able to deliver a product that good.” Lymer celebrated by taking the Canada Day weekend off to lie on a dock near Bancroft in central Ontario and enjoy some frosty beer. “Talk about relief,” he said, savouring his first free weekend since May 17.

Danylo Hawaleshka

How TROUBLESHOOTERS RESCUED THE ‘FROZEN’ ARM