Investigators say a single absentminded technician caused a 90-minute loss of power to the center that directs all of the planes over southern Vietnam, last Thursday.
The Vietnam Air Traffic Management Corporation released a statement saying that Le Tri Tinh, chief technician in charge of the power supply at the Ho Chi Minh City Area Control Center, caused a malfunction to the center's uninterruptible power supply (UPS) devices.
The center's power is routed through the UPS system specifically to a prevent sudden blackout.
Most UPS devices can only maintain power for a relatively short period of time. They are designed, however, to allow users time to fire up a standby power source or properly shut down the protected equipment.
Blackout by backup
UPS backups are typically used in hospitals to keep life support systems from shutting down or in data centers to prevent the loss of information.
The recent post-incident inspection found that Tinh failed to safely isolate a defective UPS device, which led to the collapse of the entire system, blacking out the center and affecting 92 flights. The center had to cede control over 54 aircraft in HCMC’s Flight Information Region (FIR) to the Hanoi center.
Pilots were told to circle Tan Son Nhat or land at nearby airports, while departing flights were delayed by as much as five hours.
Tan Son Nhat’s technicians are supposed to check the UPS systems twice a week.
On Thursday, they discovered it had broken down.
The team was supposed to remove the individual UPS for repairs, instead of cutting power to the whole system.
Tinh, the chief technician did the latter, causing the blackout.
After his repeated efforts to restore power failed, the team plugged the air traffic control system directly into the main power source.
Tinh and his subordinate Pham Van Dung were suspended last Friday for 15 days pending an investigation into the specific roles they played.
On Sunday, the Vietnam Air Traffic Management Corporation also suspended three high-ranking officials at the Southern Air Traffic Management Company for the incident: Tran Cong, deputy director of the company’s technical department, Le Van Tinh, head of the company’s technological support center, and Nguyen Quoc Phu, the center’s deputy head.
What happened to bypass devices?
Nguyen Ngoc Lam, deputy director of the network management center for FPT Telecom in Hanoi said his company also uses UPS devices but they employ bypass devices to divert electrical currents back to the main power source in case a UPS “dies.”
Lam said a bypass device takes about a minute to set up and Tan Son Nhat must have only suffered such a long blackout because it hadn't taken that rudimentary precaution.
Do Hoang Diep, director of the Southern Air Traffic Management Company, told Thanh Nien that “the power system at the airport has bypass devices" without explaining why they didn't kick into effect last Thursday.
Diep said he has been on medical leave for the past ten days and was not fully updated on the matter.