from Hacker News

After Alaska Airlines planes bump runway, a scramble to ‘pull the plug’

by dlgeek on 2/18/23, 3:25 PM with 183 comments

by neonate on 2/18/23, 3:49 PM
https://archive.md/XOgpi
by Aloha on 2/18/23, 3:41 PM
This seems like mostly how it ought to work -
* There were two unusual events in very short order
* Someone quickly noticed and gave the order to go to ground stop
* The problem was figured out quickly and a work-around was developed
* Flights Resumed after successfully deploying the work around to the 'production process'
* A patch was quickly developed and deployed once the underlying bug was uncovered.
I think everyone (but perhaps the developers) comes out looking like a champ.
Now, do I think telling pilots what thrust they should use to take off is wise? I dont know - I'm not a pilot, and I'd want pilots feelings on the matter before commenting further, but it feels to me like a loss of authority that would make me uncomfortable - particularly if I was held to account for undetected failures.
The other part that I'd note here, is process is just as important as software could process alone have caught this failure without the tail strikes? possibly, and its something worth looking into further - but only if it doesn't add an unacceptable workload burden to pilots workload that could otherwise compromise safety further.
by sudhirj on 2/18/23, 3:53 PM
> Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system.
That seems horribly wrong to me. I can understand software being slow under load, but being wrong under load sounds like a horrible internal architecture problem.
by 015a on 2/18/23, 4:44 PM
> Bret Peyton, Alaska’s on-duty director of operations, immediately ordered no more planes were to take off across the airline’s network. All Alaska flights not already airborne were stopped nationwide.
An absolute pro. There's a hundred variations of this story, to varying degrees of criticality and impact; seeing a pattern out of two data points, connecting the dots, making the tremendous call to immediately pull the plug, to stop the world and give engineering time, then diagnosing and triaging the problem in less than a half hour; that's world class reliability engineering.
by fwlr on 2/18/23, 4:01 PM
“the bug was missed because it only presented when many aircraft at the same time were using the system.”
The system reports data on number of passengers, weight of cargo, plane balance, etc., to the pilots. The calculation is done by the plane’s flight computer. How can it be off by 20,000 pounds, but only under heavy server load?
The explanation that comes to mind is that DynamicSource has a subservice for each source of weight and one of those subservices crashed under heavy usage. So the top-level aggregate-and-report service got an error from one subservice and said “well, guess it’s zero, lol”?
by Tempest1981 on 2/18/23, 3:53 PM
Curious about the actual bug:
> the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system
> the data was on the order of 20,000 to 30,000 pounds light. With the total weight of those jets at 150,000 to 170,000 pounds, the error was enough to skew the engine thrust and speed settings.
Multithreading/contention issue? But how would that alter the weights?
> The software code was permanently repaired about five hours later
That's surprisingly fast, isn't it?
by hodgesrm on 2/18/23, 3:48 PM
That was an impressively quick call based on ability to reason about unexpected behavior of the system. It should give pause to anyone who wants to take the human out of the loop in safety critical software.
by OliverJones on 2/18/23, 7:16 PM
A weight discrepancy of ten to fifteen TONS????
That's no small mistake. It's not a rounding error on the weight of the soda cans loaded into the galley for the flight.
That's most of the passengers. It's most of the freight. It's heavy enough to be a spare engine in the hold. It's a goodsize fraction of the fuel needed to fly from Seattle to Honolulu. It's certainly enough to foul up the mandatory weight-and-balance computation the pilot in command is required to do.
Somehow the input to this software package missed something big. It would be interesting to know exactly what was missed.
by konschubert on 2/19/23, 4:45 PM
> Still, the mishaps point to the need for more vigilance by pilots in checking automated data.
No. You don’t get to build complex configurations that only work with supporting software and then BLAME THE USER when the software fails.
“Ah, but you didn’t follow rule 137 in the 2000-page manual that nobody reads because computers made it obsolete” is a deflection of responsibility from the owners to the operators
and it’s not okay.
by ldarby on 2/19/23, 7:53 PM
> If the automatically loaded data strikes either pilot as not right, they can make a manual request for takeoff data from the airline operations center. “But 99.8% of the time, the data is accurate,” he said.
> Alaska’s Peyton said “several crews noticed the error and notified dispatch.”
> The pilot at American Airlines said “requesting manual data is not standard” and that if there’s a glitch, naturally some pilot somewhere is going to miss it.
> “Not everyone gets eight hours sleep the night before. Someone is going through a divorce. Someone is not so sharp that morning,” he said. “The sanity check isn’t perfect every day of the week.”
So they already knew (or should have known) about this glitch, and did nothing about it until it started causing damage to aircraft. I would have thought the pilots who reported it earlier would have also predicted the outcome and got it fixed then. From my lay perspective, it's obvious that if aircraft is heavier than the flight computer was told it is, then something bad is going to happen (e.g. could have been complete failure to take off and crash at the end of the runway).
Well, maybe such incidents where pilots do make the prediction, and get it fixed before it causes the problem they predicted, never make it to the news...
by sithadmin on 2/18/23, 5:00 PM
737-900s are notoriously tail-heavy. Some airlines require that they use tail stands when parked, and some airlines (like United) have taken measures to redistribute weight forward of the rear landing gear (by moving one of the economy lavatories so that it sits directly behind first class/in front of most economy seating).
by roamerz on 2/18/23, 4:32 PM
Seems like any modern jet would have a load sensing system built into the landing gear. Something so critical such as takeoff weight would be both estimated by a software system and compared to an onboard system for redundancy. Any discrepancy beyond a set limit would turn on a warning light.
by bogomipz on 2/18/23, 6:14 PM
The article states the following about the software:
>"It delivers a message to the cockpit with crucial weight and balance data, including how many people are on board, the jet’s empty and gross weight and the position of its center of gravity.
>In a cockpit check before takeoff, this data is entered into the flight computer to determine how much thrust the engines will provide and at what speed the jet will be ready to lift off."
Given that overhead bins are regularly maxed out with carry-on luggage now since airlines began charging for checked bags, how are they able to accurately account for the weight and balance? Airlines seem to almost never weigh customers carry-on at check in.
by ww520 on 2/18/23, 5:28 PM
If weight calculation is so crucial for take off, it seems like redundancy of the weight calculation is needed - a different weight calculation from a different company using a different method to verify against the main one.
by jcutrell on 2/18/23, 4:26 PM
Seems like something as critical as thrust in this phase of flight should have some kind of cross-check validation. Sounds like a bad single point of failure to me, but maybe there is a cross check I don’t know about that isn’t covered in the story.
What I do know is that with something like this, a little could go a long way. I wonder what the inspection and repair for a tail strike is, and whether that cancels out the money saved by minimum viable thrust across the fleet. I’m sure someone is punching the calculator on this to determine that.
by Max_Horstmann on 2/18/23, 4:40 PM
> The Alaska captain said that, as for many things in aviation, pilots routinely use an acronym when they do the pre-takeoff “sanity check”: TLAR, which means “That Looks About Right.”
Aviation's LGTM
by Merad on 2/18/23, 4:32 PM
I was under the impression that SOP for most aircraft was to take off at full power, and if you happen to be lightly loaded you just get into the air faster. Was I wrong, or is this a recent change? If so, it seems like a concerning way for airlines to cut costs. Having excess power available at take off means there's a greater margin for error to handle emergencies (engine failure, bird strike, etc.).
by janalsncm on 2/19/23, 10:33 AM
Dumb question. A heavier aircraft should be slower to accelerate. Given that these planes were at least 13% heavier than was thought and the force applied during takeoff is known, shouldn’t there have been some kind of alarm? “We think you should be going 120 mph but you’re actually going 100 mph”.
And based on the inferred mass, can’t you directly calculate what the takeoff velocity should be?
by drewg123 on 2/18/23, 5:51 PM
I don't understand why an airplane being heavier than what was estimated by the software would cause the tail to drag. I would assume it would cause the plane to not have sufficient thrust to take off, or to use more runway. But having the tail drag makes it seem like it had too much thrust, which translated to a steeper take off than anticipated... Can somebody explain?
by Animats on 2/18/23, 7:06 PM
"Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system."
Now that's strange. Anyone have more details? Why should there be any connection between the calculations for different aircraft?
by nop_slide on 2/18/23, 4:14 PM
This other article linked near the end of the source article is even sketchier!
It basically boils down to “this flight started nosediving, almost hit the ocean, and we don’t know why”.
https://www.nytimes.com/2023/02/13/us/united-maui-flight.htm...
by ris on 2/19/23, 1:31 PM
It's a bit of a shame the way journalism works this story will do the rounds with the headlines associating "Alaska Airlines" with "tailstrikes" ("takeoff incidents", whatever term gets used), while actually it looks like Alaska Airlines were the heroes of this story.
by dehrmann on 2/19/23, 2:17 AM
> Flights to Hawaii are typically full, with lots of baggage and a full load of fuel for the trip across the ocean. The planes are heavy.
Interesting that baggage weight changes by destination. I'm sure passenger weight does too, but for different reasons.
by batch12 on 2/18/23, 6:41 PM
This would be an effective side channel attack to use against one's enemies. Even if the true source was never known, the chaos and distrust in infrastructure after the crash would be nice second-order effects.
by mike_hock on 2/19/23, 9:37 AM
\> open website and start reading
\> an ad (or whatever, didn't read it) popup appears
\> close popup, try to start reading again
\> another popup appears and blurs the article
\> close website, I wasn't that interested anyway
by lowbloodsugar on 2/18/23, 5:26 PM
My fucking washing machine can sense how big a load is in it. Surely the plane could measure how fast it is actually accelerating given its thrust and thus derive its actual mass. If it doesn’t match what it’s been told it should alarm.
Also shocking that the system gives an incorrect answer when it is under load. Lack of load testing is a problem, sure, but an architecture that allows the answer to just be wrong is fundamentally flawed.
by signal_space on 2/18/23, 5:57 PM
calculation of total weight ends up being a pde right?
by NotYourLawyer on 2/18/23, 4:09 PM
It’s software. It’s made of bugs.
by orasis on 2/18/23, 7:36 PM
Why is there so much aviation disaster porn on here recently?
This is turning into the same sort of high drama news site that most of us are trying to get away from.
by 1101010010 on 2/18/23, 4:03 PM
Lots of blame to pass on the software, but the real culprit is spelled out clearly in the middle of the article:
> the computer then calculates just the right amount of engine thrust so the pilots don’t use more than necessary. “The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance.
This is not an accident, but rather a feature of capitalism, this time it is human lives that are commodified and have costs externalized unto.
by justinclift on 2/18/23, 3:56 PM
With Boeing's recent 737 family history (MAX fiasco, etc), it's very concerning to see "software errors" still happening in operation. :(