Posts about: "FADEC" [Posts: 231 Pages: 12]

Interesting and informative post from JustusW on 2025-06-21@ 1704 on the ins and outs of various implementations of digital logic (SW, FPGAs, ASICs) and how it has changed and is changing.

I am using my usual approach to trying to figure out what happened in this accident. Which is to perform a possibility analysis: ideally, to consider all possible scenarios and prune out ones that do not fit with the facts as we know them. Might sound easy but it's not trivial, and there aren't that many people who become really good at it (and I am not even sure that my colleagues who are good at it think that I am......).

Severe reduction of thrust, simultaneously, just after unstick is one of the "facts as we know them". The control systerns for engines and fuel systems on the 787 are based on digital-logical electronics, including SW. Every digital-logical system may have bugs. In forty years of working with and around such systems I have never encountered one which didn't. Never. (Some eminent colleagues did try to do so with the "Tokeneer" project - and it took a year or two to find the bugs).

A bug in the digital-logical FADEC is a possibility. As far as I am concerned, it stays in the possibility analysis until it can be ruled out. Which it cannot be at this stage.

For this purpose, it does not matter what the logic is based on, or whether some SW-HW architectures can be less susceptible, for whatever reasons, than others.

How can that bug affect two independently controlled and powered engines, at almost exactly the same time?

Short introduction on faults:

There are systematic and random faults.
Mechanics and SW only know systematic faults, only electronic HW does have random faults.
Why?
Mechanics are large enough to react in a reproduceable manner, SW too (on working HW).
Worn out part? -> systematic fault, you did not calculate wear correctly in your analysis, adding maintainance cycles
Broken nut due to too much torque? -> systematic fault, you did not analyse the severity and put QM measures in place within maintainance.
broken fan blade due to corrosion? -> systematic fault, incorrect testing and environment assumption, other fan blades may be corroded, too
SW crash every 10 years? -> systematic fault, you did not make it robust against some nasty timing circumstance or race condition.
So systematic faults do not mean they happen instantly, they may happen never or after dozens of years. Not on all parts but e.g. just if there is a problem in production like a cavity in aluminum casting.
But there are systematic ways to avoid them (FMEA, FTA...). This is why every accident is analyzed. If it was a systematic fault, other party could be affected.
This also counts for SW, there are techniques to avoid errors on all levels of development. But having the same SW with the same inputs and the exact same timing will give the same outcome (...on both engines).

Random faults are unique to electronic HW. The reason is just because of miniaturization. One cannot exclude migration and degradation on such small scales, there will be faulting transistors or other parts. If you scale them bigger, your processor runs on kHz instead of MHz, this is not an option. Those are random HW faults. There are calculation methods (FMEDA) to calculate the expected lifetime and error probability over the expected temperature profile. But if you give the faulting part the same input, it will show the fault again. But not any other part without the degraded transistor will give you the same error.
And there are soft faults (not SW faults, soft faults), those are not reproducible at all. Imagine some hard x-ray from the sky hitting a DRAM cell and changing it's state. You will never be able to reproduce this hard x-ray which made it's way thru the atmosphere without interaction hitting exactly the right \xb5m\xb3 to do the same bit-flip again. You can just have redundancy in information and processing (ECC, parity, lock-step etc.) to have enough probability to detect (and reset) or correct the fault.

Faults, no matter if systematic or random can be single, multiple, latent and dependent faults.
Single fault: a single error leads to a hazardous state
Multiple point fault: more than one (independent) error will lead to a hazardous state
Latent fault: a fault which remains undetected for some time (e.g. only detectable during startup / power-on) and not hazardous by itself. But together with a second fault it will lead to a hazardous event. But the probability of 2 errors within a power cycle is sufficiently low.
Dependent fault: A fault, by itself not hazardous which will influence the probability of a second fault with both in combination... exactly. (-> freedom from interference not given)

Having the same SW bug on both engines would fall within the dependent fault category. Redundancy only helps if freedom from interference is given.
So if there is a fault in SW and both FADEC instances get the same input and have the same timing, they will react the same way. If there is a hidden systematic fault, both will show it. This is why things are often programmed twice by independent teams or on different processors. They will not do the exact same mistake. Or there is a normal function and a supervisor like TCMA. One complex function but a tiny, easy to analyze supervisor just checking for the hazardous state.

All this can be analyzed and is really lots of work but necessary. So first there is an analysis how severe a fault would be on a high level, this defines the necessary level of analysis in detail.

Originally Posted by JustusW
if ( happy == true ) {
print("I'm happy!"
}

Mmm. Yes.

To be clear, this is 'happily' executed in perl , and doesn't give the results you might expect.

if ( happy == false ) {
print("I'm happy!")
}

A FADEC might faithfully implement the logic of the design, Someone might even have spent a lot of resources to formally prov e that the FADEC logic implements the specification. The trouble is, that does not tell you that the specification is correct .

Additionally, even if the specification is correct, it is possible to be implementing the code on flawed hardware.

It is not impossible that a common set of inputs* triggered an unusual FADEC response in separate FADECs: however, not impossible does not mean that it is likely .

*For example, the FADEC could be programmed to expect an input value to be a positive integer e.g. for altitude. If the device that sends the altitude value to the FADEC sends a negative value, one of several things might happen: (1) it could be rejected as being out of range. (2) It could be replaced with the lowest possible positive value: +0 (3) It could interpret the signed integer it has been given as an unsigned integer and treat it as an unrealistically large positive value.

I am not saying this is what has happened. Getting disagreement in the format of values sent between differing systems is not an unknown problem. Software problems as a result of poor, or differing interpretations of specifications have hit many high-profile projects: such as the Mars Climate Orbiter and the Ariane 5 - so I will not say that a system that includes FADECs cannot have problems: rather, that problems are unlikely .

I have to both agree and disagree with both this and the next post by TryingToLearn.

Yes, there may be (let's assume is) "identical" FADEC/TCMA hardware and firmware on both engines, but if the Left Engine is subject to Mars in the house of Uranus (wink wink), then the Right Engine cannot be, maybe it's Venus in the same House. This is simply because the Left engine TCMA 'contraption', I'm going to call it, is monitoring Left Engine Conditions (Shaft Speed, T/L setting / position data - Right or Wrong, and calculating and comparing accordingly against its internal map) while the opposite TCMA "device" is monitoring and calculating etc, Right Engine Conditions. There are some things in common, but (I say) it's virtually impossible for the Engine Conditions being individually monitored to be identical in both engines.

The Thrust Levers are electro-mechanical devices, almost certainly at this stage pushed by a somewhat squishy human hand, likely with a slight offset. What is the probability that those two levers are in identical positions, and even if they are, that the calibration (e.g. "zero points") of both levers are identical, and that the values they output (response slopes/curves) are exactly matching in every matching point in their individual travels? That's just one aspect, but consider the engines. They are different ages. Have different amounts of wear. They have separate fuel metering valves (or other names), separate HP Fuel pumps (and, I guess relief valves?), all also subject to wear), and each has a host of other, correspondingly paired, sensors, (maybe of different makes and certainly of different ages and different calibrations and response curves) from which each FADEC, supposedly independently of the TCMA, adjusts the fuel metering device settings and resulting engine power, and shaft RPMs follow in some other slightly non-matching way.

Sure, I would completely agree that the two engines and their calculated Throttle Lever positions to Shaft RPMs are always going to be similar during normal, matched operation, and they will very likely dance with each other, maybe one 'always' (75% of the time, say) leading during one dance (TO, say) with the other leading in dancing to a different tune (descent, say).

To me, the fact that this appears to have been an almost simultaneous dual engine failure, pretty much, for me, rules out a FADEC/TCMA firmware bug, especially as there don't seem to be any reports of even a single engine mid-air TCMA shutdown.

HOWEVER, and I want to stress this, that doesn't rule out the possibility that both TCMAs shutdown their respective engines simultaneously. Any lack of simultaneity observed would be due to those slight differences in other pieces of hardware, such as the time for one Shutoff valve to close versus the other.

As far as I know, there isn't enough information on what's actually inside those TCMA Black Boxes to say anything for sure, but here's a thought, which I think has been alluded to, or the question asked, here in one or other thread, earlier.

What does the TCMA firmware do when an engine is already running at a high power setting and TWO things occur in quick succession? I suspect this kind of event is a highly probable cause, but these two events have not occurred close enough together, or ever, before.

Imagine this: Plane taking off, Throttle Levers near Full Power, Engines performing correctly, also near Full Power, Rotation etc all normal, plane beginning to climb, positive rate achieved.

Pilot calls GEARUP. GearUp, activated.

The Gear Retract sequence begins. Due to some unforeseen or freshly occurring (maybe intermittent short or open circuit) linkage between the gear Up sequence and the WOW or Air/Ground System, the signal to both TCMAs suddenly switches to GROUND. All "good", so far, as the engine RPMs match the Throttle Lever settings and TCMA doesn't flinch. The plane could be in a Valid Takeoff sequence, so it had better not! But it does make a bit of sense. How is WOW / Air/Ground detected? Somewhere near the Landing Gear, I assume.

HOWEVER, now, a moment later, and perhaps due to a related system response, the Thrust Levers suddenly get pulled back to Idle, whether by man or Machine.

What would you expect the TCMA system to do? I would guess, fairly soon thereafter, two, independent, Fuel Cutoffs... Though I fully admit, I'm guessing based on a severe lack of knowledge of that Firmware.

Ok, no need for further explanation on that point, but I did refer to TCMA unflatteringly as a contraption, earlier. Last night (regrettably, before bed) I started looking at the TCMA Google Patent. Let's just say, so far, I'm aghast! My first impressions are bad ones. How did this patent even get approved? What I suspect here, now, is not a Firmware bug, but a serious Logic and Program Defect. But we'd have to see what's inside the firmware.

When I get more time, I'll dig deeper.

Sadly Machinbird , I think this has been covered, especially re the engines, in some depth already. Several who post here have extensive knowledge in FADEC and associated protections. It's all a mystery tbh and I guess we just need to wait now until some concrete evidence is produced by the AAIB.

Not wishing to dampen enthusiasm to find out a cause but, 2 Threads in, no-one is much the wiser. Just my opinion but that seems to be where we are. What a dreadful accident - I can't even begin to imagine what the crew went through in those final moments.

Again, disclaimer that my direct knowledge of the 787 specifics is limited, standard Boeing design practice is that all engine wiring is segregated between engines (and were practical, between FADEC channels).
The fuel switches are located adjacent to each other; however all the wiring would be separate.

Engine isolation means just that. No common wire bundles, no common connectors. You can move the fuel levers at any time - there is no lockout of any kind with respect to thrust lever position (imagine dropping something into the lever linkage that jams the thrust lever at max power - then being unable to shut that engine down?)
Obviously, since the thrust levers are placed next to each other - the separation that's available in the center console is limited, but as soon as the wiring exits that constrained area, the separation increases. Furthermore, the same engine-to-engine wiring separation also applies to channel A/B FADEC channels, as well as the fuel switch/fire handle wiring.
All these requirements are documented in the Boeing DR&O (Design Requirements and Objectives) - and there is an audit done late in the design process to insure compliance.
In short, you're barking up a tree stump - there is nothing there.

We know that the right-hand GEnx-1B was removed for overhaul and re-installed in March 2025 so it was at \x93zero time\x94 and zero cycles, meaning a performance asymmetry that the FADEC would have to manage every time maximum thrust is selected. If the old engine was still on the pre-2021 EEC build while the fresh engine carried the post-Service Bulletin software/hardware, a dual \x93commanded rollback\x94 is plausible. A latent fault on one channel with the mid-life core can prompt the other engine to match thrust to maintain symmetry, leading to dual rollback.

You think the Thrust Asymmetry Protection could kick in and leave the aircraft with little to no thrust?

Then why didn't that happen on the previous flight from Deli to Ahmedabad, or any of the previous flights since that engine install in March?
Thanks for the update, and in particular that bolded bit.

I too don't fly that plane or any other FADEC plane for that matter, so I'll leave that to others. I keep thinking of the DA-42 that killed both engines due to an odd circumstance they didn't think would happen. It was a gross example of a corner case, but you can have other ones much harder to root out.

This could be several issues aligning to cause the loss of thrust. If the new engine was installed and the synchronisation step was omitted by maintenance staff and the engines had different CPU/software versions then there could be an emergent failure mode when maximum thrust is applied resulting in a FADEC rollback. Almost impossible to anticipate and create that in a test scenario. Don't overlook that the pilot moving thrust levers to override rollback will be ignored by the software, if the FADEC has flipped into protective mode.

That said, the continued absence of the FAA issuing an Emergency Airworthiness Directive for the Dreamliner suggests to me the fault was something like contaminated fuel which was specific to that flight.

However, a roll back on its own to idle would not give the evidenced gear behaviour nor the RAT (I happen to concur that the RAT was deployed and probably automatically). Given the gear tilt, it is safe to assume no engine is at idle, the normal electrical systems are not functioning at all.

The idea is to set the switch to CUTOFF and then to ON as that resets the FADEC (the circuit that controls the engine) and hopefully clears any issues it might have. The hope is that the turbine is still rotating fast enough for the FADEC to restart it. I believe this works the same as the auto-relight feature.
The turbine rotation would also provide the electrical power for that.

Do a thread search on "detent" to learn more about the construction of these switches than you ever wanted to know.
There's also a section on them in paulross 's https://paulross.github.io/pprune-th...171/index.html , but it may not be up to date. (Still a great resource, though.)

Unfortunately the wikipost linking to it is gone, presumably a victim to the recent forum changes.

Muscle memory is a strange and (usually) wonderous thing. It allows us as humans to perform amazing things without actually thinking about what we are doing. Professional Athletes have perfected this to a high art, but the rest of us do things using muscle memory on a regular basis. Back when I was still racing, I happened to look down at my hands on the steering wheel in fast, bumpy corner, and I was simply amazed at the large, rapid steering inputs that I was making to compensate for the bumps - with absolutely zero conscious thought. Muscle memory at its best.

However, it can also bite us. The Delta dual engine shutdown during takeoff from LA (referenced way back when in the 1st accident thread) was caused by muscle memory - the pilot reached down to set the EEC switches (located near the fuel On-Off switches) but muscle memory caused him to do something else - set both fuel switches to OFF. Fortunately, he quickly recognized his error, placing the switches back to RUN and the engines recovered in time to prevent a water landing (barely).

It is conceivable that a pilot - reaching down to the center console to adjust something unrelated - could have muscle memory cause him to turn the fuel off to both engines. While all new engines are tested for "Quick Windmill Relight" - i.e. the fuel switch is set to CUTOFF with the engine at high power - and the engine must recover and produce thrust withing a specified time (memory says 60 or 90 seconds) - it takes a finite amount of time for the engines to recover (spool down after a power cut at high power is incredibly fast - plus moving the switch to CUTOFF causes a FADEC reset, which means it won't do anything for ~ 1 second). Doing that at a couple hundred feet and the chance that an engine will recover and start producing thrust before ground impact is pretty much zero

It would be a lot more conceivable if the pilot had been practicing this action in a simulator recently.

It's ironic that cognitive science arguably started with 'The Cambridge Cockpit'; an attempt to make sense of, and mitigate, pilots doing this sort of thing when tired, stressed and so on. This kick started an ergonomics revolution which appears to have come full circle. Now we have cognitive science offering Bayesian accounts of neural function that might explain how innocent but unfortunate priming of 'muscle memory' when practicing for emergencies could, almost predictably, lead to this sort of complex, protection overriding, error.

As non consciously executing a complex, well practiced, but unintended, action is a fairly common experience in less critical situations, I'm surprised that there isn't already a more effective ergonomic fix than the safety switches fitted.

To be less subtle- someone cuts off switches, unknown to PF and possibly PM, with throttles off idle... Would do nothing.

Worst case, at next idle TLA, engine shuts down. I bet eyes go to cutoff switches after a scan, surely EICAS/ECAM has a Captain Obvious alert set.

Runaway RPM or locked RPM, some FADECS latch at 86 or so % N1- you'd still need TLA of idle for the cut off switch to work.

Volcanic ash, loss of all engines, desire the simultaneous FADEC reset of cycling the cutoffs- just coordinate with PM for idle TLA.

Other jets have this standard, today.

I'd say the throttle needing to be at idle is one heck of a CRM/TEM-based reason. Dual resolvers can fail, but throttles no longer get locked by jammed cable runs to the FCU on the engine. A resolver bypass would possibly be required in extremis 10 to the minus 6 to 9th case.

Who flies the throttles in normal? PF
Who typically performs the steps, including idle TLA of shutdown/restart in flight? PM.

This has all been answered in previous posts, but I'll repeat it for those you don't want to look back through something like 150 pages:

Thrust Lever Angle (TLA) is measured directly by the FADEC, using a resolver hardwired to and excited by the FADEC. Both FADEC channels have their own resolver input - on most Boeing aircraft it's a common resolver with two sets of electrically isolated windings, however on the 787 it actually uses two mechanically separate resolvers. The resolver is basically read as "sine" and "cosine" which is converted in the angle. This also makes error detection easy, using the sine squared + cosine squared relationship. Any other aircraft systems that use TLA use the TLA signal relayed back to the aircraft by the FADEC.

The fuel control switch is a two-position multiple pole 'latching' switch - you have to pull it out slightly over detent to move it between the RUN and CUTOFF positions (on other aircraft there is an interposing relay for some of the functions. not sure about the implementation on the 787). Moving the switch to cutoff sends a DC signal to both the High Pressure ShutOff Valve (HPSOV) in the fuel control and the spar valve commanding them to close. HPSOV is solenoid actuated and is near instantaneous, Spar Valve takes ~one second to change positions (yes, this is different than some other airframers that only send the signal to one valve or the other, but it's been standard Boeing design practice since the early 1970s). Both the HPSOV solenoid and the Spar Valve are designed to stay in their last commanded position if airframe power is lost. Moving the switch to CUTOFF also sends a 'reset' signal to the FADEC - meaning the FADEC will be offline for roughly one second. On the 787 (and 777 and 747-8), there is a brief pause (~0.25 seconds) before the shutdown signal is sent to the engine to allow the electrical system to reconfigure to prevent a brief interrupt of electrical power to the rest of the aircraft.

Pulling the Fire Handle does the same thing as the fuel condition switch - via separate wiring (physically isolated from the fuel switch wiring to help protect from things like rotor burst damage), with the exception of the FADEC reset (since there is no requirement to be able to restart the engine after a Fire Handle shutdown).

There is absolutely no TLA input into either the fuel conditions switch or the Fire Handle - you can shutdown the engine via either regardless of Thrust Lever Angle.

All this is standard Boeing design practice (and except for the no-break electrical power transfer) has been for at least 50 years. This is enforced by the Boeing "Design Requirements and Objectives" - DR&O - compliance with is demonstrated by an audit after the final design freeze.

What the hell happened in the cockpit?

08:08:42 Engine 1 and Engine 2 fuel cutoff switches transitioned from RUN to CUTOFF position.
One of the pilots asks the other why did he cutoff.
The other pilot responded that he did not do so.
08:08:52 Engine 1 fuel cutoff switch transitioned from CUTOFF to RUN
08:08:56 Engine 2 fuel cutoff switch also transitions from CUTOFF to RUN

Who (or what?) operated the cutoff switches?