June 21, 2025, 08:26:00 GMT
permalink Post: 11907575
I am only asking about an engine failure memory item. Fire, separation or severe damage being a different beast.
Are you confirming that there is no specific engine failure memory item? When safe run the QRH?
I completely agree.
Are you confirming that there is no specific engine failure memory item? When safe run the QRH?
so it would be extremely unlikely this crew actually got the stage of touching a fuel control switch.
June 21, 2025, 08:43:00 GMT
permalink Post: 11907580
June 21, 2025, 08:56:00 GMT
permalink Post: 11907591
The issues with the "they shut down the wrong engine" theory:
1. No asymmetry evidence with flight path deviation. No roll, no yaw effects
2. No rudder inputs visible.
3. No crew should be doing memory items below 400ft. Boeing requires each crew member confirm together memory item switch/control selections.
4. Non-normal gear truck tilt position, a one engine failure should not affect the C hydraulics. As per (3) gear would be selected Up before any memory actions.
The evidence so far is an almost simultaneous dual engine failure, which rules out alot of other theories.
1. No asymmetry evidence with flight path deviation. No roll, no yaw effects
2. No rudder inputs visible.
3. No crew should be doing memory items below 400ft. Boeing requires each crew member confirm together memory item switch/control selections.
4. Non-normal gear truck tilt position, a one engine failure should not affect the C hydraulics. As per (3) gear would be selected Up before any memory actions.
The evidence so far is an almost simultaneous dual engine failure, which rules out alot of other theories.
June 21, 2025, 09:51:00 GMT
permalink Post: 11907612
There has been another incident of a TCMA equivalent function shutting down both engines. An airBaltic airbus a220 in July 2021 (YL- AAQ) had both PW1500 engines shut down by \x93TCM\x94 logic in the FADEC immediately upon landing.
This seems to have been caused by a mismatch between actual and commanded thrust caused by rapid throttle movement that was \x93saved up\x94 until TCM was subsequently activated by its air/ground logic.
A description can be found on flightglobal but I have too few posts to include a url - So google a220 revised software engine shutdown
This seems to have been caused by a mismatch between actual and commanded thrust caused by rapid throttle movement that was \x93saved up\x94 until TCM was subsequently activated by its air/ground logic.
A description can be found on flightglobal but I have too few posts to include a url - So google a220 revised software engine shutdown
June 21, 2025, 12:31:00 GMT
permalink Post: 11907707
It’s a possibility (as is virtually anything that doesn’t break the laws of physics) but all the training, practicing and checking would have been to emphasise SOPs, which are to leave all the engine controls where they are until you have done a proper interactive diagnosis at a safe height with the flightpath assured.
Where the meme has come from that jet pilots have to shut down engines as quickly as possible I don’t know but it is incorrect. If you left a failed engine without securing it for 5 minutes, little to no harm would come of it. Even if it was on fire (which is not necessarily flames, just higher than normal temperatures inside the nacelle) they are certified to be in this condition for some considerable time before it becomes a problem. Yes, I think the phrase “without undue delay” could be used for a fire indication but that’s a minimum of 400’AGL in Boeings and does not absolve you of all the cross-checking and CRM that should happen with an engine shutdown. This is practiced/checked at the least every 6 months in EASA land and any attempt to rush a shutdown at low level would lead to a debrief and more training/checking.
To put it this way, control of the aeroplane and lateral/vertical navigation is far more important than doing stuff with a failed power plant. Something like an ET should be absolutely prioritised over engine drills.
Where the meme has come from that jet pilots have to shut down engines as quickly as possible I don’t know but it is incorrect. If you left a failed engine without securing it for 5 minutes, little to no harm would come of it. Even if it was on fire (which is not necessarily flames, just higher than normal temperatures inside the nacelle) they are certified to be in this condition for some considerable time before it becomes a problem. Yes, I think the phrase “without undue delay” could be used for a fire indication but that’s a minimum of 400’AGL in Boeings and does not absolve you of all the cross-checking and CRM that should happen with an engine shutdown. This is practiced/checked at the least every 6 months in EASA land and any attempt to rush a shutdown at low level would lead to a debrief and more training/checking.
To put it this way, control of the aeroplane and lateral/vertical navigation is far more important than doing stuff with a failed power plant. Something like an ET should be absolutely prioritised over engine drills.
June 21, 2025, 13:50:00 GMT
permalink Post: 11907770
The issues with the "they shut down the wrong engine" theory:
1. No asymmetry evidence with flight path deviation. No roll, no yaw effects
2. No rudder inputs visible.
3. No crew should be doing memory items below 400ft. Boeing requires each crew member confirm together memory item switch/control selections.
4. Non-normal gear truck tilt position, a one engine failure should not affect the C hydraulics. As per (3) gear would be selected Up before any memory actions.
The evidence so far is an almost simultaneous dual engine failure, which rules out alot of other theories.
1. No asymmetry evidence with flight path deviation. No roll, no yaw effects
2. No rudder inputs visible.
3. No crew should be doing memory items below 400ft. Boeing requires each crew member confirm together memory item switch/control selections.
4. Non-normal gear truck tilt position, a one engine failure should not affect the C hydraulics. As per (3) gear would be selected Up before any memory actions.
The evidence so far is an almost simultaneous dual engine failure, which rules out alot of other theories.
June 21, 2025, 15:46:00 GMT
permalink Post: 11907858
TCMA continues to be one of the few (very unlikely) causes of/contributors to simultaneous shutdown of both engines. So far, though, I don't think we've seen a credible scenario explaining the possibility that TCMA was triggered in this accident. I'm not sure I understand your speculation.
In the scenario you are considering, it's clear that the air/ground state would be wrongly "understood" by the TCMA function. But we don't have, AFAIK , a credible theory for how that might happen. Surely it would have to result from either incorrect signals from the relevant sensors or a failure of the related logic in the FADEC TCMA function, or a combination of those. Indeed, I don't think we yet know exactly which sensor readings that logic depends on or how those readings are fed to the FADEC. Does your speculation include any thoughts about this?
Also, the FADEC TCMA function has to "believe" that the engine is operating at high power and not responding to thrust lever operation. In your proposed scenario, is this also a logic failure — in both FADECs? Or false inputs from both TLs? Or are both engines actually operating at higher than commanded power levels?
Or do I misunderstand your post?
In the scenario you are considering, it's clear that the air/ground state would be wrongly "understood" by the TCMA function. But we don't have, AFAIK , a credible theory for how that might happen. Surely it would have to result from either incorrect signals from the relevant sensors or a failure of the related logic in the FADEC TCMA function, or a combination of those. Indeed, I don't think we yet know exactly which sensor readings that logic depends on or how those readings are fed to the FADEC. Does your speculation include any thoughts about this?
Also, the FADEC TCMA function has to "believe" that the engine is operating at high power and not responding to thrust lever operation. In your proposed scenario, is this also a logic failure — in both FADECs? Or false inputs from both TLs? Or are both engines actually operating at higher than commanded power levels?
Or do I misunderstand your post?
- In error, PF reduces power to idle and/or reverse at a speed after V1 (either deciding to reject, or for some unexplained reason e.g. the recent BA incident at LGW)
- Decision is changed to continue take-off, thrust levers moved to TOGA
- Let's say the thrust inputs are similar to NM985 and TCMA is triggered; and engines shut down around the time of rotation
- A/C rotates achieving a maximum speed in the region of 184kts
Q: Would the a/c have enough kinetic energy a 184kts to climb to 100-150ft agl and then reach its final position if the engines had failed at, or just, before rotation?
A: Theoretically possible - see calculation here . NB, the a/c actually flew 1.5km from the end of the runway and 2.3km from the likely point of rotation.
Q: Doesn't the forward position of the gear mean that power failed after the pilots had selected gear up?
A: Inconclusive - had hydraulic power had been lost prior to rotation, the gear could also be in this position - explanation here
Q: If the throttle levers were brought to idle during take-off, would the A/C have applied autobrake, reversers and speedbrake?
A: Yes, although there is a built in delay before reverser and speedbrake actually deploy - see here .
Q: Is the ADS-B data consistent with this scenario?
A: Yes, e.g. the Flightradar data shows the aircraft decelerating rapidly (12 knots in 4.2 seconds) from close to rotation. However, it's not clear how accurate this data is. For one, the altitude data is +/- 25 feet, second, while I was under the impression FR would have received airspeed data from the a/c sensors, this post suggests maybe not.
Q: Does TCMA activation require the thrust levers to be at idle or does it function when the thrust levels are above idle, but where the actual thrust is above that commanded?
A: No, the latter is true (i.e. idle is not required) - confirmed here - there are of course many protections against false activation
Q. Did AI171 have the same software version / logic paths as NH-985
A. Unknown. That a/c had Trent 1000s so to some extent the software is different, but we understand the TCMA logic is broadly the same regardless of engine. I have not seen a post clarifying whether the TCMA software has been updated /changed via SB since 2019 to account for this incident.
Be grateful if posters could refrain from speculative responses "e.g. I think this is unlikely because I feel x". I am not opining on how likely this sequence of events is, simply trying to summarise whether or not this theory has been ruled in or out.
I also recommend this post for a summary to read before posting. .
Last edited by lighttwin2; 21st June 2025 at 16:13 .
June 21, 2025, 16:31:00 GMT
permalink Post: 11907891
Some assumed numbers about normal biotreatment.
https://www.biobor.com/wp-content/up...ation-IATA.pdf
If we assume 50 tonnes fuel load a 100ppmw biotreatment will be 5kg of biocide total in all tanks.
The GEnx-1B will burn about 4,5kg/s fuel each on a take off run (give or take a bit) so 9kg/s in both donks for about 20s until rotate.
So the total nominal biocide dose could be pumped in about half a second through both engines on take off power if it where not mixed at all and arrives in both engines at the same time.
This gives you an idea that with the nominal amount of biocide dose not much could have happened. If biocide is the source of this dual EFATO than an extreme overdose in addition to wrong application preventing mixture with the fuel had to be the case.
https://www.biobor.com/wp-content/up...ation-IATA.pdf
If we assume 50 tonnes fuel load a 100ppmw biotreatment will be 5kg of biocide total in all tanks.
The GEnx-1B will burn about 4,5kg/s fuel each on a take off run (give or take a bit) so 9kg/s in both donks for about 20s until rotate.
So the total nominal biocide dose could be pumped in about half a second through both engines on take off power if it where not mixed at all and arrives in both engines at the same time.
This gives you an idea that with the nominal amount of biocide dose not much could have happened. If biocide is the source of this dual EFATO than an extreme overdose in addition to wrong application preventing mixture with the fuel had to be the case.
Neither incident reports make for pretty reading. But, in neither case did they experience near-simultaneous engine failure (not far off that, but still assymetric).
Last edited by Gary Brown; 22nd June 2025 at 08:19 . Reason: Correcting math error from 100 x to 10 x ...
June 21, 2025, 17:38:00 GMT
permalink Post: 11907924
I'm sure this has been answered elsewhere and that I've just missed it by scrolling too fast through the backlog of (much appreciated) posts, but it's been buzzing round my mind and I wanted to try and get an answer if possible please.
Has it been confirmed that there was a dual engine shutdown and, if so, why weren't people commenting on this from the videos of the incident (if the audio was good enough to detect the RAT then surely it was good enough to tell whether the engines were running).
Thank you for your patience!
Has it been confirmed that there was a dual engine shutdown and, if so, why weren't people commenting on this from the videos of the incident (if the audio was good enough to detect the RAT then surely it was good enough to tell whether the engines were running).
Thank you for your patience!
June 21, 2025, 17:45:00 GMT
permalink Post: 11907933
Not
confirmed
. What is apparent is a (substantial) loss of thrust. That's what one can say with some certainty.
That has already been discussed, and there has been abundant comment. Suggest you go back and review the original thread, and this one as well.
Thank you for your interest.
if so, why weren't people commenting on this from the videos of the incident (if the audio was good enough to detect the RAT then surely it was good enough to tell whether the engines were running).
Thank you for your patience!
June 21, 2025, 18:31:00 GMT
permalink Post: 11907969
This has been setting a bit wrong with me for the entirety of the thread. When people say "software" they have an image in their head, and in this case the image is rather unhelpful if not outright wrong.
When we say "Software" we mean things like this:
This is the type of software we are usually subject to in our everyday lives basically everywhere. Your phone, your fridge, your oven, your water heater, your car, etc. pp. ad nauseam.
In case of the Safran FADEC 3 this is not actually what we're dealing with. It uses something called an FPGA (Field Programmable Gate Array) which is a very different beast to what we are used to dealing with.
In a normal computer we simply translate code like above into something less human readable but instead interpretable by a microchip or the like. This "machine code" is then run sequentially and does whatever it is meant to do (hopefully). If we were to expand our little happy check above with another condition then our computer would sequentially check these conditions. This opens up a lot of hoopla about timing, especially when interacting with the real world.
Let's say you want to track N1 and N2 in an engine. You'd have to make a measurement, that value would have to go through an AD converter, which writes it to some (likely volatile) storage, where it is then accessed by some sort of (C)PU, transferred to yet another piece of volatile memory and then used in the computation. This takes time because you can't do those things within the same clock cycle.
Unsurprisingly this is rather inconvenient when dealing with the real world and especially when dealing with volatile physical processes that need monitoring. Like a modern turbine engine.
Enter the FPGA.
While it is programmable what that actually means is that (at a very high level) you can build a thing called a truth table, that means a definitive mathematical mapping of input states to output states. Unlike our sequential CPU driven system an FPGA will be able to perform its entire logic every time it is asked to do so. We don't have to wait for our happy check to perform any other check.
This is very useful for our Turbine Engine, because now we can verify that N2 is smaller than X without delaying our check that the Throttle Control Lever setting is within acceptable distance to N1 and N2, while also creating the appropriate output for all the different bypasses and bleed valves in a modern engine, and so on. The Safran FADEC 3 does this "up to 70 times per second" as per the vendor.
In order to be manageable the actual FADEC consists of multiple different pieces, many of which are FPGAs. At least two are used for so called "input conditioning". This is where we deal with converting messy real world values from sensors subject to real physics into nice and clean numbers. Here our values from the Throttle Control Levers come in, the signal from our N1 and N2 sensors, WOW signal, and on, and on, and on.
This is then changed into logic level signals provided to a different set of FPGAs. Lacking an actual schematic I suspect that this is what sometimes is referred to as "channels". The channel could consist of both a signal conditioning stage and then a logic stage or (more likely) redundant signal conditioning stages feed separately into separate FPGA "channels" are evaluated and then the end result is likely put into yet another FPGA for control output generation.
Why is this important?
Because it is basically impossible for a "bug" to exist in this system. These systems are the epitome of a dumb computer. They do _precisely_ what they are meant to do. The TCMA is simply a set of conditions evaluated just like any other condition in those FPGAs. If there is an "error" then it is in the requirements that lead to the truth table used by these devices but never in the truth table itself. In fact these types of computation device are used precisely because of that very property.
So when people say "the FADEC software" (ie TCMA) has "failed" or "has a bug" what they're really saying is:
Regards,
Justus
When we say "Software" we mean things like this:
This is the type of software we are usually subject to in our everyday lives basically everywhere. Your phone, your fridge, your oven, your water heater, your car, etc. pp. ad nauseam.
In case of the Safran FADEC 3 this is not actually what we're dealing with. It uses something called an FPGA (Field Programmable Gate Array) which is a very different beast to what we are used to dealing with.
In a normal computer we simply translate code like above into something less human readable but instead interpretable by a microchip or the like. This "machine code" is then run sequentially and does whatever it is meant to do (hopefully). If we were to expand our little happy check above with another condition then our computer would sequentially check these conditions. This opens up a lot of hoopla about timing, especially when interacting with the real world.
Let's say you want to track N1 and N2 in an engine. You'd have to make a measurement, that value would have to go through an AD converter, which writes it to some (likely volatile) storage, where it is then accessed by some sort of (C)PU, transferred to yet another piece of volatile memory and then used in the computation. This takes time because you can't do those things within the same clock cycle.
Unsurprisingly this is rather inconvenient when dealing with the real world and especially when dealing with volatile physical processes that need monitoring. Like a modern turbine engine.
Enter the FPGA.
While it is programmable what that actually means is that (at a very high level) you can build a thing called a truth table, that means a definitive mathematical mapping of input states to output states. Unlike our sequential CPU driven system an FPGA will be able to perform its entire logic every time it is asked to do so. We don't have to wait for our happy check to perform any other check.
This is very useful for our Turbine Engine, because now we can verify that N2 is smaller than X without delaying our check that the Throttle Control Lever setting is within acceptable distance to N1 and N2, while also creating the appropriate output for all the different bypasses and bleed valves in a modern engine, and so on. The Safran FADEC 3 does this "up to 70 times per second" as per the vendor.
In order to be manageable the actual FADEC consists of multiple different pieces, many of which are FPGAs. At least two are used for so called "input conditioning". This is where we deal with converting messy real world values from sensors subject to real physics into nice and clean numbers. Here our values from the Throttle Control Levers come in, the signal from our N1 and N2 sensors, WOW signal, and on, and on, and on.
This is then changed into logic level signals provided to a different set of FPGAs. Lacking an actual schematic I suspect that this is what sometimes is referred to as "channels". The channel could consist of both a signal conditioning stage and then a logic stage or (more likely) redundant signal conditioning stages feed separately into separate FPGA "channels" are evaluated and then the end result is likely put into yet another FPGA for control output generation.
Why is this important?
Because it is basically impossible for a "bug" to exist in this system. These systems are the epitome of a dumb computer. They do _precisely_ what they are meant to do. The TCMA is simply a set of conditions evaluated just like any other condition in those FPGAs. If there is an "error" then it is in the requirements that lead to the truth table used by these devices but never in the truth table itself. In fact these types of computation device are used precisely because of that very property.
So when people say "the FADEC software" (ie TCMA) has "failed" or "has a bug" what they're really saying is:
The conditions that were experienced in the real world by the system were incorrectly assessed in the system requirements and lead to an undesired output state when taking into account the real world result we would have preferred.
A bit of a mouth full, granted, but an important mouth full. This simply wouldn't be someone missing a semicolon somewhere in a thousand lines of code.
Regards,
Justus
June 21, 2025, 20:40:00 GMT
permalink Post: 11908040
Sadly I'm unable to post links or I would have done so in my original post.
First off I should note that what I wrote is a massive oversimplification. I'd expect that neat metal box with the nice rugged connectors that Safran shows on their homepage to contain an ungodly mess of microcontrollers, ICs for signal conditioning, the FPGAs I mentioned and components that are well beyond my understanding of electronics. But from personal experience with a certain aircraft related company operating in Munich I can tell you that marketing level slides like you showed are so far removed from the technical reality that they might as well be fan-fic.
I agree with your observation that current microcontrollers are more than capable enough of dealing with these types of real time computation, but keep in mind that the Boeing 787, and thus its FADEC, had its first flight in 2009. Back then I would be rather more careful with that statement. Mostly because I would have just started my Computer Science Degree at the time XD
But maybe I should explain how I've arrived at the post above. I've basically spent the past few days on and off scavenging anything I could find on the 787 FADEC specifically. It was mentioned in this thread or the old one that the 787 uses the Safran FADEC 3, so my primary jumping off point was the Safran website and the information available on it. That plus a couple of press releases namely by Actel about their flash-based ProASIC3 and ProASICPLUS FPGAs was the evidence I used to arrive at my conclusions above. I can't for the life of me find the quote about the dual signal conditioning FPGAs anymore, but they are in there as far as I can tell.
Overall information on the actual inner workings of these things are so sparse that outside of a few patents (keyword: configurable CPU) I was hard pressed to find anything at all to be honest. This all not being helped by me now finding references to my own post, thus having created my very own Woozle. Hooray.
I have only looked into the Safran FADEC 3 used on the 787 with the
GEnx-1B engine. The system was very innovative for its time and uses technology not previously used on FADECs to my knowledge. Even saying that, this could be one of those "technically true" kind of statements. My intention was to highlight that due to the totality of the processes and architectural style of these types of computational device the typical kind of "software error" or "bug" was basically impossible and that we'd be looking at something way further back with implications for the base requirements. I merely want to correct the implied misconceptions about the "software" being used here. As I said, FPGAs are just very different kinds of beasts.
I half agree and half disagree. Obviously I shouldn't have stated an absolute because, yes, it is possible despite multiple layers of engineering coordination that an error occurs at one step or another and is not caught by the multiple layers of checking. My point is that even in normal development FPGAs are well known to be bug resistant due to the way they are programmed. In general we create the logic condition tables using software which then translates our table into an actual FPGA configuration. This process is obviously not immune to faults but highly resistant and most importantly: Can easily be simulated.
Now, in normal "everyday" usecases I'd fully agree with you, but in case of the FADEC we're talking about an FPGA with logic condition tables where every line is likely to have its own separate binder of engineering notes, discussions and recommendations. It is there where any "error" or "bug" is most likely to reside by a considerable margin.
They were, and continue to be, a valid and reliable technology for this type of application especially before the extreme proliferation of consumer microelectronics in the last 20 years. The Safran FADEC 3 was designed at basically the same time as the first iPhone. And boy do I feel old now.
I realize that my post is a massive simplification of the topic, considering the intricate details that are simply unknown to the general public in a system as complex as a FADEC I certainly agree that things can go wrong. What I wanted to point out is the reliability of the last steps versus the design and requirements engineering phase. In general software development bugs will happen at any stage with basically equal likelihood. Given the way the standard processes around FPGA development alone would influence this probability towards the design and requirements engineering stage and then combining it with how it is handled in the Aerospace Sector I felt it was justifiable to state the extreme mismatch in probabilities of error the way I did. With the system being in operation for this length of time I do still feel comfortable with my original statement although I would now decorate it with a little * for the reasons mentioned by you and others.
In the end I hope it was useful for understanding where the reluctance of apportioning blame to these systems originates and at the same time talk about some pretty nifty tech that I know few Software Engineers will ever deal with in their entire career. I do still find the idea of a fault (with the understanding above of where that fault originates) in the FADEC a convincing possibility, it's merely the nature and location of that fault that I felt could benefit from some more specifics in regards to the nature of how they were most likely to actually happen. I would love to actually hear more about how the FADEC 3 is structured internally and which components it uses, but I suspect that those who know are under NDA.
Regards,
Justus
I ask because when I look for info about FADECs generally and FADEC 3 more specifically, I find this kind of documentation (notice the mention of FADECs in the lower left corner):
... which is part of a larger Powerpoint presentation by Ansys, explaining that these products are developed with SCADE development workbench, generating either Ada or C code, and that the resulting code runs under a microkernel realtime operating system:
... which is part of a larger Powerpoint presentation by Ansys, explaining that these products are developed with SCADE development workbench, generating either Ada or C code, and that the resulting code runs under a microkernel realtime operating system:
Now, obviously enough, a CPU can be embedded in an application-specific FPGA, but it would still execute machine code. And from my experience in other embedded systems development, current CISC or RISC CPUs have more than enough computation power to implement command and control on a modern turbofan.
But maybe I should explain how I've arrived at the post above. I've basically spent the past few days on and off scavenging anything I could find on the 787 FADEC specifically. It was mentioned in this thread or the old one that the 787 uses the Safran FADEC 3, so my primary jumping off point was the Safran website and the information available on it. That plus a couple of press releases namely by Actel about their flash-based ProASIC3 and ProASICPLUS FPGAs was the evidence I used to arrive at my conclusions above. I can't for the life of me find the quote about the dual signal conditioning FPGAs anymore, but they are in there as far as I can tell.
Overall information on the actual inner workings of these things are so sparse that outside of a few patents (keyword: configurable CPU) I was hard pressed to find anything at all to be honest. This all not being helped by me now finding references to my own post, thus having created my very own Woozle. Hooray.
Whilst you\x92re right to highlight the type of software in a FADEC, it should also be recalled that after the dual uncommanded engine shutdown event on the Baltic CS300, the FAA mandated updated FADEC software for all of a family of P&W engines. FADEC s/w can contain what we\x92d call bugs (whether they\x92re mistakes of coding or mistakes of logic in the design assumptions).
"it is basically impossible for a "bug" to exist in this system." This is not my experience.There are a lot of bugs in hardware and FPGAs. The guys doing this are real experts and bugs are less common than in software. But they do exist and they can be difficult to reproduce and subtle to diagnose. For someone new to FPGA it can be very intimidating because everything is parallel. It requires a different way of thinking. You can execute 1000 instructions simultaneously.Then various tricks to speed things up or reduce glitches add complexity. For example you may want to count in gray code or one hot instead of binary. Writing bug free FPGA code is very challenging. We cannot assume the FPGA code is free of bugs.
Now, in normal "everyday" usecases I'd fully agree with you, but in case of the FADEC we're talking about an FPGA with logic condition tables where every line is likely to have its own separate binder of engineering notes, discussions and recommendations. It is there where any "error" or "bug" is most likely to reside by a considerable margin.
Now, I am sure that the state of the art has moved on a lot since I was heavily involved in this sort of engineering but I can say that when my colleagues were prototyping ASICs (which are application specific devices that are not programmable like an FPGA) using FPGAs because it can be reprogrammed to cure bugs found in the code that created it I am absolutely sure that it is possible for a latent bug to exist in what has been built despite extensive testing being carried out. I well remember sitting down with an FPGA designer and a software engineer and showing them a situation where one of my radio devices was not being commanded to transmit, they refused to believe me until we captured their output signals in the failed state and I was thus able to prove to them that it was their logic and code that was failing and that my radio system was fully working except it couldn't possibly transmit when its enable line was inactive.
I would therefore never state that an FPGA-based FADEC is infallible. The devil will be in the detail of the functions contained within the devices and how they interact with each other and how the original logical functions are described and specified and then coded and checked for logical errors before getting on to the actual physical reality of the FPGA manufacturer's design and development system that bolts on to the back of the source code. If the FPGA devices are being pushed anywhere near the edge of their performance envelope, just like an aircraft things can go wrong. If a design begins with a device that is only just large enough in terms of how many logic blocks and routing channels are available for the logic required it's almost a certainty that development will take it to this area of its performance and a lot of tweaking may be needed which means that even more testing will be needed to be reasonable sure that it does what it is supposed to.
I would therefore never state that an FPGA-based FADEC is infallible. The devil will be in the detail of the functions contained within the devices and how they interact with each other and how the original logical functions are described and specified and then coded and checked for logical errors before getting on to the actual physical reality of the FPGA manufacturer's design and development system that bolts on to the back of the source code. If the FPGA devices are being pushed anywhere near the edge of their performance envelope, just like an aircraft things can go wrong. If a design begins with a device that is only just large enough in terms of how many logic blocks and routing channels are available for the logic required it's almost a certainty that development will take it to this area of its performance and a lot of tweaking may be needed which means that even more testing will be needed to be reasonable sure that it does what it is supposed to.
In the end I hope it was useful for understanding where the reluctance of apportioning blame to these systems originates and at the same time talk about some pretty nifty tech that I know few Software Engineers will ever deal with in their entire career. I do still find the idea of a fault (with the understanding above of where that fault originates) in the FADEC a convincing possibility, it's merely the nature and location of that fault that I felt could benefit from some more specifics in regards to the nature of how they were most likely to actually happen. I would love to actually hear more about how the FADEC 3 is structured internally and which components it uses, but I suspect that those who know are under NDA.
Regards,
Justus
June 22, 2025, 00:15:00 GMT
permalink Post: 11908173
I don't want to refute your theory, but given your 30 years of experience---presuming it is relevant--I'd ask you to clarify a few things.
First, water in fuel is not a novel concept and I would presume that the designers of the 787 knew about it. You've simply stated that water might collect and settle out, but how much water might you expect under those conditions (57% humidity doesn't seem terribly high to me) and what features and procedures are already there to mitigage water contamination issues? Your theory would imply that there basically aren't any. IDK how the tank venting system works, but the idea that some huge amount of water could have condensed in the tank from the outside seems preposterous.
Second, how much water do you think it would take to cause a sustained flameout in one of those engines? Remember that they have automatic continous relight, so you're going to have to sustain your flame suppression long enough for them to wind down completely. I think those engines were probably using something like 2 gallons per second of fuel along with 250lbs of air heated to over 1100F. Any fuel in the mix would burn and the water would be converted to steam so you'd need mostly water for a long time. So if you think a hundred gallons of water could have gotten into each tank then perhaps I'd buy your theory--which, btw, does fit the known facts pretty well. But I think that short of some woeful neglect, Boeing and AI already know about and have methods of dealing with water contamination. At least I hope so.
First, water in fuel is not a novel concept and I would presume that the designers of the 787 knew about it. You've simply stated that water might collect and settle out, but how much water might you expect under those conditions (57% humidity doesn't seem terribly high to me) and what features and procedures are already there to mitigage water contamination issues? Your theory would imply that there basically aren't any. IDK how the tank venting system works, but the idea that some huge amount of water could have condensed in the tank from the outside seems preposterous.
Second, how much water do you think it would take to cause a sustained flameout in one of those engines? Remember that they have automatic continous relight, so you're going to have to sustain your flame suppression long enough for them to wind down completely. I think those engines were probably using something like 2 gallons per second of fuel along with 250lbs of air heated to over 1100F. Any fuel in the mix would burn and the water would be converted to steam so you'd need mostly water for a long time. So if you think a hundred gallons of water could have gotten into each tank then perhaps I'd buy your theory--which, btw, does fit the known facts pretty well. But I think that short of some woeful neglect, Boeing and AI already know about and have methods of dealing with water contamination. At least I hope so.
Experience. Without wishing to dox myself, I've worked in engineering at a major airline from apprentice through (in no particular order) Line Maintenance, Heavy and Light Maintenance, to technical support and maintenance control on both Boeing and Airbus products, with various qualifications and authorisations along the way. [Hmm - Scrap this sentence?]On the day 9/11 occurred, I should have been making modifications inside a fuel tank instead of staring at the TV with mouth on the floor.
However, I would describe my experience as broad, yet shallow in respect to this incident. Some of my fleet I know every rivet. Some of my fleet I've only ever seen from a distance. I don't touch airplanes for a living any more. B787 though - is not my area of specialty. I'll dig in, but am not the expert. I am not a systems design engineer, so precise numbers and flow rates, are not what I do. But what the systems do, how they operate, what they look like, smell and taste like... yeah, I'm not a muggle. And I do have access to all the manuals and know how to use them. And - let me be clear, I am speculating. I was advancing a theory. It WILL be some flavour of wrong. The investigation will reveal all.
I Agree, Water in fuel is not a novel concept. Aircraft fuel tanks attract water - fact. How much? It varies. I've sumped tanks and got no water, I've seen drops of water beading about in the bottom of a gallon jug, I've seen gallons of water. I've been so covered in fuel I cant smell it or think straight and taken gallon after gallon not being able to tell if its fuel or water. I also agree that 57% humidity doesn't seem particularly high - its not south east Asian jungle levels - but I'm not an expert at humidity, 32Deg c at 57% humidity at 02:30 am is not going to be comfortable for me though. I looked at recent weather in DEL, and those values were at the higher end of the range.
Further, I believe the prevailing weather conditions on the ground are less important when it comes to the volume of water getting in. Fuel is cold, or gets damn cold during a 9 Hr flight. Fuel Temperature Management is an issue for our Drivers. So as the fuel is used at altitude, Air enters the tank through NACA Ducts in the outboard end of the wing. Its beneficial to maintain a slight positive pressure, amongst other things to reduce evaporation. (Added complication, there is also the Nitrogen Enrichment system due to TWA800 - but that's more about processing the air in the tank to change the properties and make it non-explosive). Then as the aircraft descends, more air enters as the air pressure increases. Its the humidity of that air in the descent that is going to determine the volume of water entering the tank and potentially the fuel. The water in the air condenses on the sides of the tank because of the cold post-flight fuel. It doesn't dissolve into the fuel, but sinks to the bottom. Ground temperature / humidity and time will likely affect how much water condenses out of that air while on the ground. There won't be a huge amount of air exchange on the ground. Likely if the AC landed at 2am, then from sunrise as the tank warmed up, there would actually be a flow out of the vents.
What Features and procedures are there to mitigate Water? I apologise if my post gave the impression that there are no mitigation processes. There are. Water is well understood in the industry.
Well for a start, Features / Design. The Aircraft has a water scavenge system. Water doesn't mix with fuel, it sinks to the bottom being about 20% denser than fuel, so at the very lowest point in the tank, the water scavenge system (Powered by the Aft Fuel Pump through a jet pump, a venturi like system) will suck up the 'fluid' at the very lowest point, where the water would collect and in Boeings words 'drip' that fluid into the path of the pump pickup inlet (but I'd describe it more as a 'squirt'). The idea being that a small amount of water injected into the fuel will be consumed by the engines harmlessly.
There is also agitation. The wing tank pumps are pretty much running constantly, from before engine startup to after engine shutdown. The pumps are quite violent to the fuel and supply more pressure then the engine could ever need. Any excess pressure is dumped right back into the tank, quite close to the pump, in a direction that would further stir up the fuel and help break up any water into suspended droplets.
This all works if there is a small amount of water in the fuel. The water scavenge pickup is right next to the pump inlet, but a bit lower. Little bits of water get managed. Looking at the pictures of the system, I'd say a couple of gallons of water would do no harm at all.
But if there was significantly more water in that tank. Guessing 10-30 + gallons, then the pump would be circulating water, or highly water rich fuel.
Then there's the suction pickup. Its in the same 'bay' as the aft fuel pump and located a little 'higher' than the pump inlet and water scavenge inlet. But also located between stringers that can separate out the settled water ( I wish I could share the pictures, but more than my job is worth ) I can imagine the suction pickup being in a pool of 'stagnant' water.
I also saw a post from Metcha about the scavenge system blocking with Algae - I don't know about that (B787 not my fleet). But possible that could aggravate things. There's also the reports of the Indian AAIB looking at the Titan Biocide incident. Its possible that might be related and could modify the theory.
Procedures - There's the (at my airline weekly I think) procedure to 'sump' the tanks. There are drain points in the tank. Valves that you can push in with a tool and fluid drains. As described earlier (and videos exist on YouTube), you drain about a gallon of fluid and examine it for water. Most often in temperate climates (my experience), there's a few 'beads' of water in the bottom of the jug, moving about like mercury. Except when there's more. Sometimes there's a clear line in the jug, half water, fuel above. And sometimes a gallon of water, that smells like fuel. You drain it until you are sure there's no water.
Could 'that much' water have condensed in the tank? Well - There's the question. I guess the basis of the theory is that on descent into DEL, the wing tanks picked up some very humid air, which settled water into the tanks through the night. Then, as the theory I posited must work, the wing pumps must have circulated and suspended that water into the fuel.
By design, the water from the CDG-DEL arrival should have been consumed in the DEL-AMD Sector. But desperately clinging to defending my theory (I appreciate this is a hole), lets assume that at DEL the pumps were running for a long time. Lets assume that the pumps allowed the water to be dispersed within the tank prior to being used through the engines. Then - in the DEL-AMD sector, the wing tanks could have picked up more water.
How much water would cause a sustained flameout? I never posited a sustained flameout. I posited a significant reduction in thrust. Listening back to the rooftop video, which at first we were all listening for evidence of RAT, there's also a rhythmic pop-pop-pop of engines struggling. I think the engines were running, albeit badly. Heavily water contaminated fuel will do that. It doesn't have to be 100% water. Just enough water to make the engine lose thrust. Your 2 gallons per second figure assumes the engine running at full flow. I'm not a figures man, I'll not challenge that, I do recall flowmeters at max thrust spin like crazy. But an engine struggling due to a high perrcentage of contamination, is that using 2 gal/sec? or just trying to? What happens if there is e.g. 20% water in the fuel?
There are also reported incidents of engine flameout / thrust reduction that have all happened at altitude. Incidents that have been recovered due to the altitude and time available. I Posited that the engines would have eventually regained full thrust once the contamination worked though. But 30 seconds of rough engine is very different at 40,000 feet than it is at 100 feet.
The theory also relies on a second part - the electrical failure. That the electrical failure causes the fuel supply to switch, a few seconds after the failure. We go, at the point of electrical failure from a pumped centre tank supply to a sucked wing tank supply. It takes time for that different fuel to reach the engine.
Ive written enough and am tired. Must stop now.
June 22, 2025, 12:43:00 GMT
permalink Post: 11908512
The "bigger issue", as you put it, is Boeing company organisational and engineering effectiveness. In this accident, so far, we are looking at (at least) two nominally independent phenomena that inhibited continued safe flight, and nobody has a clue yet (or maybe the investigation team does) how those phenomena can possibly have come to be.
This singular, so far inexplicable, event occurred with an aircraft with over a decade and 30 million hours of use and no previous fatal accidents. Compare that with the A320, which had 5 fatal accidents in its first decade. The Boeing 777 had one (a refuelling incident in Denver in which the fuel operator died). Boeing organisational behaviour has been the subject of two scholarly books, one extensive US Congressional report, and a lot more (most recently since January 2025). There is a lot of information, even very interesting information. What there is not in any of that information (I ask you to take my word for it) is any indication of why two working engines simultaneously suffered serious reduction of thrust shortly after unstick. That is a different topic entirely. And in my opinion it is the topic which belongs in this thread.
Last edited by PBL; 22nd June 2025 at 13:42 . Reason: Brain bit flip: said "miles", obviously meant "hours". Duuh
June 22, 2025, 23:35:00 GMT
permalink Post: 11908907
That\x92s the nature of a common mode bug. If the software was vulnerable to Mars being in the house of Uranus, the scent of lilacs and the DOW being less than 42,000 then you\x92d expect the failure to occur everywhere when these conjoined. Same when an aeroplane\x92s systems and/or the environment present data that triggers an unplanned/unforeseen response in something like an EEC/FADEC. The experts still appear to think that this is unlikely but we have been presented with an unlikely occurrence...
Yes, there may be (let's assume is) "identical" FADEC/TCMA hardware and firmware on both engines, but if the Left Engine is subject to Mars in the house of Uranus (wink wink), then the Right Engine cannot be, maybe it's Venus in the same House. This is simply because the Left engine TCMA 'contraption', I'm going to call it, is monitoring Left Engine Conditions (Shaft Speed, T/L setting / position data - Right or Wrong, and calculating and comparing accordingly against its internal map) while the opposite TCMA "device" is monitoring and calculating etc, Right Engine Conditions. There are some things in common, but (I say) it's virtually impossible for the Engine Conditions being individually monitored to be identical in both engines.
The Thrust Levers are electro-mechanical devices, almost certainly at this stage pushed by a somewhat squishy human hand, likely with a slight offset. What is the probability that those two levers are in identical positions, and even if they are, that the calibration (e.g. "zero points") of both levers are identical, and that the values they output (response slopes/curves) are exactly matching in every matching point in their individual travels? That's just one aspect, but consider the engines. They are different ages. Have different amounts of wear. They have separate fuel metering valves (or other names), separate HP Fuel pumps (and, I guess relief valves?), all also subject to wear), and each has a host of other, correspondingly paired, sensors, (maybe of different makes and certainly of different ages and different calibrations and response curves) from which each FADEC, supposedly independently of the TCMA, adjusts the fuel metering device settings and resulting engine power, and shaft RPMs follow in some other slightly non-matching way.
Sure, I would completely agree that the two engines and their calculated Throttle Lever positions to Shaft RPMs are always going to be similar during normal, matched operation, and they will very likely dance with each other, maybe one 'always' (75% of the time, say) leading during one dance (TO, say) with the other leading in dancing to a different tune (descent, say).
To me, the fact that this appears to have been an almost simultaneous dual engine failure, pretty much, for me, rules out a FADEC/TCMA firmware bug, especially as there don't seem to be any reports of even a single engine mid-air TCMA shutdown.
HOWEVER, and I want to stress this, that doesn't rule out the possibility that both TCMAs shutdown their respective engines simultaneously. Any lack of simultaneity observed would be due to those slight differences in other pieces of hardware, such as the time for one Shutoff valve to close versus the other.
As far as I know, there isn't enough information on what's actually inside those TCMA Black Boxes to say anything for sure, but here's a thought, which I think has been alluded to, or the question asked, here in one or other thread, earlier.
What does the TCMA firmware do when an engine is already running at a high power setting and TWO things occur in quick succession? I suspect this kind of event is a highly probable cause, but these two events have not occurred close enough together, or ever, before.
Imagine this: Plane taking off, Throttle Levers near Full Power, Engines performing correctly, also near Full Power, Rotation etc all normal, plane beginning to climb, positive rate achieved.
Pilot calls GEARUP. GearUp, activated.
The Gear Retract sequence begins. Due to some unforeseen or freshly occurring (maybe intermittent short or open circuit) linkage between the gear Up sequence and the WOW or Air/Ground System, the signal to both TCMAs suddenly switches to GROUND. All "good", so far, as the engine RPMs match the Throttle Lever settings and TCMA doesn't flinch. The plane could be in a Valid Takeoff sequence, so it had better not! But it does make a bit of sense. How is WOW / Air/Ground detected? Somewhere near the Landing Gear, I assume.
HOWEVER, now, a moment later, and perhaps due to a related system response, the Thrust Levers suddenly get pulled back to Idle, whether by man or Machine.
What would you expect the TCMA system to do? I would guess, fairly soon thereafter, two, independent, Fuel Cutoffs... Though I fully admit, I'm guessing based on a severe lack of knowledge of that Firmware.
Ok, no need for further explanation on that point, but I did refer to TCMA unflatteringly as a contraption, earlier. Last night (regrettably, before bed) I started looking at the TCMA Google Patent. Let's just say, so far, I'm aghast! My first impressions are bad ones. How did this patent even get approved? What I suspect here, now, is not a Firmware bug, but a serious Logic and Program Defect. But we'd have to see what's inside the firmware.
When I get more time, I'll dig deeper.
June 29, 2025, 10:48:00 GMT
permalink Post: 11912945
Can anyone suggest a good reason why the captain should issue a Mayday call at that point? The crew should have been extremely busy with the situation. Aviate, Navigate, Communicate is a mantra we are all familiar with. So why communicate?
Having discussed the accident with experienced pilot colleagues, we have all considered that the Egyptair 990 case offered similarities. Yet this is almost a taboo subject.
And one's suspicions are raised by the fact that Air India/Tata are keeping ICAO out of the post-crash investigation.
Incidentally, I sincerely hope that we are wrong about the possibility of a deliberate dual engine shutdown shortly after rotation.
Having discussed the accident with experienced pilot colleagues, we have all considered that the Egyptair 990 case offered similarities. Yet this is almost a taboo subject.
And one's suspicions are raised by the fact that Air India/Tata are keeping ICAO out of the post-crash investigation.
Incidentally, I sincerely hope that we are wrong about the possibility of a deliberate dual engine shutdown shortly after rotation.
June 29, 2025, 14:02:00 GMT
permalink Post: 11913050
I see nothing in the video’s to suggest the aircraft was out of control. It was gliding almost exactly as you can expect from an event starting with the gear down and flaps at 5. As the aircraft nears the ground it appears there is a bit of flare to break the rate of descent. That is exactly what you would expect the pilots to do and their only course of action with a dual engine failure at low altitude.
June 30, 2025, 08:28:00 GMT
permalink Post: 11913431
We know that the right-hand GEnx-1B was removed for overhaul and re-installed in March 2025 so it was at \x93zero time\x94 and zero cycles, meaning a performance asymmetry that the FADEC would have to manage every time maximum thrust is selected. If the old engine was still on the pre-2021 EEC build while the fresh engine carried the post-Service Bulletin software/hardware, a dual \x93commanded rollback\x94 is plausible. A latent fault on one channel with the mid-life core can prompt the other engine to match thrust to maintain symmetry, leading to dual rollback.
Last edited by PPRuNeUser548247; 30th June 2025 at 11:43 .
June 30, 2025, 12:29:00 GMT
permalink Post: 11913592
We know that the right-hand GEnx-1B was removed for overhaul and re-installed in March 2025 so it was at \x93zero time\x94 and zero cycles, meaning a performance asymmetry that the FADEC would have to manage every time maximum thrust is selected. If the old engine was still on the pre-2021 EEC build while the fresh engine carried the post-Service Bulletin software/hardware, a dual \x93commanded rollback\x94 is plausible. A latent fault on one channel with the mid-life core can prompt the other engine to match thrust to maintain symmetry, leading to dual rollback.
June 30, 2025, 13:05:00 GMT
permalink Post: 11913609
India's Minister of State for Civil Aviation appears to be confirming in this
this interview
that the cause of the accident was a dual engine failure. Which is, I think, the first vaguely official confirmation of what happened that has been released? He also confirmed that all the data from the recorders has been downloaded and is being processed by the Indian AAIB, no boxes have been sent abroad.
The 30 day deadline for the preliminary report is July 12th.
The 30 day deadline for the preliminary report is July 12th.