2025-06-21T18:31:00
permalink Post: 11907969
This has been setting a bit wrong with me for the entirety of the thread. When people say "software" they have an image in their head, and in this case the image is rather unhelpful if not outright wrong.
When we say "Software" we mean things like this:
This is the type of software we are usually subject to in our everyday lives basically everywhere. Your phone, your fridge, your oven, your water heater, your car, etc. pp. ad nauseam.
In case of the Safran FADEC 3 this is not actually what we're dealing with. It uses something called an FPGA (Field Programmable Gate Array) which is a very different beast to what we are used to dealing with.
In a normal computer we simply translate code like above into something less human readable but instead interpretable by a microchip or the like. This "machine code" is then run sequentially and does whatever it is meant to do (hopefully). If we were to expand our little happy check above with another condition then our computer would sequentially check these conditions. This opens up a lot of hoopla about timing, especially when interacting with the real world.
Let's say you want to track N1 and N2 in an engine. You'd have to make a measurement, that value would have to go through an AD converter, which writes it to some (likely volatile) storage, where it is then accessed by some sort of (C)PU, transferred to yet another piece of volatile memory and then used in the computation. This takes time because you can't do those things within the same clock cycle.
Unsurprisingly this is rather inconvenient when dealing with the real world and especially when dealing with volatile physical processes that need monitoring. Like a modern turbine engine.
Enter the FPGA.
While it is programmable what that actually means is that (at a very high level) you can build a thing called a truth table, that means a definitive mathematical mapping of input states to output states. Unlike our sequential CPU driven system an FPGA will be able to perform its entire logic every time it is asked to do so. We don't have to wait for our happy check to perform any other check.
This is very useful for our Turbine Engine, because now we can verify that N2 is smaller than X without delaying our check that the Throttle Control Lever setting is within acceptable distance to N1 and N2, while also creating the appropriate output for all the different bypasses and bleed valves in a modern engine, and so on. The Safran FADEC 3 does this "up to 70 times per second" as per the vendor.
In order to be manageable the actual FADEC consists of multiple different pieces, many of which are FPGAs. At least two are used for so called "input conditioning". This is where we deal with converting messy real world values from sensors subject to real physics into nice and clean numbers. Here our values from the Throttle Control Levers come in, the signal from our N1 and N2 sensors, WOW signal, and on, and on, and on.
This is then changed into logic level signals provided to a different set of FPGAs. Lacking an actual schematic I suspect that this is what sometimes is referred to as "channels". The channel could consist of both a signal conditioning stage and then a logic stage or (more likely) redundant signal conditioning stages feed separately into separate FPGA "channels" are evaluated and then the end result is likely put into yet another FPGA for control output generation.
Why is this important?
Because it is basically impossible for a "bug" to exist in this system. These systems are the epitome of a dumb computer. They do _precisely_ what they are meant to do. The TCMA is simply a set of conditions evaluated just like any other condition in those FPGAs. If there is an "error" then it is in the requirements that lead to the truth table used by these devices but never in the truth table itself. In fact these types of computation device are used precisely because of that very property.
So when people say "the FADEC software" (ie TCMA) has "failed" or "has a bug" what they're really saying is:
Regards,
Justus
When we say "Software" we mean things like this:
This is the type of software we are usually subject to in our everyday lives basically everywhere. Your phone, your fridge, your oven, your water heater, your car, etc. pp. ad nauseam.
In case of the Safran FADEC 3 this is not actually what we're dealing with. It uses something called an FPGA (Field Programmable Gate Array) which is a very different beast to what we are used to dealing with.
In a normal computer we simply translate code like above into something less human readable but instead interpretable by a microchip or the like. This "machine code" is then run sequentially and does whatever it is meant to do (hopefully). If we were to expand our little happy check above with another condition then our computer would sequentially check these conditions. This opens up a lot of hoopla about timing, especially when interacting with the real world.
Let's say you want to track N1 and N2 in an engine. You'd have to make a measurement, that value would have to go through an AD converter, which writes it to some (likely volatile) storage, where it is then accessed by some sort of (C)PU, transferred to yet another piece of volatile memory and then used in the computation. This takes time because you can't do those things within the same clock cycle.
Unsurprisingly this is rather inconvenient when dealing with the real world and especially when dealing with volatile physical processes that need monitoring. Like a modern turbine engine.
Enter the FPGA.
While it is programmable what that actually means is that (at a very high level) you can build a thing called a truth table, that means a definitive mathematical mapping of input states to output states. Unlike our sequential CPU driven system an FPGA will be able to perform its entire logic every time it is asked to do so. We don't have to wait for our happy check to perform any other check.
This is very useful for our Turbine Engine, because now we can verify that N2 is smaller than X without delaying our check that the Throttle Control Lever setting is within acceptable distance to N1 and N2, while also creating the appropriate output for all the different bypasses and bleed valves in a modern engine, and so on. The Safran FADEC 3 does this "up to 70 times per second" as per the vendor.
In order to be manageable the actual FADEC consists of multiple different pieces, many of which are FPGAs. At least two are used for so called "input conditioning". This is where we deal with converting messy real world values from sensors subject to real physics into nice and clean numbers. Here our values from the Throttle Control Levers come in, the signal from our N1 and N2 sensors, WOW signal, and on, and on, and on.
This is then changed into logic level signals provided to a different set of FPGAs. Lacking an actual schematic I suspect that this is what sometimes is referred to as "channels". The channel could consist of both a signal conditioning stage and then a logic stage or (more likely) redundant signal conditioning stages feed separately into separate FPGA "channels" are evaluated and then the end result is likely put into yet another FPGA for control output generation.
Why is this important?
Because it is basically impossible for a "bug" to exist in this system. These systems are the epitome of a dumb computer. They do _precisely_ what they are meant to do. The TCMA is simply a set of conditions evaluated just like any other condition in those FPGAs. If there is an "error" then it is in the requirements that lead to the truth table used by these devices but never in the truth table itself. In fact these types of computation device are used precisely because of that very property.
So when people say "the FADEC software" (ie TCMA) has "failed" or "has a bug" what they're really saying is:
The conditions that were experienced in the real world by the system were incorrectly assessed in the system requirements and lead to an undesired output state when taking into account the real world result we would have preferred.
A bit of a mouth full, granted, but an important mouth full. This simply wouldn't be someone missing a semicolon somewhere in a thousand lines of code.
Regards,
Justus
2025-06-21T20:40:00
permalink Post: 11908040
Sadly I'm unable to post links or I would have done so in my original post.
First off I should note that what I wrote is a massive oversimplification. I'd expect that neat metal box with the nice rugged connectors that Safran shows on their homepage to contain an ungodly mess of microcontrollers, ICs for signal conditioning, the FPGAs I mentioned and components that are well beyond my understanding of electronics. But from personal experience with a certain aircraft related company operating in Munich I can tell you that marketing level slides like you showed are so far removed from the technical reality that they might as well be fan-fic.
I agree with your observation that current microcontrollers are more than capable enough of dealing with these types of real time computation, but keep in mind that the Boeing 787, and thus its FADEC, had its first flight in 2009. Back then I would be rather more careful with that statement. Mostly because I would have just started my Computer Science Degree at the time XD
But maybe I should explain how I've arrived at the post above. I've basically spent the past few days on and off scavenging anything I could find on the 787 FADEC specifically. It was mentioned in this thread or the old one that the 787 uses the Safran FADEC 3, so my primary jumping off point was the Safran website and the information available on it. That plus a couple of press releases namely by Actel about their flash-based ProASIC3 and ProASICPLUS FPGAs was the evidence I used to arrive at my conclusions above. I can't for the life of me find the quote about the dual signal conditioning FPGAs anymore, but they are in there as far as I can tell.
Overall information on the actual inner workings of these things are so sparse that outside of a few patents (keyword: configurable CPU) I was hard pressed to find anything at all to be honest. This all not being helped by me now finding references to my own post, thus having created my very own Woozle. Hooray.
I have only looked into the Safran FADEC 3 used on the 787 with the
GEnx-1B engine. The system was very innovative for its time and uses technology not previously used on FADECs to my knowledge. Even saying that, this could be one of those "technically true" kind of statements. My intention was to highlight that due to the totality of the processes and architectural style of these types of computational device the typical kind of "software error" or "bug" was basically impossible and that we'd be looking at something way further back with implications for the base requirements. I merely want to correct the implied misconceptions about the "software" being used here. As I said, FPGAs are just very different kinds of beasts.
I half agree and half disagree. Obviously I shouldn't have stated an absolute because, yes, it is possible despite multiple layers of engineering coordination that an error occurs at one step or another and is not caught by the multiple layers of checking. My point is that even in normal development FPGAs are well known to be bug resistant due to the way they are programmed. In general we create the logic condition tables using software which then translates our table into an actual FPGA configuration. This process is obviously not immune to faults but highly resistant and most importantly: Can easily be simulated.
Now, in normal "everyday" usecases I'd fully agree with you, but in case of the FADEC we're talking about an FPGA with logic condition tables where every line is likely to have its own separate binder of engineering notes, discussions and recommendations. It is there where any "error" or "bug" is most likely to reside by a considerable margin.
They were, and continue to be, a valid and reliable technology for this type of application especially before the extreme proliferation of consumer microelectronics in the last 20 years. The Safran FADEC 3 was designed at basically the same time as the first iPhone. And boy do I feel old now.
I realize that my post is a massive simplification of the topic, considering the intricate details that are simply unknown to the general public in a system as complex as a FADEC I certainly agree that things can go wrong. What I wanted to point out is the reliability of the last steps versus the design and requirements engineering phase. In general software development bugs will happen at any stage with basically equal likelihood. Given the way the standard processes around FPGA development alone would influence this probability towards the design and requirements engineering stage and then combining it with how it is handled in the Aerospace Sector I felt it was justifiable to state the extreme mismatch in probabilities of error the way I did. With the system being in operation for this length of time I do still feel comfortable with my original statement although I would now decorate it with a little * for the reasons mentioned by you and others.
In the end I hope it was useful for understanding where the reluctance of apportioning blame to these systems originates and at the same time talk about some pretty nifty tech that I know few Software Engineers will ever deal with in their entire career. I do still find the idea of a fault (with the understanding above of where that fault originates) in the FADEC a convincing possibility, it's merely the nature and location of that fault that I felt could benefit from some more specifics in regards to the nature of how they were most likely to actually happen. I would love to actually hear more about how the FADEC 3 is structured internally and which components it uses, but I suspect that those who know are under NDA.
Regards,
Justus
I ask because when I look for info about FADECs generally and FADEC 3 more specifically, I find this kind of documentation (notice the mention of FADECs in the lower left corner):
... which is part of a larger Powerpoint presentation by Ansys, explaining that these products are developed with SCADE development workbench, generating either Ada or C code, and that the resulting code runs under a microkernel realtime operating system:
... which is part of a larger Powerpoint presentation by Ansys, explaining that these products are developed with SCADE development workbench, generating either Ada or C code, and that the resulting code runs under a microkernel realtime operating system:
Now, obviously enough, a CPU can be embedded in an application-specific FPGA, but it would still execute machine code. And from my experience in other embedded systems development, current CISC or RISC CPUs have more than enough computation power to implement command and control on a modern turbofan.
But maybe I should explain how I've arrived at the post above. I've basically spent the past few days on and off scavenging anything I could find on the 787 FADEC specifically. It was mentioned in this thread or the old one that the 787 uses the Safran FADEC 3, so my primary jumping off point was the Safran website and the information available on it. That plus a couple of press releases namely by Actel about their flash-based ProASIC3 and ProASICPLUS FPGAs was the evidence I used to arrive at my conclusions above. I can't for the life of me find the quote about the dual signal conditioning FPGAs anymore, but they are in there as far as I can tell.
Overall information on the actual inner workings of these things are so sparse that outside of a few patents (keyword: configurable CPU) I was hard pressed to find anything at all to be honest. This all not being helped by me now finding references to my own post, thus having created my very own Woozle. Hooray.
Whilst you\x92re right to highlight the type of software in a FADEC, it should also be recalled that after the dual uncommanded engine shutdown event on the Baltic CS300, the FAA mandated updated FADEC software for all of a family of P&W engines. FADEC s/w can contain what we\x92d call bugs (whether they\x92re mistakes of coding or mistakes of logic in the design assumptions).
"it is basically impossible for a "bug" to exist in this system." This is not my experience.There are a lot of bugs in hardware and FPGAs. The guys doing this are real experts and bugs are less common than in software. But they do exist and they can be difficult to reproduce and subtle to diagnose. For someone new to FPGA it can be very intimidating because everything is parallel. It requires a different way of thinking. You can execute 1000 instructions simultaneously.Then various tricks to speed things up or reduce glitches add complexity. For example you may want to count in gray code or one hot instead of binary. Writing bug free FPGA code is very challenging. We cannot assume the FPGA code is free of bugs.
Now, in normal "everyday" usecases I'd fully agree with you, but in case of the FADEC we're talking about an FPGA with logic condition tables where every line is likely to have its own separate binder of engineering notes, discussions and recommendations. It is there where any "error" or "bug" is most likely to reside by a considerable margin.
Now, I am sure that the state of the art has moved on a lot since I was heavily involved in this sort of engineering but I can say that when my colleagues were prototyping ASICs (which are application specific devices that are not programmable like an FPGA) using FPGAs because it can be reprogrammed to cure bugs found in the code that created it I am absolutely sure that it is possible for a latent bug to exist in what has been built despite extensive testing being carried out. I well remember sitting down with an FPGA designer and a software engineer and showing them a situation where one of my radio devices was not being commanded to transmit, they refused to believe me until we captured their output signals in the failed state and I was thus able to prove to them that it was their logic and code that was failing and that my radio system was fully working except it couldn't possibly transmit when its enable line was inactive.
I would therefore never state that an FPGA-based FADEC is infallible. The devil will be in the detail of the functions contained within the devices and how they interact with each other and how the original logical functions are described and specified and then coded and checked for logical errors before getting on to the actual physical reality of the FPGA manufacturer's design and development system that bolts on to the back of the source code. If the FPGA devices are being pushed anywhere near the edge of their performance envelope, just like an aircraft things can go wrong. If a design begins with a device that is only just large enough in terms of how many logic blocks and routing channels are available for the logic required it's almost a certainty that development will take it to this area of its performance and a lot of tweaking may be needed which means that even more testing will be needed to be reasonable sure that it does what it is supposed to.
I would therefore never state that an FPGA-based FADEC is infallible. The devil will be in the detail of the functions contained within the devices and how they interact with each other and how the original logical functions are described and specified and then coded and checked for logical errors before getting on to the actual physical reality of the FPGA manufacturer's design and development system that bolts on to the back of the source code. If the FPGA devices are being pushed anywhere near the edge of their performance envelope, just like an aircraft things can go wrong. If a design begins with a device that is only just large enough in terms of how many logic blocks and routing channels are available for the logic required it's almost a certainty that development will take it to this area of its performance and a lot of tweaking may be needed which means that even more testing will be needed to be reasonable sure that it does what it is supposed to.
In the end I hope it was useful for understanding where the reluctance of apportioning blame to these systems originates and at the same time talk about some pretty nifty tech that I know few Software Engineers will ever deal with in their entire career. I do still find the idea of a fault (with the understanding above of where that fault originates) in the FADEC a convincing possibility, it's merely the nature and location of that fault that I felt could benefit from some more specifics in regards to the nature of how they were most likely to actually happen. I would love to actually hear more about how the FADEC 3 is structured internally and which components it uses, but I suspect that those who know are under NDA.
Regards,
Justus
3 users liked this post.
2025-06-22T00:15:00
permalink Post: 11908173
I don't want to refute your theory, but given your 30 years of experience---presuming it is relevant--I'd ask you to clarify a few things.
First, water in fuel is not a novel concept and I would presume that the designers of the 787 knew about it. You've simply stated that water might collect and settle out, but how much water might you expect under those conditions (57% humidity doesn't seem terribly high to me) and what features and procedures are already there to mitigage water contamination issues? Your theory would imply that there basically aren't any. IDK how the tank venting system works, but the idea that some huge amount of water could have condensed in the tank from the outside seems preposterous.
Second, how much water do you think it would take to cause a sustained flameout in one of those engines? Remember that they have automatic continous relight, so you're going to have to sustain your flame suppression long enough for them to wind down completely. I think those engines were probably using something like 2 gallons per second of fuel along with 250lbs of air heated to over 1100F. Any fuel in the mix would burn and the water would be converted to steam so you'd need mostly water for a long time. So if you think a hundred gallons of water could have gotten into each tank then perhaps I'd buy your theory--which, btw, does fit the known facts pretty well. But I think that short of some woeful neglect, Boeing and AI already know about and have methods of dealing with water contamination. At least I hope so.
First, water in fuel is not a novel concept and I would presume that the designers of the 787 knew about it. You've simply stated that water might collect and settle out, but how much water might you expect under those conditions (57% humidity doesn't seem terribly high to me) and what features and procedures are already there to mitigage water contamination issues? Your theory would imply that there basically aren't any. IDK how the tank venting system works, but the idea that some huge amount of water could have condensed in the tank from the outside seems preposterous.
Second, how much water do you think it would take to cause a sustained flameout in one of those engines? Remember that they have automatic continous relight, so you're going to have to sustain your flame suppression long enough for them to wind down completely. I think those engines were probably using something like 2 gallons per second of fuel along with 250lbs of air heated to over 1100F. Any fuel in the mix would burn and the water would be converted to steam so you'd need mostly water for a long time. So if you think a hundred gallons of water could have gotten into each tank then perhaps I'd buy your theory--which, btw, does fit the known facts pretty well. But I think that short of some woeful neglect, Boeing and AI already know about and have methods of dealing with water contamination. At least I hope so.
Experience. Without wishing to dox myself, I've worked in engineering at a major airline from apprentice through (in no particular order) Line Maintenance, Heavy and Light Maintenance, to technical support and maintenance control on both Boeing and Airbus products, with various qualifications and authorisations along the way. [Hmm - Scrap this sentence?]On the day 9/11 occurred, I should have been making modifications inside a fuel tank instead of staring at the TV with mouth on the floor.
However, I would describe my experience as broad, yet shallow in respect to this incident. Some of my fleet I know every rivet. Some of my fleet I've only ever seen from a distance. I don't touch airplanes for a living any more. B787 though - is not my area of specialty. I'll dig in, but am not the expert. I am not a systems design engineer, so precise numbers and flow rates, are not what I do. But what the systems do, how they operate, what they look like, smell and taste like... yeah, I'm not a muggle. And I do have access to all the manuals and know how to use them. And - let me be clear, I am speculating. I was advancing a theory. It WILL be some flavour of wrong. The investigation will reveal all.
I Agree, Water in fuel is not a novel concept. Aircraft fuel tanks attract water - fact. How much? It varies. I've sumped tanks and got no water, I've seen drops of water beading about in the bottom of a gallon jug, I've seen gallons of water. I've been so covered in fuel I cant smell it or think straight and taken gallon after gallon not being able to tell if its fuel or water. I also agree that 57% humidity doesn't seem particularly high - its not south east Asian jungle levels - but I'm not an expert at humidity, 32Deg c at 57% humidity at 02:30 am is not going to be comfortable for me though. I looked at recent weather in DEL, and those values were at the higher end of the range.
Further, I believe the prevailing weather conditions on the ground are less important when it comes to the volume of water getting in. Fuel is cold, or gets damn cold during a 9 Hr flight. Fuel Temperature Management is an issue for our Drivers. So as the fuel is used at altitude, Air enters the tank through NACA Ducts in the outboard end of the wing. Its beneficial to maintain a slight positive pressure, amongst other things to reduce evaporation. (Added complication, there is also the Nitrogen Enrichment system due to TWA800 - but that's more about processing the air in the tank to change the properties and make it non-explosive). Then as the aircraft descends, more air enters as the air pressure increases. Its the humidity of that air in the descent that is going to determine the volume of water entering the tank and potentially the fuel. The water in the air condenses on the sides of the tank because of the cold post-flight fuel. It doesn't dissolve into the fuel, but sinks to the bottom. Ground temperature / humidity and time will likely affect how much water condenses out of that air while on the ground. There won't be a huge amount of air exchange on the ground. Likely if the AC landed at 2am, then from sunrise as the tank warmed up, there would actually be a flow out of the vents.
What Features and procedures are there to mitigate Water? I apologise if my post gave the impression that there are no mitigation processes. There are. Water is well understood in the industry.
Well for a start, Features / Design. The Aircraft has a water scavenge system. Water doesn't mix with fuel, it sinks to the bottom being about 20% denser than fuel, so at the very lowest point in the tank, the water scavenge system (Powered by the Aft Fuel Pump through a jet pump, a venturi like system) will suck up the 'fluid' at the very lowest point, where the water would collect and in Boeings words 'drip' that fluid into the path of the pump pickup inlet (but I'd describe it more as a 'squirt'). The idea being that a small amount of water injected into the fuel will be consumed by the engines harmlessly.
There is also agitation. The wing tank pumps are pretty much running constantly, from before engine startup to after engine shutdown. The pumps are quite violent to the fuel and supply more pressure then the engine could ever need. Any excess pressure is dumped right back into the tank, quite close to the pump, in a direction that would further stir up the fuel and help break up any water into suspended droplets.
This all works if there is a small amount of water in the fuel. The water scavenge pickup is right next to the pump inlet, but a bit lower. Little bits of water get managed. Looking at the pictures of the system, I'd say a couple of gallons of water would do no harm at all.
But if there was significantly more water in that tank. Guessing 10-30 + gallons, then the pump would be circulating water, or highly water rich fuel.
Then there's the suction pickup. Its in the same 'bay' as the aft fuel pump and located a little 'higher' than the pump inlet and water scavenge inlet. But also located between stringers that can separate out the settled water ( I wish I could share the pictures, but more than my job is worth ) I can imagine the suction pickup being in a pool of 'stagnant' water.
I also saw a post from Metcha about the scavenge system blocking with Algae - I don't know about that (B787 not my fleet). But possible that could aggravate things. There's also the reports of the Indian AAIB looking at the Titan Biocide incident. Its possible that might be related and could modify the theory.
Procedures - There's the (at my airline weekly I think) procedure to 'sump' the tanks. There are drain points in the tank. Valves that you can push in with a tool and fluid drains. As described earlier (and videos exist on YouTube), you drain about a gallon of fluid and examine it for water. Most often in temperate climates (my experience), there's a few 'beads' of water in the bottom of the jug, moving about like mercury. Except when there's more. Sometimes there's a clear line in the jug, half water, fuel above. And sometimes a gallon of water, that smells like fuel. You drain it until you are sure there's no water.
Could 'that much' water have condensed in the tank? Well - There's the question. I guess the basis of the theory is that on descent into DEL, the wing tanks picked up some very humid air, which settled water into the tanks through the night. Then, as the theory I posited must work, the wing pumps must have circulated and suspended that water into the fuel.
By design, the water from the CDG-DEL arrival should have been consumed in the DEL-AMD Sector. But desperately clinging to defending my theory (I appreciate this is a hole), lets assume that at DEL the pumps were running for a long time. Lets assume that the pumps allowed the water to be dispersed within the tank prior to being used through the engines. Then - in the DEL-AMD sector, the wing tanks could have picked up more water.
How much water would cause a sustained flameout? I never posited a sustained flameout. I posited a significant reduction in thrust. Listening back to the rooftop video, which at first we were all listening for evidence of RAT, there's also a rhythmic pop-pop-pop of engines struggling. I think the engines were running, albeit badly. Heavily water contaminated fuel will do that. It doesn't have to be 100% water. Just enough water to make the engine lose thrust. Your 2 gallons per second figure assumes the engine running at full flow. I'm not a figures man, I'll not challenge that, I do recall flowmeters at max thrust spin like crazy. But an engine struggling due to a high perrcentage of contamination, is that using 2 gal/sec? or just trying to? What happens if there is e.g. 20% water in the fuel?
There are also reported incidents of engine flameout / thrust reduction that have all happened at altitude. Incidents that have been recovered due to the altitude and time available. I Posited that the engines would have eventually regained full thrust once the contamination worked though. But 30 seconds of rough engine is very different at 40,000 feet than it is at 100 feet.
The theory also relies on a second part - the electrical failure. That the electrical failure causes the fuel supply to switch, a few seconds after the failure. We go, at the point of electrical failure from a pumped centre tank supply to a sucked wing tank supply. It takes time for that different fuel to reach the engine.
Ive written enough and am tired. Must stop now.
11 users liked this post.
2025-06-29T10:48:00
permalink Post: 11912945
Can anyone suggest a good reason why the captain should issue a Mayday call at that point? The crew should have been extremely busy with the situation. Aviate, Navigate, Communicate is a mantra we are all familiar with. So why communicate?
Having discussed the accident with experienced pilot colleagues, we have all considered that the Egyptair 990 case offered similarities. Yet this is almost a taboo subject.
And one's suspicions are raised by the fact that Air India/Tata are keeping ICAO out of the post-crash investigation.
Incidentally, I sincerely hope that we are wrong about the possibility of a deliberate dual engine shutdown shortly after rotation.
Having discussed the accident with experienced pilot colleagues, we have all considered that the Egyptair 990 case offered similarities. Yet this is almost a taboo subject.
And one's suspicions are raised by the fact that Air India/Tata are keeping ICAO out of the post-crash investigation.
Incidentally, I sincerely hope that we are wrong about the possibility of a deliberate dual engine shutdown shortly after rotation.
4 users liked this post.
2025-06-30T22:21:00
permalink Post: 11913922
OK - Fair Challenges - good post, I'll have a go at answering and simultaneously expanding my own thoughts. In fact I'm not having a go at you, I'm more working my theory....
Experience. Without wishing to dox myself, I've worked in engineering at a major airline from apprentice through (in no particular order) Line Maintenance, Heavy and Light Maintenance, to technical support and maintenance control on both Boeing and Airbus products, with various qualifications and authorisations along the way. [Hmm - Scrap this sentence?]On the day 9/11 occurred, I should have been making modifications inside a fuel tank instead of staring at the TV with mouth on the floor.
However, I would describe my experience as broad, yet shallow in respect to this incident. Some of my fleet I know every rivet. Some of my fleet I've only ever seen from a distance. I don't touch airplanes for a living any more. B787 though - is not my area of specialty. I'll dig in, but am not the expert. I am not a systems design engineer, so precise numbers and flow rates, are not what I do. But what the systems do, how they operate, what they look like, smell and taste like... yeah, I'm not a muggle. And I do have access to all the manuals and know how to use them. And - let me be clear, I am speculating. I was advancing a theory. It WILL be some flavour of wrong. The investigation will reveal all.
I Agree, Water in fuel is not a novel concept. Aircraft fuel tanks attract water - fact. How much? It varies. I've sumped tanks and got no water, I've seen drops of water beading about in the bottom of a gallon jug, I've seen gallons of water. I've been so covered in fuel I cant smell it or think straight and taken gallon after gallon not being able to tell if its fuel or water. I also agree that 57% humidity doesn't seem particularly high - its not south east Asian jungle levels - but I'm not an expert at humidity, 32Deg c at 57% humidity at 02:30 am is not going to be comfortable for me though. I looked at recent weather in DEL, and those values were at the higher end of the range.
Further, I believe the prevailing weather conditions on the ground are less important when it comes to the volume of water getting in. Fuel is cold, or gets damn cold during a 9 Hr flight. Fuel Temperature Management is an issue for our Drivers. So as the fuel is used at altitude, Air enters the tank through NACA Ducts in the outboard end of the wing. Its beneficial to maintain a slight positive pressure, amongst other things to reduce evaporation. (Added complication, there is also the Nitrogen Enrichment system due to TWA800 - but that's more about processing the air in the tank to change the properties and make it non-explosive). Then as the aircraft descends, more air enters as the air pressure increases. Its the humidity of that air in the descent that is going to determine the volume of water entering the tank and potentially the fuel. The water in the air condenses on the sides of the tank because of the cold post-flight fuel. It doesn't dissolve into the fuel, but sinks to the bottom. Ground temperature / humidity and time will likely affect how much water condenses out of that air while on the ground. There won't be a huge amount of air exchange on the ground. Likely if the AC landed at 2am, then from sunrise as the tank warmed up, there would actually be a flow out of the vents.
What Features and procedures are there to mitigate Water? I apologise if my post gave the impression that there are no mitigation processes. There are. Water is well understood in the industry.
Well for a start, Features / Design. The Aircraft has a water scavenge system. Water doesn't mix with fuel, it sinks to the bottom being about 20% denser than fuel, so at the very lowest point in the tank, the water scavenge system (Powered by the Aft Fuel Pump through a jet pump, a venturi like system) will suck up the 'fluid' at the very lowest point, where the water would collect and in Boeings words 'drip' that fluid into the path of the pump pickup inlet (but I'd describe it more as a 'squirt'). The idea being that a small amount of water injected into the fuel will be consumed by the engines harmlessly.
There is also agitation. The wing tank pumps are pretty much running constantly, from before engine startup to after engine shutdown. The pumps are quite violent to the fuel and supply more pressure then the engine could ever need. Any excess pressure is dumped right back into the tank, quite close to the pump, in a direction that would further stir up the fuel and help break up any water into suspended droplets.
This all works if there is a small amount of water in the fuel. The water scavenge pickup is right next to the pump inlet, but a bit lower. Little bits of water get managed. Looking at the pictures of the system, I'd say a couple of gallons of water would do no harm at all.
But if there was significantly more water in that tank. Guessing 10-30 + gallons, then the pump would be circulating water, or highly water rich fuel.
Then there's the suction pickup. Its in the same 'bay' as the aft fuel pump and located a little 'higher' than the pump inlet and water scavenge inlet. But also located between stringers that can separate out the settled water ( I wish I could share the pictures, but more than my job is worth ) I can imagine the suction pickup being in a pool of 'stagnant' water.
I also saw a post from Metcha about the scavenge system blocking with Algae - I don't know about that (B787 not my fleet). But possible that could aggravate things. There's also the reports of the Indian AAIB looking at the Titan Biocide incident. Its possible that might be related and could modify the theory.
Procedures - There's the (at my airline weekly I think) procedure to 'sump' the tanks. There are drain points in the tank. Valves that you can push in with a tool and fluid drains. As described earlier (and videos exist on YouTube), you drain about a gallon of fluid and examine it for water. Most often in temperate climates (my experience), there's a few 'beads' of water in the bottom of the jug, moving about like mercury. Except when there's more. Sometimes there's a clear line in the jug, half water, fuel above. And sometimes a gallon of water, that smells like fuel. You drain it until you are sure there's no water.
Could 'that much' water have condensed in the tank? Well - There's the question. I guess the basis of the theory is that on descent into DEL, the wing tanks picked up some very humid air, which settled water into the tanks through the night. Then, as the theory I posited must work, the wing pumps must have circulated and suspended that water into the fuel.
By design, the water from the CDG-DEL arrival should have been consumed in the DEL-AMD Sector. But desperately clinging to defending my theory (I appreciate this is a hole), lets assume that at DEL the pumps were running for a long time. Lets assume that the pumps allowed the water to be dispersed within the tank prior to being used through the engines. Then - in the DEL-AMD sector, the wing tanks could have picked up more water.
How much water would cause a sustained flameout? I never posited a sustained flameout. I posited a significant reduction in thrust. Listening back to the rooftop video, which at first we were all listening for evidence of RAT, there's also a rhythmic pop-pop-pop of engines struggling. I think the engines were running, albeit badly. Heavily water contaminated fuel will do that. It doesn't have to be 100% water. Just enough water to make the engine lose thrust. Your 2 gallons per second figure assumes the engine running at full flow. I'm not a figures man, I'll not challenge that, I do recall flowmeters at max thrust spin like crazy. But an engine struggling due to a high perrcentage of contamination, is that using 2 gal/sec? or just trying to? What happens if there is e.g. 20% water in the fuel?
There are also reported incidents of engine flameout / thrust reduction that have all happened at altitude. Incidents that have been recovered due to the altitude and time available. I Posited that the engines would have eventually regained full thrust once the contamination worked though. But 30 seconds of rough engine is very different at 40,000 feet than it is at 100 feet.
The theory also relies on a second part - the electrical failure. That the electrical failure causes the fuel supply to switch, a few seconds after the failure. We go, at the point of electrical failure from a pumped centre tank supply to a sucked wing tank supply. It takes time for that different fuel to reach the engine.
Ive written enough and am tired. Must stop now.
Experience. Without wishing to dox myself, I've worked in engineering at a major airline from apprentice through (in no particular order) Line Maintenance, Heavy and Light Maintenance, to technical support and maintenance control on both Boeing and Airbus products, with various qualifications and authorisations along the way. [Hmm - Scrap this sentence?]On the day 9/11 occurred, I should have been making modifications inside a fuel tank instead of staring at the TV with mouth on the floor.
However, I would describe my experience as broad, yet shallow in respect to this incident. Some of my fleet I know every rivet. Some of my fleet I've only ever seen from a distance. I don't touch airplanes for a living any more. B787 though - is not my area of specialty. I'll dig in, but am not the expert. I am not a systems design engineer, so precise numbers and flow rates, are not what I do. But what the systems do, how they operate, what they look like, smell and taste like... yeah, I'm not a muggle. And I do have access to all the manuals and know how to use them. And - let me be clear, I am speculating. I was advancing a theory. It WILL be some flavour of wrong. The investigation will reveal all.
I Agree, Water in fuel is not a novel concept. Aircraft fuel tanks attract water - fact. How much? It varies. I've sumped tanks and got no water, I've seen drops of water beading about in the bottom of a gallon jug, I've seen gallons of water. I've been so covered in fuel I cant smell it or think straight and taken gallon after gallon not being able to tell if its fuel or water. I also agree that 57% humidity doesn't seem particularly high - its not south east Asian jungle levels - but I'm not an expert at humidity, 32Deg c at 57% humidity at 02:30 am is not going to be comfortable for me though. I looked at recent weather in DEL, and those values were at the higher end of the range.
Further, I believe the prevailing weather conditions on the ground are less important when it comes to the volume of water getting in. Fuel is cold, or gets damn cold during a 9 Hr flight. Fuel Temperature Management is an issue for our Drivers. So as the fuel is used at altitude, Air enters the tank through NACA Ducts in the outboard end of the wing. Its beneficial to maintain a slight positive pressure, amongst other things to reduce evaporation. (Added complication, there is also the Nitrogen Enrichment system due to TWA800 - but that's more about processing the air in the tank to change the properties and make it non-explosive). Then as the aircraft descends, more air enters as the air pressure increases. Its the humidity of that air in the descent that is going to determine the volume of water entering the tank and potentially the fuel. The water in the air condenses on the sides of the tank because of the cold post-flight fuel. It doesn't dissolve into the fuel, but sinks to the bottom. Ground temperature / humidity and time will likely affect how much water condenses out of that air while on the ground. There won't be a huge amount of air exchange on the ground. Likely if the AC landed at 2am, then from sunrise as the tank warmed up, there would actually be a flow out of the vents.
What Features and procedures are there to mitigate Water? I apologise if my post gave the impression that there are no mitigation processes. There are. Water is well understood in the industry.
Well for a start, Features / Design. The Aircraft has a water scavenge system. Water doesn't mix with fuel, it sinks to the bottom being about 20% denser than fuel, so at the very lowest point in the tank, the water scavenge system (Powered by the Aft Fuel Pump through a jet pump, a venturi like system) will suck up the 'fluid' at the very lowest point, where the water would collect and in Boeings words 'drip' that fluid into the path of the pump pickup inlet (but I'd describe it more as a 'squirt'). The idea being that a small amount of water injected into the fuel will be consumed by the engines harmlessly.
There is also agitation. The wing tank pumps are pretty much running constantly, from before engine startup to after engine shutdown. The pumps are quite violent to the fuel and supply more pressure then the engine could ever need. Any excess pressure is dumped right back into the tank, quite close to the pump, in a direction that would further stir up the fuel and help break up any water into suspended droplets.
This all works if there is a small amount of water in the fuel. The water scavenge pickup is right next to the pump inlet, but a bit lower. Little bits of water get managed. Looking at the pictures of the system, I'd say a couple of gallons of water would do no harm at all.
But if there was significantly more water in that tank. Guessing 10-30 + gallons, then the pump would be circulating water, or highly water rich fuel.
Then there's the suction pickup. Its in the same 'bay' as the aft fuel pump and located a little 'higher' than the pump inlet and water scavenge inlet. But also located between stringers that can separate out the settled water ( I wish I could share the pictures, but more than my job is worth ) I can imagine the suction pickup being in a pool of 'stagnant' water.
I also saw a post from Metcha about the scavenge system blocking with Algae - I don't know about that (B787 not my fleet). But possible that could aggravate things. There's also the reports of the Indian AAIB looking at the Titan Biocide incident. Its possible that might be related and could modify the theory.
Procedures - There's the (at my airline weekly I think) procedure to 'sump' the tanks. There are drain points in the tank. Valves that you can push in with a tool and fluid drains. As described earlier (and videos exist on YouTube), you drain about a gallon of fluid and examine it for water. Most often in temperate climates (my experience), there's a few 'beads' of water in the bottom of the jug, moving about like mercury. Except when there's more. Sometimes there's a clear line in the jug, half water, fuel above. And sometimes a gallon of water, that smells like fuel. You drain it until you are sure there's no water.
Could 'that much' water have condensed in the tank? Well - There's the question. I guess the basis of the theory is that on descent into DEL, the wing tanks picked up some very humid air, which settled water into the tanks through the night. Then, as the theory I posited must work, the wing pumps must have circulated and suspended that water into the fuel.
By design, the water from the CDG-DEL arrival should have been consumed in the DEL-AMD Sector. But desperately clinging to defending my theory (I appreciate this is a hole), lets assume that at DEL the pumps were running for a long time. Lets assume that the pumps allowed the water to be dispersed within the tank prior to being used through the engines. Then - in the DEL-AMD sector, the wing tanks could have picked up more water.
How much water would cause a sustained flameout? I never posited a sustained flameout. I posited a significant reduction in thrust. Listening back to the rooftop video, which at first we were all listening for evidence of RAT, there's also a rhythmic pop-pop-pop of engines struggling. I think the engines were running, albeit badly. Heavily water contaminated fuel will do that. It doesn't have to be 100% water. Just enough water to make the engine lose thrust. Your 2 gallons per second figure assumes the engine running at full flow. I'm not a figures man, I'll not challenge that, I do recall flowmeters at max thrust spin like crazy. But an engine struggling due to a high perrcentage of contamination, is that using 2 gal/sec? or just trying to? What happens if there is e.g. 20% water in the fuel?
There are also reported incidents of engine flameout / thrust reduction that have all happened at altitude. Incidents that have been recovered due to the altitude and time available. I Posited that the engines would have eventually regained full thrust once the contamination worked though. But 30 seconds of rough engine is very different at 40,000 feet than it is at 100 feet.
The theory also relies on a second part - the electrical failure. That the electrical failure causes the fuel supply to switch, a few seconds after the failure. We go, at the point of electrical failure from a pumped centre tank supply to a sucked wing tank supply. It takes time for that different fuel to reach the engine.
Ive written enough and am tired. Must stop now.
Last edited by Senior Pilot; 30th Jun 2025 at 23:01 . Reason: Quote from a week ago; this is not a Hamsterwheel thread, thanks!