Posts about: "TCMA (Shutdown)" [Posts: 39 Pages: 2]

OK, another hour spent going through all the posts since I was on last night...
I won't quote the relevant posts as they go back ~15 pages, but a few more comments:

TAT errors affecting N1 power set: The FADEC logic (BTW, this is pretty much common on all Boeing FADEC) will use aircraft TAT if it agrees with the dedicated engine inlet temp probe - but if they differ it will use the engine probe . The GE inlet temp probe is relatively simple and unheated, so (unlike a heated probe) a blocked or contaminated probe will still read accurately - just with greater 'lag' to actual temperature changes.

TCMA - first off, I have to admit that this does look rather like an improper TCMA activation, but that is very, very unlikely. For those who don't know, TCMA is a system to shutdown a runaway engine that's not responding to the thrust lever - basic logic is an engine at high power with the thrust lever at/near idle, and the engine not decelerating. However, TCMA is only active on the ground (unfamiliar with the 787/GEnx TCMA air/ground logic - on the 747-8 we used 5 sources of air/ground - three Radio Altimeters and two Weight on Wheels - at least one of each had to indicate ground to enable TCMA). TCMA will shutdown the engine via the N2 overspeed protection - nearly instantaneous. For this to be TCMA, it would require at least two major failures - improper air ground indication or logic, and improper TCMA activation logic (completely separate software paths in the FADEC). Like I said, very, very unlikely.

Fuel contamination/filter blockage: The fuel filters have a bypass - if the delta P across the filter becomes excessive, the filter bypasses and provides the contaminated fuel to the engine. Now this contaminated fuel could easy foul up the fuel metering unit causing a flameout, but to happen to two engines at virtually the same time would be tremendous unlikely.

Auto Thrust thrust lever retard - the TO lockup in the logic makes this very unlikely (it won't unlock below (IIRC) 400 ft., and even that requires a separate pilot action such as a mode select change or thrust lever movement). And if it did somehow happen, all the pilot needs to do is push the levers back up.

Engine parameters on the FDR: I don't know what exactly is on the 787 FDR with regards to engine parameters, but rest assured that there is plenty of engine data that gets recorded - most at one/second. Getting the FDR readout from a modern FDR is almost an embarrassment of riches. Assuming the data is intact, we'll soon have a very good idea of what the engines were doing

TCMA is on both the Trent 1000 and GEnx-1B 'basic' - it was required for certification. There is no reason for TCMA to be listed in the MMEL as the only 'functional' portion is the via the electronic overspeed protection system (which is required for dispatch - no MEL relief) - the rest is software resident in the FADEC.

Another hour spent sifting through the stuff since last night (my sympathies to the mods ). A few more comments:

"Real time engine monitoring" is typically not 'real time' - it's recorded and sent in periodic bursts. Very unlikely anything was sent from the event aircraft on this flight.

Commanded engine cutoff - the aisle stand fuel switch sends electrical signals to the spar valve and the "High Pressure Shutoff Valve" (HPSOV) in the Fuel Metering Unit, commanding them to open/close using aircraft power. The HPSOV is solenoid controlled, and near instantaneous. The solenoid is of a 'locking' type that needs to be powered both ways (for obvious reasons, you wouldn't want a loss of electrical power to shut down the engine). The fire handle does the same thing, via different electrical paths (i.e. separate wiring).

As I've noted previously, a complete loss of aircraft electrical power would not cause the engines to flameout (or even lose meaningful thrust) during takeoff. In the takeoff altitude envelope, 'suction feed' (I think Airbus calls it 'gravity feed') is more than sufficient to supply the engine driven fuel pumps. It's only when you get up to ~20k ft. that suction feed can become an issue - and this event happened near sea level.

Not matter what's happening on the aircraft side - pushing the thrust levers to the forward stop will give you (at least) rated takeoff power since the only thing required from the aircraft is fuel and thrust lever position (and the thrust lever position resolver is powered by the FADEC).

The TCMA logic is designed and scrubbed so as to be quite robust - flight test data of the engine response to throttle slams is reviewed to insure there is adequate margin between the TCMA limits and the actual engine responses to prevent improper TCMA activation. Again, never say never, but a whole lot would have had to go wrong in the TCMA logic for it to have activated on this flight.

Now, if I assume the speculation that the RAT deployed is correct, I keep coming up with two potential scenarios that could explain what's known regarding this accident:
1) TCMA activation shutdown the engines
or
2) The fuel cutoff switches were activated.
I literally can come up with no other plausible scenarios.

In all due respect to all the pilots on this forum, I really hope it wasn't TCMA. It wouldn't be the first time a mandated 'safety system' has caused an accident (it wouldn't just be Boeing and GE - TCMA was forced by the FAA and EASA to prevent a scenario that had never caused a fatal accident) - and there would be a lot embarrassing questions for all involved. But I personally know many of the people who created, validated, and certified the GEnx-1B TCMA logic - and can't imagine what they would be going through if they missed something (coincidentally, one of them was at my birthday party last weekend and inevitably we ended up talking about what we used to do at Boeing (he's also retired)). Worse, similar TCMA logic is on the GEnx-2B (747-8) - which I was personally responsible for certifying - as well as the GE90-115B and the 737 MAX Leap engine - the consequences of that logic causing this accident would be massive.

I hate to disappoint you, but the people (like me) who design, test, and certify aircraft are not idiots. We design for failures. Yes, on rare occasion, something gets missed (e.g. MCAS), but we know that aircraft power systems sometimes fail (or suffer short term interuptions) and we design for that. EVERY VALVE IN THE FUEL SYSTEM MUST BE POWERED TO CHANGE STATE!!!! If electrical power is lost, they just stay where they are. The engine fuel valve must be powered open, and it must be powered closed. Same with the spar valve. The pilot moves a switch, that provides electrical signals to the spar valve and the engine fuel valve to open or close. It's not complicated and has been in use for decades.
TCMA (not TCAM) - Thrust Control Malfunction Accommodation - is a FADEC based system. It's resident in the engine FADEC (aka EEC) - the ONLY inputs from the aircraft that go into the TCMA is air/ground (to enable) and thrust lever position (to determine if the engine is doing what it's being commanded to do. The FADEC has the ability to shutdown the engine via the N2 overspeed protection system - this is separate from the aircraft run/cutoff signal, although it uses the same HPSOV to effect the shutdown. That same system is used by TCMA to shutoff fuel if it determines the engine is 'running away'.

Hint, you might try going back a few pages and reading where all this has been posted previously.

the answer to that was already provided earlier

Retired but extremely doubtful.

Let me share an experience some time ago (not to pretend that there is a similitude here, but it could explain parts of it).

Way back we took off in a 773 with Trent 800 donkeys. We were used to fly the GE90s but still had a few RR dogs left. Long runway with many intersecs, we had covered the first 3, planned was the first. ATC offered the third to get us out quick, so we did. Being used to the kick in the rear by the GE90s, i felt acceleration sluggish, but mentioned to the fo that we were sitting in a dog at more than 35 Celsius.

The runway end seemed near, so my rotation turned out a bit nervous and we ended up in the typical 15degs that i had to reduce rapidly to 12ish to get a meager climb. I recalled silently: Did we calculate the 3rd? Yes. Did we take the 3rd and not the 4rth? No longer 100% sure … So i decided to go for max TO thrust (pushed GA) and increased pitch. - Both engines reacted, but they reacted with a very ugly jolt, almost as if they had to swallow first before thankfully giving the unspectacular full of the Trents. As they were EPR controlled, i can’t recall the N1 and especially N2 values and i can’t recall if their FADECs had a N2 overspeed protection or shut-down mode.

- So after reading a lot of speculation here, i gather: If in doubt firewalling engines out of reduced TO settings in scorching heat and humidity might make even modern engines “swallow” a microsecond. With today’s multitude of sensors and even more consequential modes i would not be surprised that this could trigger something extremely undesirable. I was always somewhat sceptical towards automation and had my mantra to keep it as simple as possible. It makes overviewing and controlling more easy and thats what pilots are still in for. With all that modern automatic background “protection”, i fear us becoming mere scapegoats for many incidents that need to be brushed under the pilots seat for obvious reasons.

There\x92s at least N2 overspeed protection that actually uses the same hardware as TCMA to stop the noise. There might exists crosstalk and inhibit for the N2 overspeed protection if the N2 overspeed protection has shut down the other engine. In fact it\x92s not confirmed that no such crosstalk exists in 787 TCMA system. It would complie with \x94no single fault should cause\x85\x94 certification requirements. Other than that I see no practical difference in the propability of TCMA and N2 overspeed protection to shut down both engine during take-off.

mechpowl:
Thanks for that. That begs the question, if there is overspead protection already, perhaps multiple channels and sensors, why is TCMA needed at all ?. Blanket overspeed protection already covers the underlying requirement, ie: prevention of overspeed, all cases. Seems to be adding complexity for no reason.

That is how it's done on the EC-135 helicopter (also FADEC controlled). One failed engine will disable the overspeed protection for the remaining engine. Of course a helicopter is a whole different story, though.

TCMA is there to prevent higher than commanded thrust while on the ground in purpose to ascertain expected deceleration. N2 overspeed protection is a much older concept from the days before FADECs to prevent overspeeding engine disintegrsting and potentially causing lots of damage to the aircraft. In a GE engined 787 the N2 overpeed protection, while operating as designed, first try to lower the N2 and only if that fails it will shut down the engine.

You sound like you know what you’re talking about. I’m a software engineer. I think software glitches are more common for this type of event than mechanical failures or pilot errors. It can take years before software errors are discovered.

I read one post in here of a 747 flaps retracting on takeoff. No Master Caution, no warnings. Apparently, due to some maintenance triggering a software glitch, the computer thought reverse thrust had been activated during a take off. Whether it was still in ground mode I don’t know.

Point is, being a software glitch in TMCA has already shut down two engines on a 787, I don’t see why the same or another software glitch in TMCA or somewhere else couldn’t do the same. Hadn’t this plane just been in for maintenance?

In software terms, they've reused an existing function to action new functionality. Raises a couple of questions: how many other functions make use of the same N2 overspeed protection functionality; what else could cause N2 overspeed, especially on two engines simultaneously, given the outcome? (Ignoring the software maintenance problems that such secondary purposing can cause later down the road.)

I read, maybe in the preceding thread, a post from a (?) chemical additive manufacturing specialist, referring to n2 speed problems caused by one of their additives incorrectly getting to a bearing (?) and creating a metallic oxide powder and subsequent issues. (Details vague as I can't find the original post - different problem domain to this though). Are there engine lubrication maintenance tasks in a roughly 2 hour turnaround?

Long time lurker, ex aerospace engineering design software engineer. Please delete if inappropriate.

[Edit: spoilling]

I\x92m sure this is wrong; was looking for confirmation. I read somewhere that the 787 keeps the fuel valve open by an electric driven actuator, and closes it by spring force.

I seem to remember Fred Dibner talking about how railway cars brake by draining the piston not by pressurising it, so trains will stop when supply lines break.

The electrical system updates to 787s for ADs and SBs - do any of these include software updates? For example the integer overflow causing GCU failsafe rectified under AD 2018-20-15. If so, who is writing and implementing these software updates? The original engineers? Their apprentices who had years long handovers? Or have they been outsourced and offshored? When these updates occur, does the entire system get tested and ratified or just the bit the bug fix is meant to fix? Because I\x92ve seen new bugs introduced by bug fixes in areas seemingly nothing to do with the original problem.

Interesting thread towards the end, regarding the previous TCMA malfunction on landing : (pprune archived thread 617426, can\x92t post links yet)

according to Dave Therhino who claimed to have seen a detailed report, the TCMA would initialise the thrust contour logic when touching the ground. This was nominally not an issue, as the throttles and engine would be close to idle. However, if the reverser was briefly deployed right before weight on wheels, and then cancelled when the wheels touched, TCMA would see high thrust but throttles at idle and trigger.

But this was supposedly fixed and all FADECs updated. Plus, during take-off there should not be such large fluctuation in throttle position or thrust, so intermittent switching of ground-air-ground should not cause an issue.

Also, according to tdracer V2 overspeed protection cuts thrust so quickly, that if it triggered (via TCMA or whatever other reason) it was likely after the plane had lifted off the ground. I wonder though if there\x92s still enough kinetic energy to fly the profile of the incident flight with the engines cut right around rotation.

Also, would hydraulics go out so quickly, that wheels would not retract? Wonder if a faulty WoW sensor could be a contributing factor and would also not manifest itself as the wheels not retracting :thinking

I don't have any direct knowledge, but yes, that's my understanding based primarily on tdracer's comments. It also just makes sense. I'm pretty confident that all the necessary hardware already existed because of the need for N2 overspeed protection. A failure in one FADEC channel could drive the FMV fully open, leading to an overspeed and uncontained engine failure. For regulatory purposes, it would be unacceptable to have a single point of failure with catastrophic consequences, so it would be necessary to make the inactive FADEC channel capable of cutting off fuel in that case.

The air/ground signal would've already been present as well. It would be needed for switching between ground idle, flight idle, and approach idle. Tdracer has discussed that as well, in past threads.

Yes. I simplified. The point stands that the throttle needs to be pulled back, as it was in the ANA event, because that was a landing and not a take-off.

First, you posted a good summary. I'd have added "unanticipated hardware fault" and "unanticipated software fault" as generic causes.

Note that the thrust lever actuators are wired to the FADECs, and that the TCMA gets the T/L position from that. For TCMA to trigger, it has to determine that its FADEC (on that engine) failed to achieve a commanded reduction in thrust. So we're either looking at a weird, unprecedented edge case, or a FADEC failure, or both.

It has been mentioned before that this capability existed as part of the N2 overspeed protection: the FADEC would shut down a runaway engine by cutting its fuel before it disintegrates.
The thrust lever sensors are wired directly to the FADEC (and hence the TCMA). No data bus is involved with this item.

With a MCAS crash, it required a hardware problem with an AOA sensor, used as input to a correctly working MCAS, to cause the aircraft to behave erratically. With a correctly working TCMA, I believe it'd require two hardware problems to get TCMA to shut down the engine, as there'd have to be an implausible thrust lever reading, and a FADEC/engine failure to process it within the TCMA allowed range ("contour"?). On both engines, separately and simultaneously.

That leaves a software problem; it's not hard to imagine. The issue is, at this point it's just that: imagination. I could detail a possible software failure chain, but without examining the actual code, it's impossible to verify. We simply don't have the evidence.
I could just as well imagine a microwave gun frying the electronics on both engines. An escaped hamster under the floor peeing on important contacts. A timed device installed by a psychopathic mechanic. There's no evidence for that, either.

This process is a way to psychologically cope with the unexplained accident, but because it lacks evidence, it's not likely to identify the actual cause. We've run the evidence down to "most likely both engines failed or shut off close to rotation, and the cause for that is inside the aircraft". Since the take-off looked normal until that failure, we have no clues as to the cause hidden inside the aircraft. We need to rely on the official investigation to discover and analyse sufficient evidence. The post-crash fire is going to make that difficult.

"Both engines failed or shut off close to rotation" explains all of the evidence : it explains an unremarkable take-off roll, loss of lift, absence of pronounced yaw, loss of electrical power, loss of the ADS-B transponder, RAT deployment, the noise of the RAT banging into place and revving up, emergency signs lighting up, a possible mayday call reporting loss of thrust/power/lift, and a physically plausible glide from a little over 200 ft AAL to the crash site 50 feet (?) below aerodrome elevation .
It explains what we saw on the videos, what the witness reported, where the aircraft ended up, and the ensuing sudden catastrophe.

I don't believe we have evidence for anything else right now—I'd be happily corrected on that.

-----
Edit: the evidence of the crash photo with the open APU inlet door, and the main gear bogeys tilted forward, are also explained by the dual engine failure/shut off.

I see the logic. If the N2 exceedance functionality in the FADEC already included authority to shut off fuel in some circumstances, might as well 'piggyback' TMCA output onto that part of the FADEC functionality.

Re air/ground input to the TMCA, I am more interested in whether that is hardwired from the sensors / systems involved or - as appears to be the case just based upon what I've read here - read from ARINC or other format data.

In general, we can classify computer errors into 3 categories:

1. Errors in system design, specifications, or algorithms. These are cases where the computer does exactly what it was designed to do, but the design itself was flawed or inadequate, or had unforeseen consequences. This would include things like MCAS on the 737 MAX.
2. Software errors. These are cases where the computer does exactly what the code tells it to do, but not what the designers wanted it to do. This results from mistakes in writing the actual code, and this is usually what we'd refer to as a "bug." This includes things like race conditions, loops that fail to terminate, incorrect type conversion, etc.
3. Hardware malfunctions. These are cases where the computer does something different from what the code instructs. It can involve memory corruption or data bus corruption. It can result in a system that appears to work, but returns incorrect outputs, e.g. a calculator saying 2+2=5. It can also cause the computer to execute valid instructions, but at an inappropriate time. It can result from manufacturing defects in components, cosmic rays (SEU or SEE), radiofrequency interference (HIRF), moisture ingress, failed solder joints, and numerous other things.

As I said previously, I think we can rule out category 3 in this case. Hardware malfunctions are essentially random events. They're inherently unpredictable since there's little to no relationship between what the system was supposed to do and what it actually does. It would be astronomically improbable for the same failure to occur on both engines at the same moment.

Categories 1 and 2 would be common to both engines, so they both remain plausible. For category 2, it would be impossible to identify the issue without analyzing the complete source code. Since we don't have access to that code, this is a dead end. It could be the cause, but we won't be able to figure it out. Looking at how the FADECs are designed to work isn't going to be very useful here, since by definition, they'd be doing something they weren't supposed to.

Category 1 is a bit different. There are 2 functions we know of that can close the fuel shutoff valve: TCMA and N2 overspeed protection. We don't have the complete specifications, but the basic logic of both functions has been described. If we assume that one of these was the cause, then the conditions for one of those functions must have been met.

The conditions for TCMA, at least as it's been described in this thread, are:

• Airplane on ground
• Thrust higher than commanded by thrust lever angle (TLA) for some period of time

It's reasonable to assume that the first condition is common to both engines. But that still leaves the second. To my knowledge, there's no plausible scenario that would cause that condition to be met on both engines simultaneously. Each thrust lever has 2 resolvers which are wired directly to the corresponding FADECs. That means the signals don't pass through any common component. An incorrect reading from one resolver could conceivably trigger TCMA, but I don't see any reason why that would happen to both engines simultaneously.

As for the overspeed protection, as far as I know, there's only one condition: N2 greater than a certain value. That reading comes from sensors that are inside each engine and wired directly to the FADECs. I don't see any way this could affect both engines simultaneously either, but it still seems a bit more likely than something involving TCMA since it only requires 2 separate, simultaneous failures rather than 3 or more.

For the sake of accuracy, I should also note that not everything fits neatly into one of my 3 categories. For example, let's say we have a machine that's programmed to shut down if any one of 3 parameters goes above a certain value. If one of those values gets corrupted by a faulty memory chip, the machine could shut down unnecessarily. If we add more parameters to the list, the probability of an inadvertent shutdown increases since there are more critical areas in memory. As another example, consider a case where corruption of the CPU's program counter causes it to inadvertently jump to a particular subroutine. If we add more subroutines that can trigger a shutdown, we make the machine more vulnerable, albeit to a very small degree. Changes like these are sometimes referred to as "increasing the surface area."

Due to those types of scenario, I will admit that the existence of something like TCMA probably makes an engine ever-so-slightly more likely to fail. Whether the benefit is worth the cost could be debated. In any case, I still find it pretty unlikely that any of this will turn out to have been a factor in this accident.

I have to both agree and disagree with both this and the next post by TryingToLearn.

Yes, there may be (let's assume is) "identical" FADEC/TCMA hardware and firmware on both engines, but if the Left Engine is subject to Mars in the house of Uranus (wink wink), then the Right Engine cannot be, maybe it's Venus in the same House. This is simply because the Left engine TCMA 'contraption', I'm going to call it, is monitoring Left Engine Conditions (Shaft Speed, T/L setting / position data - Right or Wrong, and calculating and comparing accordingly against its internal map) while the opposite TCMA "device" is monitoring and calculating etc, Right Engine Conditions. There are some things in common, but (I say) it's virtually impossible for the Engine Conditions being individually monitored to be identical in both engines.

The Thrust Levers are electro-mechanical devices, almost certainly at this stage pushed by a somewhat squishy human hand, likely with a slight offset. What is the probability that those two levers are in identical positions, and even if they are, that the calibration (e.g. "zero points") of both levers are identical, and that the values they output (response slopes/curves) are exactly matching in every matching point in their individual travels? That's just one aspect, but consider the engines. They are different ages. Have different amounts of wear. They have separate fuel metering valves (or other names), separate HP Fuel pumps (and, I guess relief valves?), all also subject to wear), and each has a host of other, correspondingly paired, sensors, (maybe of different makes and certainly of different ages and different calibrations and response curves) from which each FADEC, supposedly independently of the TCMA, adjusts the fuel metering device settings and resulting engine power, and shaft RPMs follow in some other slightly non-matching way.

Sure, I would completely agree that the two engines and their calculated Throttle Lever positions to Shaft RPMs are always going to be similar during normal, matched operation, and they will very likely dance with each other, maybe one 'always' (75% of the time, say) leading during one dance (TO, say) with the other leading in dancing to a different tune (descent, say).

To me, the fact that this appears to have been an almost simultaneous dual engine failure, pretty much, for me, rules out a FADEC/TCMA firmware bug, especially as there don't seem to be any reports of even a single engine mid-air TCMA shutdown.

HOWEVER, and I want to stress this, that doesn't rule out the possibility that both TCMAs shutdown their respective engines simultaneously. Any lack of simultaneity observed would be due to those slight differences in other pieces of hardware, such as the time for one Shutoff valve to close versus the other.

As far as I know, there isn't enough information on what's actually inside those TCMA Black Boxes to say anything for sure, but here's a thought, which I think has been alluded to, or the question asked, here in one or other thread, earlier.

What does the TCMA firmware do when an engine is already running at a high power setting and TWO things occur in quick succession? I suspect this kind of event is a highly probable cause, but these two events have not occurred close enough together, or ever, before.

Imagine this: Plane taking off, Throttle Levers near Full Power, Engines performing correctly, also near Full Power, Rotation etc all normal, plane beginning to climb, positive rate achieved.

Pilot calls GEARUP. GearUp, activated.

The Gear Retract sequence begins. Due to some unforeseen or freshly occurring (maybe intermittent short or open circuit) linkage between the gear Up sequence and the WOW or Air/Ground System, the signal to both TCMAs suddenly switches to GROUND. All "good", so far, as the engine RPMs match the Throttle Lever settings and TCMA doesn't flinch. The plane could be in a Valid Takeoff sequence, so it had better not! But it does make a bit of sense. How is WOW / Air/Ground detected? Somewhere near the Landing Gear, I assume.

HOWEVER, now, a moment later, and perhaps due to a related system response, the Thrust Levers suddenly get pulled back to Idle, whether by man or Machine.

What would you expect the TCMA system to do? I would guess, fairly soon thereafter, two, independent, Fuel Cutoffs... Though I fully admit, I'm guessing based on a severe lack of knowledge of that Firmware.

Ok, no need for further explanation on that point, but I did refer to TCMA unflatteringly as a contraption, earlier. Last night (regrettably, before bed) I started looking at the TCMA Google Patent. Let's just say, so far, I'm aghast! My first impressions are bad ones. How did this patent even get approved? What I suspect here, now, is not a Firmware bug, but a serious Logic and Program Defect. But we'd have to see what's inside the firmware.

When I get more time, I'll dig deeper.