Posts about: "TCMA (All)" [Posts: 279 Pages: 14]

Why should it? It\x92s part of the FADEC as are the TLA sensors.

Found It
Truth Table Line 101 = Service Terminated (licence agreement expired please contact your service provider)

Could the testing of TCMA logic less robust for the portion that works only when is not armed (i.e. not on ground)? I am asking this because from previous posts the ground logic needs only one signal (WoW, radio altimeter) to be true, if so is correct a faulty sensor could have armed the TCMA? That would have removed a safety layer on the system.

I read the whole threads, keeping my hands on the mousewheel so far since I'm not a pilot, just a EE / safety / systems engineer.
The hamsterwheel ist spinning a lot here, and of course it could be anything including some VHDL FPGA code line or a broken RAM cell in a cheap memory bar within the computer it was compiled with. Anything is possible, but to be honest: development processes, if followed, are usually pushing the probability to a level where it becomes pure theory. BOSCH uses FPGA+\xb5C on the brake control box of cars. They sold 100 millions of those, used 4000h each (car lifetime) without error, with less strict development process. Most errors are made on requirement level, not code.

Also, so far there is no evidence I've seen regarding the 'chicken-egg' problem, did the engines fall below idle (fuel, stall...) and this caused an electrical blackout (-> battery, RAT...) or did an EE problem cause the engines to reduce thrust (FADEC, SW bug...). And where is the common cause in all this?
There has to be a systematic error common to both engines, an external failure affecting both or a dependent fault with one affecting the other within seconds. This is the only thing I think everyone agrees here. And I refuse to beleave the external failure or dependent fault was sitting in the cockpit.
I think it is something not common to every aircraft type for the last 50 years.
So I started searching and found a candidate.

I read myself into the EE architecture of this unique 'bleed-less' design and it's megawatt powergrid since this is the part where I may be able to contribute (and I'm most curious about). Generators on the 787 are >250kW instead of <100kW each and there are two per engine instead of just one. In fact, they can go up to 516 kW and shear off the gearbox at >2200Nm (equal to >2 MW, per generator).
https://www.easa.europa.eu/en/downloads/7641/en (page 11)
So while on any other aircraft the generator is more like the dynamo on your bicycle, those generators are massive (x10).
The gearbox is connected to the HP shaft (N2) on the GEnx. I learned from Wikipedia that RR moved this gearbox to the IP shaft on the Trend 1000. And RR is happy that the A330neo Trend 7000 uses bleed air and less load on the gearbox, since this maintains stability on the HP shaft at light load (also Wikipedia).
Those generators are not in phase and frequency sync, or in other words: If you parallel them, they fight each other, it's like a short. They will almost block if this is not handled by the control box if possible (or some melting fuse blows at some point).
787 electrical system - variable frequency generators?
Somehow I find it hard to believe that they are not able to disturb the engines despite that everyone here so far is claiming that there is no way an electrical problem could influence them because FADEC has it's own supply. I read that one is sufficent to start the engine, usually both are used.
In my mind I find lot's of ways this could influence both engines simultanously. If just the BTBs on the 230V grid got some humidity (hot, no AC, water cooling...) and went up in one big arc (I think they made them semiconductor relays, too).
Could those gearboxes and engines handle 4500Nm / almost 5 MW on each HP shaft, applied within a fraction of a second without any problem?
Or if the engines were in a condition not far from compressor stall, one was stalling and 400kW load jumped from one engines generators to the other...
I did some rough estimation and one of the generators could push N2 below idle in a second or less without fuel just with its normal 250kW load (just inertia).
This is one point which is unique to this airplane model, so maybe worth a closer look.

I know that those engines are burning at >100MW at full power, but how fragile in the balance between compressor load and this one turbine stage on the HP shaft / N2, without the inertia of a 2.8 meter fan? This is just out of my background, any thermodynamic expert here?
Of course I also have no insight in SW and communication within the control boxes, how much they are talking to each other, delaying/ramping load redistribution etc. If FADEC recognizes a flameout, could it instantly command the generators to cut the load, even above idle rpm?

I would assume that some fuel contamination, valve blockade, even compressor stall would pop up slower. But such a generator could kick in within milliseconds.

As a safety guy I learned that one tends to look first at things one is familiar with (SW, HW, mechanics, pilot behaviour, maintainance, depending on one's profession) and in the end it's often the interface and dependent faults within which are not carefully considered (e.g. takeoff situation vs. thermodynamics vs. mechanics vs. power generation vs. humidity vs. generator control...) together with transient behaviour. It was the same with MCAS (safety culture vs. pilot training vs. SW design (repeated action) vs. single AOA input vs. bird strike probability close to ground vs. trim loading/blockade vs. stickshaker noise/distraction).
In fact, I was trying to find information on all those systems and directly found slides on how the engines and generators could be simulated and the power grid tested in a HIL (hardware in the loop) environment. My experience from automotive is that such simulated environments are often far from reality and HIL environment programming finished after the product is already at the customer. But of course its far easier and cheaper to apply and test faults there. But then, some programmer programms what he thinks the reaction of the engine would be.
This 'bleed-less' design was some massive change in airplane EE architecture with hugh consequences on the whole airplane design and extremely hard to fully analyze.

I'm just asking questions and hope that we all learn a lot and this was fully considered or just not an issue. It's just an aspect I found worth mentioning and not only spinning the wheel.

PS: I doubt it was TCMA. The air/ground decision is done in a different box, evaluating 5 inputs in a 1/3 and 1/2 decision according to this discussion. This is then safely sent to the FADEC (as one input) and combined with the thrust lever position and N2. But if the thrust lever position is sensed (redundant and direct) close to idle, you do not need TCMA or ground mode to expect reduced thrust.

Perhaps my earlier post was incredible and that's what prompted the SLF's question.

Let us assume a simple, hypothetical WoW sensor arrangement: One sensor per main landing gear.

One of those sensors is indicating weight OFF wheels and the other is indicating weight ON wheels. What does the TCMA in each engine interpret that ostensibly contradictory sensor information to mean? (Note: For the time being, ignore the question whether the information is erroneous. It may be true.)

Are both engine TCMA's in the 'in the air' state, are both 'on the ground', or is one 'on the ground' and the other 'in the air'?

Given the purpose of the TCMA, I would have thought that any 'doubt' in this case would be resolved in favour of the 'on ground' state for both TCMAs.

But maybe it's the other way around. Maybe any 'doubt' would be resolved in favour of both TCMA's being in the 'in the air' state.

I have difficulty in envisaging any advantage in the TCMA system being designed such that one engine's TCMA is in the 'in the air' state and the engine's 'on the ground'.

Whichever the design and outcome, there will be benefits and there will be risks.

This was already addressed in one of tdracer's posts about the TCMA system: the air/ground decision is made by the aircraft itself (using multiple redundant inputs and voting logic), and only a single air/ground binary state is provided to the EEC(FADEC). Further, he also stated that the EEC assumes "air", as that is the safer state given the nature of its operations.

Based on this, both engines will get the same air/ground indication from the aircraft and hence will always make the same TCMA decisions (subject to their individual throttle positions and thrust outputs).

Thanks Chernobyl. I had understood trdracer's posts to be qualified on the basis of experience on types other than the 78, but I may be mistaken and, in any event, the 78 design may be exactly the same as those types.

I will suggest some amendments to your last sentence, though: Based on this, both engines will should get the same air/ground indication from the aircraft and hence will should always make the same TCMA decisions (subject to their individual throttle positions and thrust outputs).

Let's not lose sight of the fact that a 787 has had a TCMA 'commanded' double engine shut down, luckily only during the landing roll. That double shut down was not in circumstances of a rejected take-off where one or both engines delivered 'too much' thrust despite thrust levers being set to idle or 'low power'. Some might say incredible. But it's fact.

The best designed systems and software sometimes do strange, unexpected things even when everything is working 'properly', and even stranger things when some defect or damage occurs.

Yes, all true. And I didn't think your earlier post was incredible, just that it was an exercise not intended to explain everything about air/ground logic. I think that, in the real world, it's most likely that the air/ground decision will take inputs from other sensors, not just WoW. Radio altimeters seem the most likely choice and we've been told they've been used on earlier Boeings (which I think I already knew or assumed). I'd certainly feel safer with that arrangement.

That's kind of you to say. However, I have to concede that my surmising that any one sensor failure in the 'on the ground' versus 'in the air' logic would result in an 'on the ground' state in both TCMA was waaaay off track.

Without knowing the 787-8 gear system, we know that is is supposed to be hydraulically moved from \x94nose up\x94 to nose down as the first step in the gear up sequence. But do we know that it would end up \x94nose down\x94 without hyd pressure?

Another point pointing to that the aircraft did consider itself being \x94In Air\x94 is the ADS-B data sending Altitude from the first 575 feet at 08:08:46.55 until at least 08:50.87\x85?

I would think the sub systems like TCMA would use the same In Air / On Ground logic as the aircraft normally use?
I come from an FBW aircraft with a Air/Ground logic that seems rather bullet proof and would guess the 787 wouldn\x92t use a less solid logic which probably, in doubt would consider it being \x94In Air\x94?
It would be \x94logic\x94 for the TCMA to use this logic?

Thats right, and (guessing here), it would be better to use the solid logic In Air / On Ground than grab a single WoW sensor specially for things that could go very wrong if that sensor start showing a false value.

Qualified guess: You cannot certify a system that affects safety (TCMA for example) without considering failure of inputs etc.

As the TCMA seems to be widely used it should not have been able to fly under the radar like the MCAS sort of did.

SLF Engineer (electrical - not aerospace) so no special knowledge

Perceived wisdom may be applicable in normal circumstances but not when all the holes line up.

For example I've seen it quoted many times that the engine FADECs are self powered
by the engines, the TCMAs-whether part of the FADEC or a separate unit, similarly self contained
within the engine. The perceived wisdom seems to be that there is no common single fault
which can take out both engines.

And yet we're also told that the TCMA function can only function in ground mode and receives ground-air
signals from a combination of inputs from Rad Alts and WOW sensors.
There is therefore a connection from the central EE bay to the engine.

Yes I'm sure the Rad/Alt and WOW sensor processing will use different sensors for each side and powered from different
low voltage buses.
However as an analogy, in your house your toaster in the kitchen may be on a separate circuit from the water heater in
the bathroom, each protected by a fuse at the main switchboard. In normal operation a fault in one cannot affect the other.
However a lightning strike outside the house can send much higher voltages than normal operation throughout the entire
system and trash every electrical appliance not physically disconnected at the time.

Now I'm not suggesting the aircraft was hit by lightning but FDR has proposed a single event, buildup from a water leak entering
one of the EE bays at rotate. It would be possible for one or more of the HV electrical buses to short so that all the low voltage
buses go high voltage. I have no knowledge of how the FADEC / TCMA systems connect to or process the Ground-Air signals but
there is a single fault mechanism whereby high voltage could be simultaneously and inappropriately applied to both engine control systems.
It would be unfortunate if this failure mechanism did cause power to be applied to drive the fuel shut off valve closed.

Since the likelihood is that we're looking at a low probability event then perceived wisdom about normal operations and fault modes
might not be applicable.

The equipment on RAT/battery is limited:


(from the online 2010 FCOM)


(from the maintenance training )

The 787 battery fire report says the two recorders are on the left and right 28VDC buses. I don't think those get powered on RAT by the looks of it. I would wager you get whatever is on the 235VAC 'backup bus', plus the captain's and F/O's instrument buses via C1/C2 TRUs. You won't get all of that (like the F/O's screens) because the 787 energises/de-energises specific bits of equipment, not just whole buses.

Losing recorder power looks entirely expected.


400VAC/540VDC (+-270V) is not really known for blowing past input protection in the same way as actual HV or lightning. I would expect some optocouplers and/or transformers to be both present and adequate. There's definitely some big MOVs scattered around the main 235VAC buses.

Weight on wheels appears to go into data concentrators that go into the common core system (i.e. data network).

Presumably there is a set of comms buses between the FADECs and the CCS to allow all the pretty indicators and EICAS alerts in the cockpit to work. The WoW sensors might flow back via that, or via dedicated digital inputs from whatever the reverse of a data concentrator is called (surely they have need for field actuators other than big motors?). Either way, left and right engine data should come from completely different computers, that are in the fwd e/e bay (or concentrators/repeaters in the wings, maybe) rather than in with the big power stuff in the aft e/e bay.

MCAS wasn't "under the radar". The designers thought:
* all MCAS can do is affect the trim
* if something goes wrong with the trim, the crew works the "runaway trim" checklist
* this cuts MCAS off from the trim
* therefore, MCAS failure of any sort is going to be contained

It just turned out that if a crew is stuck, shortly after takeoff, in an aircraft that wants to go down, and they have no clue why because "AOA disagree" indicators are considered a luxury item for Boeing and Boeing also did not want to train crews for this, the crew may not be in the right mindset to prioritise that checklist.
Today, everyone is aware, so it's no longer an issue.

TCMA was motivated by a similar observation: that crews can fail to shut down an engine that no longer follows command input. So the FAA requires aircraft to detect that condition and do it automatically when on the ground, where an engine running at significant thrust is a danger to people and movable objects in the vicinity. The safety consideration here is, if you're on the ground and the thrust reverser is not deployed, you're not going to need the engine that badly. (I think there are actually two more conditions that I don't remember right now.)

In safety, you kinda need to weigh the consequences of having this system (with a chance that it might malfunction) vs not having it. Also consider that the benefit of having it, all of the occasions where it correctly shuts an engine off, don't get reported in the press.

If a TCMA bug caused this accident, the investigation will find out.
But right now, we don't have any evidence that that's what happened.

\x94Designers thought\x94 = MCAS flying under the radar from Boeing themself.

Anyway, I think we agree here.

I cannot se TCMA logic flying under the Boeing radar in this case?

TCMA is a logic nuilt in the EEC/FADEC by the Engine manufacturer I guess?

Yes. and whe\x92re discussing scenarios here.

With extremely little data for us to use I think people are grabbing anything as a cause.

TCMA as a cause has been interresting, learning. But it should be designed safe. Can we find a data point that takes us away from TCMA or can\x92t we?

This is absolutely a fair comment (i.e., should vs will), given that we still don't know the cause of this accident. Since it appears that some highly unlikely combination of circumstances has nonetheless taken place, ruling anything out as "impossible" is unscientific at this time. Another poster above put it well: "as long as something doesn't violate the laws of physics, then it should be considered a possible cause" (I paraphrase).

Short introduction on faults:

There are systematic and random faults.
Mechanics and SW only know systematic faults, only electronic HW does have random faults.
Why?
Mechanics are large enough to react in a reproduceable manner, SW too (on working HW).
Worn out part? -> systematic fault, you did not calculate wear correctly in your analysis, adding maintainance cycles
Broken nut due to too much torque? -> systematic fault, you did not analyse the severity and put QM measures in place within maintainance.
broken fan blade due to corrosion? -> systematic fault, incorrect testing and environment assumption, other fan blades may be corroded, too
SW crash every 10 years? -> systematic fault, you did not make it robust against some nasty timing circumstance or race condition.
So systematic faults do not mean they happen instantly, they may happen never or after dozens of years. Not on all parts but e.g. just if there is a problem in production like a cavity in aluminum casting.
But there are systematic ways to avoid them (FMEA, FTA...). This is why every accident is analyzed. If it was a systematic fault, other party could be affected.
This also counts for SW, there are techniques to avoid errors on all levels of development. But having the same SW with the same inputs and the exact same timing will give the same outcome (...on both engines).

Random faults are unique to electronic HW. The reason is just because of miniaturization. One cannot exclude migration and degradation on such small scales, there will be faulting transistors or other parts. If you scale them bigger, your processor runs on kHz instead of MHz, this is not an option. Those are random HW faults. There are calculation methods (FMEDA) to calculate the expected lifetime and error probability over the expected temperature profile. But if you give the faulting part the same input, it will show the fault again. But not any other part without the degraded transistor will give you the same error.
And there are soft faults (not SW faults, soft faults), those are not reproducible at all. Imagine some hard x-ray from the sky hitting a DRAM cell and changing it's state. You will never be able to reproduce this hard x-ray which made it's way thru the atmosphere without interaction hitting exactly the right \xb5m\xb3 to do the same bit-flip again. You can just have redundancy in information and processing (ECC, parity, lock-step etc.) to have enough probability to detect (and reset) or correct the fault.

Faults, no matter if systematic or random can be single, multiple, latent and dependent faults.
Single fault: a single error leads to a hazardous state
Multiple point fault: more than one (independent) error will lead to a hazardous state
Latent fault: a fault which remains undetected for some time (e.g. only detectable during startup / power-on) and not hazardous by itself. But together with a second fault it will lead to a hazardous event. But the probability of 2 errors within a power cycle is sufficiently low.
Dependent fault: A fault, by itself not hazardous which will influence the probability of a second fault with both in combination... exactly. (-> freedom from interference not given)

Having the same SW bug on both engines would fall within the dependent fault category. Redundancy only helps if freedom from interference is given.
So if there is a fault in SW and both FADEC instances get the same input and have the same timing, they will react the same way. If there is a hidden systematic fault, both will show it. This is why things are often programmed twice by independent teams or on different processors. They will not do the exact same mistake. Or there is a normal function and a supervisor like TCMA. One complex function but a tiny, easy to analyze supervisor just checking for the hazardous state.

All this can be analyzed and is really lots of work but necessary. So first there is an analysis how severe a fault would be on a high level, this defines the necessary level of analysis in detail.

I have to both agree and disagree with both this and the next post by TryingToLearn.

Yes, there may be (let's assume is) "identical" FADEC/TCMA hardware and firmware on both engines, but if the Left Engine is subject to Mars in the house of Uranus (wink wink), then the Right Engine cannot be, maybe it's Venus in the same House. This is simply because the Left engine TCMA 'contraption', I'm going to call it, is monitoring Left Engine Conditions (Shaft Speed, T/L setting / position data - Right or Wrong, and calculating and comparing accordingly against its internal map) while the opposite TCMA "device" is monitoring and calculating etc, Right Engine Conditions. There are some things in common, but (I say) it's virtually impossible for the Engine Conditions being individually monitored to be identical in both engines.

The Thrust Levers are electro-mechanical devices, almost certainly at this stage pushed by a somewhat squishy human hand, likely with a slight offset. What is the probability that those two levers are in identical positions, and even if they are, that the calibration (e.g. "zero points") of both levers are identical, and that the values they output (response slopes/curves) are exactly matching in every matching point in their individual travels? That's just one aspect, but consider the engines. They are different ages. Have different amounts of wear. They have separate fuel metering valves (or other names), separate HP Fuel pumps (and, I guess relief valves?), all also subject to wear), and each has a host of other, correspondingly paired, sensors, (maybe of different makes and certainly of different ages and different calibrations and response curves) from which each FADEC, supposedly independently of the TCMA, adjusts the fuel metering device settings and resulting engine power, and shaft RPMs follow in some other slightly non-matching way.

Sure, I would completely agree that the two engines and their calculated Throttle Lever positions to Shaft RPMs are always going to be similar during normal, matched operation, and they will very likely dance with each other, maybe one 'always' (75% of the time, say) leading during one dance (TO, say) with the other leading in dancing to a different tune (descent, say).

To me, the fact that this appears to have been an almost simultaneous dual engine failure, pretty much, for me, rules out a FADEC/TCMA firmware bug, especially as there don't seem to be any reports of even a single engine mid-air TCMA shutdown.

HOWEVER, and I want to stress this, that doesn't rule out the possibility that both TCMAs shutdown their respective engines simultaneously. Any lack of simultaneity observed would be due to those slight differences in other pieces of hardware, such as the time for one Shutoff valve to close versus the other.

As far as I know, there isn't enough information on what's actually inside those TCMA Black Boxes to say anything for sure, but here's a thought, which I think has been alluded to, or the question asked, here in one or other thread, earlier.

What does the TCMA firmware do when an engine is already running at a high power setting and TWO things occur in quick succession? I suspect this kind of event is a highly probable cause, but these two events have not occurred close enough together, or ever, before.

Imagine this: Plane taking off, Throttle Levers near Full Power, Engines performing correctly, also near Full Power, Rotation etc all normal, plane beginning to climb, positive rate achieved.

Pilot calls GEARUP. GearUp, activated.

The Gear Retract sequence begins. Due to some unforeseen or freshly occurring (maybe intermittent short or open circuit) linkage between the gear Up sequence and the WOW or Air/Ground System, the signal to both TCMAs suddenly switches to GROUND. All "good", so far, as the engine RPMs match the Throttle Lever settings and TCMA doesn't flinch. The plane could be in a Valid Takeoff sequence, so it had better not! But it does make a bit of sense. How is WOW / Air/Ground detected? Somewhere near the Landing Gear, I assume.

HOWEVER, now, a moment later, and perhaps due to a related system response, the Thrust Levers suddenly get pulled back to Idle, whether by man or Machine.

What would you expect the TCMA system to do? I would guess, fairly soon thereafter, two, independent, Fuel Cutoffs... Though I fully admit, I'm guessing based on a severe lack of knowledge of that Firmware.

Ok, no need for further explanation on that point, but I did refer to TCMA unflatteringly as a contraption, earlier. Last night (regrettably, before bed) I started looking at the TCMA Google Patent. Let's just say, so far, I'm aghast! My first impressions are bad ones. How did this patent even get approved? What I suspect here, now, is not a Firmware bug, but a serious Logic and Program Defect. But we'd have to see what's inside the firmware.

When I get more time, I'll dig deeper.

If we are to take the TCMA patent at face value, the fuel cut-off switches are directly-acting, not some sort of signalling protocol.

That's a pretty big "if" but here's the patent drawing: