Fault handling and safe states
Assume faults will occur: detect them in time, contain their effects, move the system to a hazard-informed condition and preserve enough evidence to understand what happened.
After this module, you should be able to:
- Distinguish faults, errors and failures
- Derive responses from hazards and operating context
- Design detection, containment and recovery
- Plan fault-injection verification
Study guide · Updated · About 9 minutes reading + 20–30 minutes practice
Before you start: Read TEA-101, TEA-103 and TEA-105. A fault response must relate to a physical hazard.
A fault becomes dangerous through a chain of effects.
A fault is an abnormal condition or cause, an error is an incorrect internal state, and a failure is externally visible loss of required behaviour. The purpose of fault handling is to break the progression before unacceptable harm occurs.
A safe state is specific to the hazard and operating context.
“Switch everything off” is not a universal answer. Removing power may release a brake, stop cooling, lose containment or interrupt a physical action at its most hazardous point. Some systems need a safe sequence, a degraded mode or continued operation long enough to transfer control.
| Question | Design implication | Evidence needed |
|---|---|---|
| What hazard can this fault create? | Define the physical condition that must be prevented or limited. | Hazard analysis and fault-response requirement. |
| How quickly must control act? | Allocate detection and response time across hardware and software. | Fault-tolerant time interval and timing verification. |
| Is immediate stop safe? | Choose stop, hold, complete-current-step, controlled shutdown or degradation. | State-dependent response rationale. |
| Can the primary controller be trusted? | Add independence, diversity or an external protection mechanism where necessary. | Architecture and independence analysis. |
| Can operation resume? | Define latching, acknowledgement, self-test, service and restart conditions. | Recovery specification and test results. |
| What must be retained? | Record fault identity, time, context and configuration without blocking the response. | Diagnostic and event-log specification. |
Typical fault classes
Consider sensor absence, drift and implausibility; actuator non-response; communication corruption or silence; missed deadlines; memory corruption; task lock-up; stack or resource exhaustion; brownout and reset; configuration mismatch; excessive temperature; and loss of an external dependency.
Design the complete response, including recovery.
- Start with hazards. Identify which system conditions are unacceptable and which faults can lead to them.
- Set the response bound. Determine how long the system may remain uncontrolled before the condition becomes hazardous.
- Select diagnostics. Use range, plausibility, temporal, sequence, redundancy or active self-test checks with stated coverage.
- Allocate containment. Decide what hardware, firmware, software or mechanical feature limits the effect.
- Define state-dependent action. Specify the transition, outputs, user indication and handling of in-progress work.
- Define persistence and recovery. State whether the fault latches, which tests permit restart and when service is required.
- Challenge common causes. Examine shared power, clocks, memory, software, connectors, environments and assumptions.
Worked example: position-controlled drive
The controller commands a moving carriage while monitoring position and motor current. Position feedback becomes frozen but remains numerically within range.
A simple range check would miss this fault. A temporal plausibility diagnostic compares commanded motion, elapsed time, encoder change and current. The drive removes torque only after the mechanical design confirms that the brake can engage safely. Restart requires a successful sensor self-test and a deliberate operator action.
Demonstrate the response at the system boundary.
Faults, causes, affected functions, detection methods, latency and coverage assumptions.
State-specific containment, outputs, timing, indication, latching and recovery.
Independence, common causes, single points of failure and protection layers.
Method, injection point, system state, expected sequence and acceptance criteria.
Time-correlated sensor, command, diagnostic, state, output and alarm evidence.
Power-cycle, reset, reconnection, self-test, acknowledgement and service scenarios.
Common failure patterns
“System stops safely” does not define outputs, sequence, time or behaviour of the physical load.
A reset may repeat the fault, erase evidence or briefly re-energise an unsafe output.
A failed value remains in range and is accepted because no independent relationship is checked.
The initial fault response works, but reconnect, restart or repeated faults create a hazardous state.
Specify detection, containment and recovery separately
A fault requirement should name the detectable condition, detection latency, response, completion limit and recovery authority. “Stop on sensor failure” leaves open which failures can be detected and what stopping means. An open circuit, a frozen plausible value and a biased measurement require different detection arguments.
Separate a diagnostic observation from a fault decision. One outlier may justify rejecting a sample, while repeated disagreement may justify degraded operation or a latched fault. Debouncing and confirmation delays reduce false alarms but consume response time. Allocate that delay using the hazard analysis.
After detection, prevent the failed information or output from continuing to influence control. Latching a software flag is insufficient if another task can overwrite the output. Give fault handling clear precedence and one owner for the final actuator state.
Challenge the safe state against stored energy
“Power off” is only one possible response. A brake may need energy to hold a load, a cooling fan may need to keep running and a valve may trap pressure when closed. Consider the transition as well as the final state: sudden stopping can itself cause harm.
Fault tolerance depends on independence. Two measurements from the same ADC reference can drift together. A watchdog refreshed by a timer interrupt may continue to be serviced while the control task has stopped. Identify common causes and verify that the monitoring mechanism observes useful progress.
Where the product continues in a degraded mode, specify permitted functions, time limits, user information and further-fault behaviour. Degraded operation is a designed operating mode with requirements, not a catch-all exception that bypasses controls.
| Stage | Question to answer |
|---|---|
| Detect | What fault manifestation is observable, and how quickly? |
| Contain | Which commands or data must lose authority? |
| Respond | What physical condition must be reached within the limit? |
| Recover | What evidence permits reset or resumption, and who authorises it? |
Make recovery a controlled state transition
A reset does not prove the fault disappeared. Before re-enabling an actuator, re-establish configuration validity, trustworthy measurements and the required physical starting condition. Do not automatically resume an interrupted operation whose outcome is uncertain.
Retain enough diagnostic context to distinguish cause from cascade: initial fault, operating state, relevant timestamps and reset history. Keep storage and reporting bounded so fault handling cannot exhaust resources. Inject faults during start-up, active operation, shutdown and recovery, because the same electrical failure can have different effects in each mode.
Practical exercise
An illustrative engineering exercise; values are assumptions for this lesson.
Design a pump response to missing feedback
A pump controller loses speed feedback during delivery. The motor driver can still run, and immediately removing power may allow fluid to continue moving through the mechanism.
- Separate the observable failure from the possible physical hazards.
- Specify a response that includes energy and flow after the motor command changes.
- Define what must be known before a new delivery is allowed.
Show the worked answer
The observable event is loss of valid speed feedback, not proof that the motor stopped. Analyse excess delivery, insufficient delivery and uncontrolled residual flow. The chosen response may require disabling drive, closing an independently justified flow path and marking the operation’s outcome uncertain. The mechanism and use case determine the correct allocation.
Budget feedback-loss detection and the time to reach the required flow condition. Verify actual flow after the output command changes, including a failed driver and credible pressure conditions. Before restart, establish the fault disposition, valid sensing and a known fluid-path state; reconcile the interrupted delivery rather than automatically repeating it. These are exercise design principles, not a specification for a clinical device.
Review criteria
- The safe response is derived from the physical hazard.
- Residual energy or flow is included.
- Recovery cannot silently repeat an uncertain operation.
Check your understanding
Is a watchdog a complete safety concept?
No. Its diagnostic coverage, servicing logic, reset effect and shared dependencies need justification, and it addresses only certain failure modes.
Why test faults in several operating modes?
Detection may be disabled or different data and energy may be present. A response valid during idle can be inadequate during active operation.
Further learning
- NASA Fault Management HandbookPrinciples and practices for detecting, isolating, reporting and responding to faults.
- NASA Systems Engineering HandbookSystem design, technical risk, verification and validation across the lifecycle.
Safe behaviour must be derived, timed and tested.
Effective fault handling connects hazard, detection, containment, system response and recovery—with evidence that the complete physical system reaches and maintains an acceptable condition.