Verification, debugging and test automation
Build confidence across code, electronics and physical behaviour: verify requirements, preserve diagnostic evidence and automate tests without hiding the system under test.
After this module, you should be able to:
- Choose the right test level for each engineering claim
- Design embedded software for controllability and observability
- Debug anomalies without losing evidence
- Build reliable host, target and hardware-in-the-loop automation
Study guide · Updated · About 9 minutes reading + 20–30 minutes practice
Before you start: Read TEA-104 and TEA-108. Distinguish a requirement, a test procedure and a recorded result.
Verification asks whether evidence supports a defined claim.
Testing is one verification method. Analysis, inspection, review and demonstration may be more effective for particular requirements. The method should be chosen from the claim, risk, failure modes and the level at which behaviour becomes observable.
Place each test where it gives the clearest answer.
| Level | Best suited to | Limitations to remember |
|---|---|---|
| Static verification | Standards, data flow, control flow, interfaces, complexity and defect patterns. | Cannot demonstrate dynamic timing or physical behaviour. |
| Host unit test | Algorithms, state machines, conversions, boundary values and error logic. | Compiler, word size, concurrency and hardware behaviour may differ. |
| Target component test | Drivers, memory layout, interrupts, peripherals and target-specific behaviour. | Fixtures and instrumentation can alter timing or load. |
| Software integration test | Tasks, queues, components, protocols, start-up and recovery sequences. | Simulated hardware models may omit physical faults. |
| Hardware-in-the-loop | Real firmware with controlled sensor, load, network and fault stimuli. | Model fidelity, calibration and fixture control need evidence. |
| System test | End-to-end behaviour, physical performance, user interaction and safe response. | Root cause is harder to isolate and edge cases may be expensive to reproduce. |
Use many fast tests close to the code, but retain enough target and system testing to address compiler, processor, timing, electronics and physical behaviour. Trace each safety or security control to a method that can actually challenge it.
Automation needs control of both product and test system.
- Identify the configuration. Record source revision, build options, toolchain, binary hash, board revision, calibration and fixture version.
- Control preconditions. Reset state, clear retained data where required, stabilise supplies and verify instruments.
- Stimulate through defined interfaces. Use public APIs and electrical interfaces; reserve internal hooks for justified diagnostic tests.
- Measure independently. Observe physical outputs, buses and timing rather than accepting the product’s own report alone.
- Apply objective oracles. Include tolerances, time windows, permitted intermediate states and unambiguous pass/fail rules.
- Preserve artefacts. Store logs, traces, waveforms, versions and failure context automatically.
- Detect test-system faults. Distinguish product failure, fixture failure, infrastructure failure and inconclusive execution.
Worked example: communication-loss safe response
A HIL rig must show that a controller removes drive within 250 ms after loss of valid host messages.
The acceptance point is measured electrically, not from a firmware log that shares the same clock and logic as the function under test. The rig repeats the test around timeout boundaries and under maximum background load, then retains bus capture, output waveform and product configuration with the result.
Preserve the symptom before changing the system.
Exact configuration, preconditions, stimulus, timing, symptom and occurrence rate.
Software events, bus traffic, electrical signals and physical outputs on a common timeline.
Predicted evidence for plausible causes and one controlled experiment at a time.
Trigger, underlying defect, propagation path and explanation of all material observations.
A test that fails before the fix, passes after it and covers neighbouring conditions.
The fastest trustworthy automated level that will prevent recurrence.
Common failure patterns
Single-stepping, halted peripherals or altered optimisation removes the timing-dependent fault.
The same defective software both performs the function and declares that it succeeded.
Intermittent automation failures are retried away until genuine timing defects are normalised.
Executed lines are counted without showing that requirements, boundaries and failure modes were challenged.
Choose an oracle independent of the implementation
A test oracle is the basis for deciding whether behaviour is correct. Reusing the product’s conversion code inside the test can reproduce the same error on both sides. Prefer an independently derived expected value, an analytical relation or a suitable reference instrument.
Map each claim to the most useful evidence level. Unit tests can explore parsing boundaries cheaply; target tests reveal processor and peripheral behaviour; hardware-in-the-loop tests exercise the physical interface under repeatable conditions. End-to-end tests establish integration but can make failures harder to locate.
A coverage percentage does not measure requirements completeness. Structural coverage shows what code was exercised under its defined metric; requirements coverage shows which claims have evidence. Review both gaps and unnecessary behaviour rather than treating a single percentage as release permission.
Make automation detect invalid tests
A test can report a pass because the stimulus never reached the product or the recorder stopped early. Verify fixture health, stimulus delivery, time synchronisation and data completeness. Use a deliberately failing case to demonstrate that the harness can detect the intended fault.
Separate product failure from test-system error and inconclusive evidence. A lost instrument connection is not proof that the product failed, and it cannot be accepted as a pass. Retain raw observations before generating a summary so later review can distinguish the categories.
Control fixture hardware, scripts, instrument settings, calibration status and product configuration. Automated repetition is valuable only when the conditions and expected result are known. A retry policy that silently reruns failures until one passes destroys evidence of intermittent behaviour.
| Result | Meaning | Action |
|---|---|---|
| Pass | Valid evidence meets the specified criterion. | Retain result and configuration. |
| Fail | Valid evidence contradicts a criterion. | Investigate product or requirement defect. |
| Invalid | The fixture or procedure did not perform the intended test. | Repair the test and repeat with the reason recorded. |
| Inconclusive | Evidence is insufficient to decide. | Resolve the uncertainty before closure. |
Turn debugging into regression knowledge
Preserve the original symptom and configuration before modifying the system. Reduce the scenario until it distinguishes the suspected cause from alternatives. A passing run after an unrelated change is weak evidence of a fix unless the causal mechanism is understood.
Add a targeted regression that observes the relevant behaviour, then assess neighbouring risks. A fix to retry timing may affect queue occupancy and response deadlines. Choose additional tests from that impact rather than rerunning an arbitrary large suite and assuming broad volume compensates for a missing case.
Practical exercise
An illustrative engineering exercise; values are assumptions for this lesson.
Repair a false-pass fault-injection test
An automated test disconnects a sensor and expects a fault within 100 ms. It passes whenever a fault message exists in the log, even if that message came from an earlier run.
- Define a reliable test start and a run-specific observation.
- Require evidence that the sensor disconnection actually occurred.
- Specify pass, fail and invalid outcomes and one harness challenge.
Show the worked answer
Use a unique run identity, known initial state and a fresh observation interval. Confirm the injection electrically or through an independently monitored fixture signal. Measure from the defined disconnection event to the required product response, and associate the fault with the current operation. Do not rely on the presence of an unqualified historical log line.
A valid injection followed by late or absent response fails. A relay that did not open, lost acquisition or untrustworthy timing makes the test invalid or inconclusive. Challenge the harness with a deliberately inhibited detector in a controlled test configuration, or a synthetic trace with a response outside the limit, and verify that it cannot pass.
Review criteria
- Evidence is tied to the current run and actual stimulus.
- Timing uses explicit observation points.
- The harness demonstrably detects the failure it claims to test.
Check your understanding
Why retain raw data after generating a report?
It supports independent review, diagnosis and re-evaluation if a threshold or interpretation is later found wrong.
Can a large regression suite replace impact analysis?
No. The suite may omit the behaviour affected by the change. Impact analysis identifies which claims and configurations need new evidence.
Further learning
- Zephyr Test Runner (Twister)An example of automated test execution across emulated and physical target platforms.
- NASA Software Engineering HandbookGuidance covering software verification, testing, reviews, configuration and assurance evidence.
- GoogleTest documentationA widely used framework for repeatable unit and component tests in C++.
Trust evidence that is controlled, independent and repeatable.
Strong embedded verification combines the right test level, known configurations, objective observations and automation that preserves rather than obscures diagnostic evidence.