Verification and test engineers
Create trustworthy evidence by challenging requirements, controlling test conditions and observing the complete embedded system under normal and adverse operation.
After this module, you should be able to:
- Translate requirements into objective verification methods
- Select the right test level and representative environment
- Design reproducible automation and traceable results
- Investigate failures without weakening independence
Study guide · Updated · About 8 minutes reading + 20–30 minutes practice
Before you start: Read TEA-109. TEA-105 and TEA-106 provide the timing and fault-response background.
Verification asks whether the claim is supported.
Embedded-system testing must control software identity, hardware revision, fixtures, instruments, calibration, stimuli, timing and environment. The role does more than execute procedures: it exposes ambiguity, improves observability and designs evidence that another competent person can reproduce.
Build a verification strategy, not a pile of tests.
| Area | Question | Evidence |
|---|---|---|
| Method selection | Is test, analysis, inspection or demonstration most credible? | Verification plan and rationale |
| Test level | Which claims belong at unit, integration, bench or system level? | Coverage map |
| Environment | Which loads, interfaces, timing and physical conditions matter? | Facility and fixture specification |
| Observability | How will state, timing and physical outcomes be measured? | Instrumentation design |
| Result integrity | Can the exact configuration and raw result be recovered? | Immutable result package |
Use automation to increase control
Automation should identify the unit under test, configure equipment, check prerequisites, timestamp raw observations, evaluate explicit criteria and preserve logs. A fast script that hides configuration is not reliable evidence.
Design tests from risks and boundaries.
- Review testability early. Challenge ambiguous, compound or unobservable requirements.
- Partition coverage. Put each claim at the lowest credible level, then retain system confirmation.
- Define the oracle. State expected outputs, timing, tolerances and allowed transients.
- Control the setup. Record versions, calibration, topology and environmental conditions.
- Exercise faults and edges. Include limits, transitions, overload and corrupted inputs.
- Triage without erasing history. Preserve failing evidence and link reruns to the correction.
Worked hand-off: sensor timeout
The requirement defines maximum detection and safe-state times. The test fixture stops sensor traffic at controlled phases, measures output disable independently and captures internal events. Runs cover start-up, steady state and recovery. Results bind the firmware build, board revision, fixture version and raw timing traces.
Make the result reviewable and repeatable.
Scope, methods, levels, responsibilities and independence.
Setup, preconditions, stimuli, observations and criteria.
Hardware, software, tools, fixtures and calibration.
Logs, traces, captures and environmental measurements.
Pass, fail or anomaly against a cited criterion.
Requirements, risks and unresolved gaps.
Common traps
The test repeats wording without defining measurable criteria.
The environment omits latency, loading and electrical faults.
Reruns replace investigation of intermittent failures.
A result cannot identify the tested binary or board.
Review testability before a requirement is baselined
Ask what observation would show that a requirement is met or violated. “Recover gracefully” needs a defined final state, time limit and data policy. Agree tolerances and timing points before selecting instruments or automating the procedure.
Map requirements and risks to methods. Some claims need analysis, some inspection and some testing at the physical boundary. A software-only test can verify command logic but cannot establish the stopping distance of an actuator. Place evidence where the claimed behaviour is actually observable.
Identify prerequisites that can invalidate the result: calibration, environmental condition, load, configuration and fixture health. Record them as test conditions so another engineer can judge whether the conclusion applies to the released product.
Make independence practical and proportionate
Independence can mean a separately derived expected result, a different reviewer or an organisational requirement imposed by the applicable assurance framework. Define the needed form and record how it is achieved. A different person running the same flawed oracle does not remove a shared logic error.
Use boundary and adverse cases deliberately. Cover minimum and maximum inputs, transitions, stale data, resource limits and fault combinations justified by analysis. Avoid counting many similar nominal cases as broad coverage. Review the cases that could most plausibly defeat the design’s assumptions.
Challenge the test harness. Confirm that an absent stimulus, stale log or disconnected instrument cannot create a pass. Keep invalid and inconclusive outcomes separate from product failures, with a controlled repeat policy.
| Review question | Why it matters |
|---|---|
| Was the intended stimulus applied? | The fixture may fail silently. |
| Is the expected result independent? | Shared calculation errors can create false agreement. |
| Is this the release configuration? | Debug settings and dependencies can alter behaviour. |
| Are all anomalies dispositioned? | An incomplete run is not complete evidence. |
Report a conclusion with its actual limits
A verification report should connect claim, method, configuration, result and anomaly disposition. Explain what was not established when coverage or environmental scope is limited. Avoid saying that the whole product is verified when only a subsystem or selected version was evaluated.
When a defect is fixed, reproduce the failing condition or retain a convincing causal explanation if reproduction is impractical. Select regression from change impact. The test engineer’s role includes challenging whether the proposed evidence answers the question, not merely executing a supplied script.
Practical exercise
An illustrative engineering exercise; values are assumptions for this lesson.
Plan evidence for a 100 ms safe-response requirement
A system must reach a defined output state within 100 ms of sensor disconnection. A proposed test measures the delay from firmware fault-flag assertion to a logged Stop message.
- Identify the portions of the requirement omitted by that measurement.
- Define an end-to-end test with an independent timing reference.
- State how measurement uncertainty and invalid fixture operation will be handled.
Show the worked answer
The proposed test omits detection time and the physical response after the log message. Trigger on the defined disconnection event and observe the required output or physical state using an appropriate common timing basis. Keep a software trace as diagnostic evidence, not the sole proof of completion.
Challenge worst relevant event phase, load and operating condition. Confirm that the fixture opened the circuit and that the output measurement remained valid. Include measurement uncertainty in the acceptance decision, particularly near 100 ms. A missing injection or acquisition failure produces an invalid or inconclusive run and must not be counted as a pass.
Review criteria
- The measured interval matches the requirement.
- The final physical condition is observed.
- Uncertainty and invalid tests have explicit treatment.
Check your understanding
Is a test report complete when all recorded cases pass?
Only if required coverage, configuration validity and unresolved anomalies have also been assessed.
What should drive regression selection?
The changed behaviour, affected interfaces, shared resources and risk controls, supported by impact analysis.
Further learning
- Zephyr Twister test runnerAn official example of embedded test configuration and automation.
- TEA-109 · Verification, debugging and test automationVerification strategy, observability and automated evidence.
Test the claim, control the context.
Credible verification combines an observable requirement, a representative method, controlled configuration and evidence that survives independent review.