A model can pass its offline evaluation and still fail as a product. At the edge, the shipped system includes the converted model, runtime, accelerator, enclosure, sensors, update mechanism, operators and recovery path. The useful question is therefore not only does the model work? It is which evidence shows that this exact system keeps working inside its intended operating boundary?
This white paper uses four failure classes as a pre-mortem: the deployed artefact, the sustained operating envelope, the field input and the fleet lifecycle. They are not a statistical claim about every incident, and they are not mutually exclusive. They are a compact way to expose expensive gaps before a pilot becomes a fleet. For each class the paper sets out mechanism, detection signals, mitigations, measurement and the research behind it.
Source status — reviewed 4 November 2026. Mechanisms are anchored in primary documentation, research papers and European Commission summaries of EU law. Thresholds and acceptance limits must be set for the specific product; this paper does not invent universal numbers. Quantities marked hypothetical are illustrations.
Executive summary
- The shipped computation is a chain, not a weight file: ONNX Runtime's documentation states that quantisation "may negatively affect a model's accuracy", and graph partitioning and pre-processing change what runs as well.
- Average accuracy can hide the regression that matters: research on compressed networks found that compression "disproportionately impacts model performance on the underrepresented long-tail of the data distribution", so release gates should bind to critical slices.
- Peak throughput is the wrong purchasing metric: Jetson documentation describes over-current throttling of which "the host OS is not notified", and a 2026 preprint measured one phone losing "nearly half its throughput within two iterations".
- Confidence alone is not a drift alarm: shift benchmarks report "substantially lower out-of-distribution than in-distribution performance", and post-hoc calibration "does indeed fall short" under shift.
- Update and recovery need a threat model and an application-level commit gate: TUF names rollback and freeze attacks, while an A/B scheme alone handles a failed boot, not a release that boots and decides wrongly.
- Exposure before a halt is a design variable: an original reasoning model bounds it by cohort size, install rate, detection latency and halt-propagation latency.
- Evidence expires when the system changes: an original change-impact matrix shows which of the four evidence packets each kind of change invalidates.
The problem and why it matters now
From pilot to fleet
A pilot can be judged by held-out accuracy and a working demonstration unit. A fleet is judged by whether many device-hours of operation, across sites and hardware revisions, stay inside an acceptable boundary, and whether the organisation can change and recover devices when they do not.
A survey of case studies in ACM Computing Surveys, Challenges in Deploying Machine Learning, maps reported challenges to the deployment workflow and concludes that "practitioners face issues at each stage of the deployment process." Operationalizing Machine Learning, based on "semi-structured ethnographic interviews with 18 MLEs", names "three variables that govern success for a production ML deployment: Velocity, Validation, and Versioning". Edge deployment adds physical constraints to each: validation must happen on the target, and versioning must cover hardware and firmware, not only code.
Regulatory and assurance context
- Cyber Resilience Act. The European Commission's CRA summary says the Act addresses "the lack of timely security updates" and "requires manufacturers to handle vulnerabilities during the lifecycle of their products." Reporting obligations "apply as of 11 September 2026", and "The main obligations introduced by the Act will apply from 11 December 2027".
- AI Act. The Commission's AI Act overview lists, for high-risk systems, a "high level of robustness, cybersecurity and accuracy" and requires that "providers have a post-market monitoring system in place". After the AI Omnibus "entered into force on 27 July 2026", the Annex III rules "have been extended to 2 December 2027", and rules for systems "embedded into regulated products (Annex I) have an extended transition period until 2 August 2028". Whether a given edge product is high-risk is a classification question this paper does not answer.
- NIST monitoring report. NIST's AI 800-4, Challenges to the Monitoring of Deployed AI Systems, lists six monitoring categories — functionality, operational, human factors, security, compliance and large-scale impacts — and barriers including "Detecting performance degradation and drift" and "Fragmented logging across distributed infrastructure". It does not offer a universal drift score, and neither should a product team.
None of these texts prescribes the engineering below; they raise the cost of being unable to show what is deployed, monitored and updated.
The shipped system
flowchart LR A["Training checkpoint"] --> B["Conversion and quantisation"] B --> C["Runtime and execution providers"] C --> D["Accelerator and drivers"] D --> E["Enclosure, power and thermal"] E --> F["Sensors and field conditions"] F --> G["Decisions and operators"] G --> H["Monitoring and feedback"] H --> U["Update and recovery"] U --> B B -.-> M1["Boundary 1: artefact"] E -.-> M2["Boundary 2: envelope"] F -.-> M3["Boundary 3: field input"] U -.-> M4["Boundary 4: fleet lifecycle"]
| Boundary | What changes between lab and field | First visible symptom | Primary evidence |
|---|---|---|---|
| 1. Artefact | Numerical format, operator placement, pre- and post-processing | Slice-level regression or unexpected latency | Target-device evaluation of the signed bundle |
| 2. Operating envelope | Ambient, airflow, power, co-resident load, duration | Latency drift after warm-up, queue growth, resets | Sustained traces at envelope corners |
| 3. Field input | Sensors, environment, process, human behaviour | Override, abstention or outcome-signal shift | Monitoring specification plus injected-shift tests |
| 4. Fleet lifecycle | Bundle versions, keys, connectivity, local state | Bricked or silently degraded devices after a release | Update, rollback and recovery fault-injection results |
Technical deep dive: four failure modes
Failure mode 1: the deployed artefact is not the evaluated artefact
Mechanism
Training checkpoints can be converted, fused, quantised or compiled for a target accelerator. Three mechanisms change the computation:
- Numerical transformation. ONNX Runtime's quantisation documentation states: "Quantization is not a loss-less transformation. It may negatively affect a model's accuracy." Static quantisation "first runs the model using a set of inputs called calibration data", so the calibration set is part of the model. A large accuracy drop "may be caused by saturation", and "Per-channel quantization can improve the accuracy for models whose weight ranges are large."
- Graph partitioning. ONNX Runtime assigns nodes through the execution provider
GetCapability()interface; providers are "ordered by Priority", and a node runs on the preferred provider "if capable, otherwise execute using CPUExecutionProvider." An unsupported operator can therefore move silently to a slower device. - Pipeline skew. Google's Rules of Machine Learning defines training-serving skew as "a difference between performance during training and performance during serving", including "A discrepancy between how you handle data in the training and serving pipelines." On an embedded device that discrepancy can arise from a re-implemented resize, colour order, normalisation or unit conversion.
Why the mean hides it
What Do Compressed Deep Neural Networks Forget? (Hooker et al.) found that top-line accuracy "conceals significant differences": a subset of examples, Pruning Identified Exemplars, is "systematically more impacted by the introduction of sparsity" and these examples "over-index on atypical or noisy images". A White Paper on Neural Network Quantization (Nagel et al.) reports that 8-bit post-training quantisation can achieve close-to-floating-point accuracy, but an aggregate accuracy figure can mask a regression in a rare, consequential class.
Detection signals
- aggregate performance looks acceptable while an important class or operating condition regresses;
- an unsupported operator falls back to a slower execution provider;
- pre-processing differs in colour order, resize, normalisation or sensor units;
- the model hash is correct but the runtime, calibration set or compiler version is not the one that was approved.
Measurement
Gate on the worst critical slice, not on the average:
Here each slice s belongs to the consequence-weighted set S_c, M is the task metric, the reference is the approved floating-point pipeline, the deployed figure comes from the target device running the complete bundle, and the tolerance tau_s is set per slice by the product owner. ONNX Runtime provides "Python APIs for matching weights and activation tensors" between float and quantised models — "This is called quantization debugging" — which can help investigate a failing slice.
Verification gate
Evaluate the deployable bundle on the target device and compare intermediate outputs where the toolchain permits. If a sensitive layer cannot meet the acceptance bar after quantisation, retain greater precision for that part or choose a different target. Do not average a safety- or quality-critical class out of the decision. Bind into one release record:
- source checkpoint and hash;
- conversion/quantisation configuration and tool version;
- calibration/evaluation dataset versions and provenance;
- pre- and post-processing implementation;
- runtime, backend, driver and target-device identity;
- class-, condition- and consequence-specific results;
- the acceptance decision and named owner.
Evidence to keep. A model card is not enough. Keep the signed bundle manifest, toolchain bill of materials, conversion log, operator-placement report, evaluation report, known limitations and rollback target.
Failure mode 2: the target misses its sustained operating envelope
Mechanism
A development kit on an open bench is not the deployed thermal system. Enclosure, ambient temperature, airflow, mounting, dust, power supply and co-resident workloads determine sustainable clocks and throughput. NVIDIA's Jetson Linux power and performance guide for the Orin series documents layers of control between the model and its clock speed:
- Power modes. The module "supports multiple optimized power budgets, such as 10 watts, 15 watts, and 30 watts", selected with
nvpmodel. - Software thermal management. "Thermal management uses trip points to communicate with thermal zones"; throttling reduces frequency and voltage, which "helps to control the temperature"; and "A critical trip point triggers a software thermal shutdown."
- Hardware protection. For over-current "OC alarms", firmware "triggers clock hardware throttling". "Like software throttling, hardware throttling may reduce performance", and "The host OS is not notified of these events." The final failsafe "is a hardware thermal reset, or thermtrip."
These details are one vendor's; the general lesson is that power and thermal controls constrain performance dynamically, and some are invisible unless the team reads the counters — the guide lists sysfs nodes giving "the number of OC events occurred".
stateDiagram-v2 state "Nominal clocks" as Nominal state "Software throttling" as Soft state "Hardware throttling" as Hard state "Degraded product mode" as Degraded state "Thermal shutdown or reset" as Shutdown [*] --> Nominal Nominal --> Soft : trip point crossed Nominal --> Hard : over-current alarm Soft --> Nominal : temperature recovers Hard --> Nominal : event clears Soft --> Degraded : deadline floor missed Hard --> Degraded : deadline floor missed Degraded --> Nominal : headroom restored Soft --> Shutdown : critical trip point Shutdown --> [*]
Degraded product mode is not a platform feature: the product must define it (load shedding, reduced frame rate, safe state or operator message).
Research evidence on sustained load
The 2026 preprint LLM Inference at the Edge (Tummalapalli et al.) benchmarked "Qwen 2.5 1.5B (4-bit quantised) across four platforms" using "a fixed 258-token prompt over 20 warm-condition iterations per device". It reports that "For mobile platforms, thermal management supersedes peak compute as the primary constraint": "the iPhone 16 Pro loses nearly half its throughput within two iterations", and "the S24 Ultra suffers a hard OS-enforced GPU frequency floor that terminates inference entirely", while "the Hailo-10H sustains 6.9 tok/s at under 2 W with near-zero variance". The authors call these "platform-level deployment characterisations for a single model and prompt type".
Measurement
MLPerf Inference (ISCA 2020) sets out a method for comparable inference measurement. Its inference rules use a "90%-ile early-stopping latency estimate" for Single Stream, a "99%-ile early-stopping latency estimate" for Multistream, and a minimum duration of "600 seconds". Two lessons transfer: report a tail percentile under a defined load, and fix a minimum duration. A benchmark duration does not prove thermal steady state; the product test must run until temperature and clocks stop drifting at the hottest credible corner.
An original reasoning metric, the sustained headroom ratio, makes the purchasing question explicit:
with co-resident services running. A value below 1 means the product misses its requirement inside its declared envelope; a value close to 1 leaves no margin for dust, ageing fans or a heavier model update. A companion ratio, steady-state over cold-start throughput, shows how misleading a short demonstration was. Neither has a universal threshold.
Detection signals
- latency or frame deadlines degrade after warm-up;
- the queue grows even though short tests passed;
- voltage or thermal events trigger throttling, resets or unavailable accelerators;
- a co-resident process creates memory pressure or steals compute during a critical window.
Verification gate
Define an operating-envelope matrix before hardware selection:
| Variable | Boundary to test | Evidence |
|---|---|---|
| Ambient and enclosure | Minimum/maximum intended conditions and worst credible cooling state | Instrumented chamber or representative enclosure run |
| Power | Nominal, brownout and restart behaviour | Power trace, fault injection and recovery log |
| Workload | Peak request/frame distribution plus co-resident services | Queue, latency and dropped-work telemetry |
| Duration | Long enough to reach steady thermal behaviour | Sustained trace, not a short peak result |
| Degradation | What the product does when the target is exceeded | Load shedding, safe state and operator message |
| Hidden throttling | Events the operating system does not report | Throttle and over-current counters logged with each run |
The acceptance metric is the sustained floor inside the declared envelope, with a defined degradation path — not the highest number observed.
Evidence to keep. Store the enclosure/BOM revision, firmware and power mode beside the run. A thermal result cannot be reused after a heatsink, fan policy, enclosure or workload changes without assessing whether revalidation is required.
Failure mode 3: the field input leaves the validated boundary
Mechanism
Physical systems do not receive a stationary stream. Sensors age, move, get dirty or get replaced. Lighting, weather, surfaces, process recipes, firmware and human behaviour change. The model may be unchanged while the relationship between input, context and required decision changes. Three kinds of change need different detectors: the input distribution moves (covariate shift), the sensor reports something physically different for the same scene (measurement fault), or the correct decision for the same input changes (concept change). Input monitoring can reveal some distribution shifts and measurement faults; determining whether the required decision has changed needs additional context or outcome evidence.
Research evidence
- Shift degrades accuracy. WILDS assembles real shifts, "such as shifts across hospitals for tumor identification; across camera traps for wildlife monitoring", and reports that "standard training yields substantially lower out-of-distribution than in-distribution performance."
- Everyday corruptions are measurable. ImageNet-C (ICLR 2019) "evaluates performance on common corruptions and perturbations" rather than adversarial attacks. For a camera product, a corruption suite rehearses dirt, blur and lighting change.
- Confidence is not a reliable alarm. Can You Trust Your Model's Uncertainty? (NeurIPS 2019) studied "the effect of dataset shift on accuracy and calibration" and found that "traditional post-hoc calibration does indeed fall short", while "some methods that marginalize over models give surprisingly strong results".
- Detection must be designed. Failing Loudly (NeurIPS 2019) found that "a two-sample-testing-based approach, using pre-trained classifiers for dimensionality reduction, performs best" across the shifts it explored.
The NIST AI RMF Core asks that performance be "demonstrated for conditions similar to deployment setting(s)" (MEASURE 2.3), that "Limitations of the generalizability beyond the conditions under which the technology was developed are documented" (MEASURE 2.5), that the system "can fail safely, particularly if made to operate beyond its knowledge limits" (MEASURE 2.6), and mechanisms "to supersede, disengage, or deactivate" (MANAGE 2.4).
Detection signals by layer
| Signal layer | Example signal | What it can detect | What it cannot show on its own |
|---|---|---|---|
| Sensor health | Exposure, focus, frame drops, calibration age | Measurement faults before decisions degrade | Whether decisions are wrong |
| Input distribution | Two-sample test on learned features | Covariate shift relative to validation data | Harm; a shift may be benign |
| Model output | Confidence, abstention and class-mix trends | Changes in model behaviour | Correctness, given calibration limits under shift |
| Human interaction | Override and manual-correction rates | Disagreement where operators are present | Errors operators do not see |
| Outcome | Downstream rejects, rework, audits, labelled samples | Actual consequence | Fast detection; labels arrive late |
Fast signals are not evidence of harm, and evidence of harm arrives slowly; no single drift score carries the decision.
sequenceDiagram participant Dev as Edge device participant Fleet as Fleet backend participant Own as Response owner participant Ops as Site operator Dev->>Dev: Log decision with sensor id and context Dev->>Fleet: Send aggregates and sampled inputs Fleet->>Fleet: Compare against validation baseline Fleet->>Own: Raise alarm with affected cohort Own->>Ops: Request inspection or labelled samples Ops-->>Own: Return findings Own->>Fleet: Decide recalibrate, retrain, restrict or withdraw Fleet->>Dev: Apply configuration or fallback mode Note over Dev: Safe mode must work if the backend is unreachable
Verification gate
Build monitoring around the failure consequence, not around whichever metric the MLOps platform exposes by default. At minimum:
- specify the intended input and operating boundary;
- log sensor identity, calibration/firmware and relevant context with each decision or aggregate;
- monitor data quality, system functionality, operator intervention and outcome signals separately;
- predefine the review, fallback and data-capture action for each alarm;
- test detection with known shifts and sensor faults;
- prove that the safe/degraded mode works when monitoring itself is unavailable.
A distribution-distance statistic can be useful, but it is not evidence of harm by itself. Likewise, a stable input distribution does not prove the model remains correct.
Evidence to keep. Retain the monitoring specification, alarm rationale, response owner, field-event samples, override/abstention analysis and decisions about recalibration, retraining or withdrawal, versioned against device, sensor and model identities.
Failure mode 4: the fleet cannot update or recover safely
Mechanism
A lab team can reinstall a broken machine; a distributed device may be remote, intermittently connected or costly to reach. Model updates interact with application code, schemas, configuration and cached state, and a downloaded image is not a product recovered to a known-good state.
- Threat model. The Update Framework names rollback attacks (files "older than those the client has already seen"), indefinite freeze attacks and mix-and-match attacks, and its principles include "Trust should expire if it is not renewed" and "Multiple key usage and a threshold/quorum of signatures."
- Fleet-scale verification. Uptane, "a secure software update framework for ground vehicles", separates an Image repository from a Director repository that "instructs ECUs as to which images will be installed". It requires that "Full verification SHALL be performed by Primary ECUs" and that "ECUs SHALL have a secure source of time." The pattern fits any fleet where what exists and what this device should run are separate decisions.
- Robust installation. The IETF's RFC 9019 states that "Devices must not fail when a disruption, such as a power failure or network interruption" occurs, and that "A device presented with an old but valid manifest and firmware must not be tricked into installing such firmware".
- Recovery. Mender's documentation describes "a dual redundant scheme (also known as A/B scheme)": "On the first boot following an update, the Mender Client will commit the update"; if the device reboots before committing, the bootloader "will roll back to the previous version by flipping the active and inactive partitions back again." NIST's SP 800-193 frames the goal as protection plus "detecting unauthorized changes that occur" and "recovering from attacks rapidly and securely".
An A/B scheme solves a failed boot, not a release that boots, passes an operating-system check and then decides wrongly. The commit step must be gated on application-level health.
stateDiagram-v2 state "Metadata verified" as Verified state "Written to inactive slot" as Written state "Trial boot" as Trial state "Application health check" as Health state "Committed" as Committed state "Rolled back" as Rolled [*] --> Verified : signatures, version and compatibility pass Verified --> Written Written --> Trial : reboot Trial --> Health : boots Trial --> Rolled : boot fails or reboot before commit Health --> Committed : inference path and config pass Health --> Rolled : health criteria fail Rolled --> [*] Committed --> [*]
Staged rollout and an exposure model
Google's SRE workbook defines canarying as "a partial and time-limited deployment of a change in a service and its evaluation", notes that impact "is directly proportional to the amount of traffic exposed to defects", and requires that "you must be able to compare the canary signals to the control signals." For a fleet, the equivalent of traffic is devices running the release.
An original reasoning model, derived from that proportionality rather than measured, bounds how many devices are exposed before a release is stopped:
where C_k is the current cohort size, r the rate at which devices install the release, t_detect the time from first install to a trustworthy alarm, and t_halt the time for a stop decision to reach devices. Field-only defects detected through slow outcome signals make t_detect long, so cohort caps matter more than rollout speed. And for intermittently connected devices t_halt includes the time until the next check-in; a device that has already downloaded a bundle may install it before it learns of the halt. Halting a rollout and recalling a release are separate capabilities to test.
Detection signals
- a valid but incompatible combination of model, runtime and application reaches a device;
- power loss leaves no bootable or operational image;
- a health check validates the operating system while the inference path is broken;
- rollback restores binaries but not a compatible configuration or local database;
- a rollout reaches too much of the fleet before a field-only fault is visible.
Verification gate
Treat update and recovery as a product capability with its own acceptance tests:
- signed metadata and artefacts, trusted key roles and rotation/revocation procedure;
- hardware- and dependency-aware compatibility manifest;
- atomic installation or equivalent interruption-safe design;
- application-level health criteria, not only successful boot;
- automatic rollback or an explicitly tested recovery route;
- staged cohorts with stop conditions and an authorised decision owner;
- fault injection for interrupted download, interrupted installation, invalid signature, incompatible bundle, failed health check and unavailable backhaul;
- a field-service path for devices that cannot recover remotely.
Evidence to keep. Keep the release decision, signatures, cohort progression, health results, stop/rollback events and device state transitions. The update record should answer which device received which complete bundle, why it advanced, and what recovery path remains.
Research evidence: what the sources do and do not establish
| Source | Type | Supports | Does not establish |
|---|---|---|---|
| Hooker et al. | Research paper (arXiv) | Compression can hit atypical examples harder than the mean shows | Effect size for a given product model |
| Nagel et al. | Research white paper (arXiv) | 8-bit PTQ can achieve close-to-float accuracy | That a specific slice survives |
| Tummalapalli et al. | Preprint; one model, one prompt, four platforms | Sustained behaviour can diverge sharply from peak | Hardware rankings; enclosure behaviour |
| WILDS; ImageNet-C | Benchmarks | Shift and corruption degrade accuracy | The shift a given site will see |
| Ovadia et al.; Rabanser et al. | Peer-reviewed (NeurIPS 2019) | Weak calibration under shift; two-sample detection works in their tests | Detection latency or false-alarm rate in a fleet |
| ONNX Runtime; Jetson Linux; Mender | Vendor documentation | How those mechanisms behave | Other vendors' stacks |
| MLPerf rules; TUF; Uptane; RFC 9019; NIST | Benchmark rules, standards, guidance | What sound measurement or update design includes | That any product meets them |
| Commission CRA and AI Act pages | Official summaries of EU law | Which obligations exist and when | Whether a product is in scope |
Research results come from public benchmarks under controlled shifts, so transfer to a specific site is an assumption to test. Vendor documents are authoritative for mechanisms but silent on design margins. Product-specific evidence fills the gap.
Business and decision analysis
Cost drivers
Each failure mode's cost can be expressed in units a product owner can estimate:
- Artefact: engineering hours to rebuild evaluation on target hardware; accelerator-hours for calibration and slice evaluation; device-hours of incorrect decisions after a wrong release.
- Envelope: chamber or enclosure test hours; hardware margin bought (watts of thermal design power, cooling); device-hours in degraded mode.
- Field input: labelling hours for outcome signals; backhaul bandwidth for sampled inputs; operator time on overrides and inspections.
- Lifecycle: storage for a second partition; engineering hours for keys and fault injection; technician hours and downtime for devices that cannot recover remotely.
The asymmetry is that pre-deployment costs are bounded and known, while field costs scale with fleet size and with time-to-detect.
Decision table
| Decision | Option A | Option B | Criteria that should drive it |
|---|---|---|---|
| Fit model to hardware | Quantise or compress further | Buy compute headroom | Worst critical-slice delta versus sustained headroom ratio and thermal budget |
| Where monitoring runs | Aggregates on device | Raw samples to backend | Backhaul capacity, privacy constraints, labelling latency |
| Recovery mechanism | A/B partitions with commit gate | Single image plus recovery partition or field service | Storage budget, effort of a site visit, interruption risk |
| Update stack | Open framework (TUF- or Uptane-based) | Vendor-managed service | Key custody, multi-vendor hardware, audit evidence the organisation must hold |
| Rollout policy | Fast rollout, small first cohort | Slow rollout, long canary window | Detection latency of the dominant signal and halt-propagation latency |
Organisation and ownership
The boundaries cross organisational lines: machine learning, hardware, operations and platform security. Each evidence packet needs one named owner with authority to block a release, and the advance-or-hold decision needs an owner who can weigh customer consequence against all four. NIST's MANAGE 2.4 wording, responsibilities "assigned and understood", is a useful test.
A method to apply: the Boundary Evidence Review
The Boundary Evidence Review is a Hyperion reasoning device, not a published standard and not field evidence. It combines the four evidence packets with a change-impact matrix, so evidence is treated as something that expires when the system changes.
Step 1: one review, four packets
| Boundary | Decision question | Minimum packet |
|---|---|---|
| Artefact | Is the shipped computation the one that was approved? | Bundle identity, target-device evaluation, limitations |
| Operating envelope | Does it meet the sustained requirement in context? | Thermal/power/workload traces and degradation test |
| Field input | Will the team know when the validated boundary changes? | Monitoring specification, injected-shift tests, response playbook |
| Fleet lifecycle | Can it be changed and recovered without losing control? | Signed update, staged rollout, rollback and field-recovery tests |
A gate passes only when the evidence, acceptance threshold and owner are explicit. The demo worked is not an entry in any packet.
Step 2: the change-impact matrix
The original insight is that the boundaries are coupled: a change aimed at one can silently invalidate evidence held for another. The matrix is reasoning from the mechanisms above; teams should adapt it to their architecture. "Re-run" means the packet's acceptance evidence is no longer valid; "re-check" means a bounded assessment decides whether a re-run is needed.
| Change | Artefact | Envelope | Field input | Lifecycle |
|---|---|---|---|---|
| New weights, same architecture | Re-run | Re-check | Re-baseline monitors | Re-test compatibility |
| New quantisation or compiler version | Re-run | Re-check | Re-baseline monitors | Re-test compatibility |
| Runtime or driver update | Re-run | Re-check | Holds | Re-test compatibility and rollback |
| Enclosure, fan policy or power mode | Holds | Re-run | Holds | Holds |
| New co-resident service | Holds | Re-run | Holds | Re-test health criteria |
| New sensor model or mounting | Re-run pre-processing tests | Re-check power | Re-baseline and re-test detection | Re-test compatibility |
| New site or process recipe | Holds | Re-check ambient | Re-run | Holds |
| New signing keys or update service | Holds | Holds | Holds | Re-run |
flowchart TD
A["Proposed change"] --> B["Look up change-impact matrix"]
B --> C{"Any packet marked re-run?"}
C -- "yes" --> D["Re-run affected packets on target"]
C -- "no" --> E{"Any packet marked re-check?"}
E -- "yes" --> F["Owner assesses and records rationale"]
E -- "no" --> G["Record change against current evidence"]
D --> H{"All thresholds met?"}
F --> H
H -- "yes" --> I["Advance to staged rollout"]
H -- "no" --> J["Hold, redirect or stop"]
G --> I
Worked example (hypothetical)
Every quantity here is hypothetical, chosen for illustration only. A camera-based inspection device on a production line must classify parts within a frame deadline. A proposed bundle updates the runtime and moves the model from 16-bit to 8-bit integer weights to free compute for a second camera.
- Matrix lookup. New quantisation plus a runtime update mark the artefact packet re-run, the envelope re-check, the field-input monitors re-baseline and the lifecycle packet re-test compatibility and rollback.
- Artefact. Suppose the overall metric drops by a fraction of a point but the rare "hairline crack" slice drops by more than its tolerance. Quantisation debugging points to the first convolution block; keeping that block at higher precision brings the slice back within tolerance. The average alone would have passed.
- Envelope. With the second camera running, suppose the steady-state tail throughput at the hottest enclosure corner gives a sustained headroom ratio of 0.9, while the cold-start figure gave 1.4. Buying on the cold-start figure would have shipped a deadline miss, so either the second camera or the thermal design changes.
- Lifecycle. Suppose devices check in every few hours and the slowest reliable alarm (rework outcomes) takes about a day. Without a hold between cohorts, later cohorts could receive the release before an alarm arrives, so the first cohort is capped and held for the full outcome-signal window.
The change that looked like "just a model update" touched all four boundaries.
Implementation roadmap
| Phase | Work | Exit evidence |
|---|---|---|
| 1. Boundaries and owners | Write the operating boundary; define consequence-weighted slices and tolerances; name packet owners | Boundary specification and owner list |
| 2. Artefact | Build the seven-item release record; evaluate the complete bundle on target; keep quantisation-debugging output for near-tolerance slices | Per-slice deltas and operator-placement report |
| 3. Envelope | Sustained traces at envelope corners with co-resident load; log throttle and over-current counters; test degraded mode | Sustained headroom ratio with traces |
| 4. Field input | Signals per layer and action per alarm; rehearse shifts and sensor faults; test safe mode offline | Injected-shift results and response playbook |
| 5. Lifecycle | Signed metadata with key roles, expiry and revocation; commit gated on application health; full fault-injection set | Fault-injection report including rollback of configuration and state |
| 6. First staged rollout | Size cohorts from the exposure model; compare canary with control devices | Recorded advance and stop decisions |
| Ongoing | Apply the change-impact matrix to every release | Revalidation record per change |
Limitations and open questions
- No universal thresholds. No cited source gives an acceptable slice delta, headroom ratio, drift statistic or cohort size; those are product, sector and regulatory decisions.
- Transfer is unproven. Shift and corruption results come from public datasets; the sustained-load result covers one model and prompt on four platforms.
- Vendor specificity. The thermal and power mechanisms quoted come from one vendor; other accelerators expose different controls and may not expose throttle counters.
- Concept change is slow to detect. When the correct decision changes for the same input, input monitoring alone does not reveal it; it needs additional context or outcome evidence, and the latency of that evidence bounds any monitor's response.
- The exposure model is a simplification. It assumes a constant install rate and one detection time; real fleets have heterogeneous connectivity and several alarm sources.
FAQ
Is 8-bit quantisation safe to ship?
It depends on the slices that matter. Nagel et al. report that 8-bit post-training quantisation can achieve close-to-floating-point accuracy; other work shows compression can hurt atypical examples disproportionately. Gate on the worst critical slice measured on the target.
How long should a thermal soak test run?
Until clocks, temperature and tail latency stop drifting at the hottest credible enclosure condition, with co-resident load running. A benchmark minimum such as MLPerf's 600 seconds fixes comparability, not thermal steady state.
Can model confidence serve as the drift alarm?
Not on its own. Calibration has been shown to degrade under dataset shift, so confidence should be one layer among sensor health, input-distribution tests, operator interaction and outcome signals.
Does an A/B partition scheme solve recovery?
It solves a failed boot. It does not catch a release that boots but decides wrongly, nor restore incompatible configuration or local data, unless the commit is gated on application-level health and rollback covers state.
The system-level conclusion
Edge AI failures can be integration and lifecycle failures expressed through a model-shaped interface. That is why product ownership matters: somebody must connect customer consequence, model evaluation, hardware envelope, operations and recovery before the next commitment. For the placement decision upstream of these boundaries, see Cloud, Edge or Controller? A Decision Architecture for Industrial AI; for robots, see Reliable Autonomy: From Robot Demonstration to Sustained Duty.
Use this paper as a pre-mortem, then tailor the thresholds and evidence to the intended use, sector standards and applicable law. For a structured review, the Physical AI Product System and Product Decision Review turn these packets into an explicit advance, hold, redirect or stop decision.
