The 90-Day Decision Clock for a Physical AI Pilot: Week-by-Week Mechanics, Exit Criteria and Ownership

A Physical AI pilot should not become an indefinite research program. Set a decision clock. Ninety days is a useful default for forcing assumptions into the open, assigning owners and collecting the evidence needed for a clear recommendation.

It is not a delivery guarantee. Product scope, hardware, field access, safety obligations and regulatory path can require a different timeline. The clock governs the decision process; the evidence governs the product plan.

This white paper extends that method into week-by-week mechanics, exit criteria, measurement and organizational ownership. It is a public method, not an account of a Hyperion client engagement. The examples are illustrative and contain no claimed outcome.

Executive summary

• The clock decides, it does not deliver. Ninety days is a governance default for reaching one of four written verdicts (go, redirect, sequence or stop), not a promise that a pilot reaches production.
• The model is the small part. Published engineering work describes the ML code as a small fraction of a real-world ML system, so a pilot that tests only the model has not tested the product.
• Field conditions are where evidence breaks. Research on in-the-wild distribution shifts reports substantially lower out-of-distribution performance, so the clock reserves its middle weeks for the real site, sensors and operators.
• Zero failures is weak evidence unless trials are many. Zero failures in N independent trials bounds the failure rate at about 3/N with 95% confidence, so the reliability a ninety-day pilot can show should be computed in week 1.
• Readiness is a weakest-link property. A Google production-readiness rubric scores a system by the minimum of its section scores; the clock applies that rule, so a strong model cannot offset a missing rollback path.
• Someone must be able to stop the work. The NIST AI RMF calls for documented roles and for mechanisms to deactivate systems inconsistent with intended use; the clock gives that authority to a named sponsor on day 1.
• Regulation shapes the timeline. The EU Machinery Regulation applies from 20 January 2027; where learning-based modules ensure safety functions, conformity evidence is a decisive assumption, not a later phase.

The problem and why it matters now

A pilot that never ends is not neutral. It consumes engineering time, field access and sponsor attention while deferring the one output that matters: a decision about whether the product case survives contact with reality. The pattern to guard against is activity substituting for evidence: more demos, a longer backlog, another integration sprint, and no written answer.

Three strands of published work explain why this happens and why it happens more sharply in Physical AI.

First, the model is a minority of the system. In their NeurIPS paper on hidden technical debt in machine learning systems, Sculley and colleagues state that "only a small fraction of real-world ML systems is composed of the ML code" and that "the required surrounding infrastructure is vast and complex." Their CACE principle, "Changing Anything Changes Everything", describes how entangled inputs prevent isolated improvements. A pilot that measures model accuracy has measured the smallest box in that figure.

Second, deployment problems appear at every stage. A survey of published deployment case studies by Paleyes, Urma and Lawrence concludes that "practitioners face issues at each stage of the deployment process."

Third, data problems compound late. Interviewing 53 practitioners, Sambasivan and colleagues describe "data cascades" as "compounding events causing negative, downstream effects from data issues" and report them as "pervasive (92% prevalence)" in that sample. The 92% figure describes their interviewees, not the industry; the mechanism is what matters here. In a pilot, that mechanism could mean a labelling choice made in week 2 surfacing as a field failure in week 9 (an illustration, not a reported case).

Physical AI adds hardware, safety and site access to all three. Robot deployment is growing: the International Federation of Robotics reports "542,000 robots installed in 2024 - more than double the number 10 years ago". Regulation is also addressing learning-based behaviour in safety functions. The EU-OSHA summary of the EU Machinery Regulation states that "The Regulation applies from 20 January 2027" and that it "intends to better cover new technologies such as autonomous mobile machinery (robots)" and "artificial intelligence (AI), where specific modules of AI using learning techniques ensure safety functions." The same page states that "Machinery and related products ('high risk machinery products') are subject to third party conformity assessment procedures." A pilot that ignores its product's regulatory route may be testing a product that cannot ship as designed.

The pilot's real output is therefore a decision under uncertainty. The NIST AI RMF Core makes that explicit in MANAGE 1.1: "A determination is made as to whether the AI system achieves its intended purposes and stated objectives and whether its development or deployment should proceed." The clock exists to make that determination on time and on evidence.

Technical deep dive: what the clock actually tests

The four layers of a Physical AI product candidate

A product candidate has at least four layers, and each fails differently. The clock tests all four, because a failure in any one of them can reverse the recommendation.

The table is a reasoning device, not a standard taxonomy; it stops the pilot collapsing into the first row.

Why field conditions dominate the middle of the clock

The WILDS benchmark from Koh and colleagues states that distribution shifts "can substantially degrade the accuracy of machine learning (ML) systems deployed in the wild," that "standard training yields substantially lower out-of-distribution than in-distribution performance," and that "this gap remains even with models trained by existing methods for tackling distribution shifts." For a Physical AI pilot, the distribution shift is concrete: a different camera unit, a new lighting condition, a faster line, a worn gripper, a new operator.

Simulation helps but does not settle the question. Tobin and colleagues frame the problem as "bridging the 'reality gap' that separates simulated robotics from experiments on hardware" and report a real-world object detector "accurate to 1.5 cm" trained "using only data from a simulator with non-realistic random textures." Under its stated setup that is a meaningful result. It makes simulation a credible input, not field acceptance; transfer must still be tested on the actual hardware.

NASA's technology readiness levels draw the same line in a different domain: "A TRL 6 technology has a fully functional prototype or representational model," while "TRL 7 technology requires that the working model or prototype be demonstrated in a space environment." By analogy (TRLs were written for space systems), a pilot that has only run in a lab or simulator has not yet been demonstrated in its operational environment, however good the demo looks.

The clock therefore grades every piece of evidence by where it was produced, and limits what each grade may support. The grading below is a reasoning device for the evidence register, not a standard.

flowchart TD
    E1["Estimate or vendor claim"] --> U1["May support: which test to run first"]
    E2["Simulation"] --> U2["May support: design choices and test design"]
    E3["Lab or bounded field test"] --> U3["May support: envelope hypotheses"]
    E4["Field trial on actual site and hardware"] --> U4["May support: field claims within logged conditions"]
    E5["Repeated field operation with monitoring"] --> U5["May support: reliability bounds and residual-risk acceptance"]
    E1 -.->|"never promoted by repetition"| E4

Measurement: four families of metrics

The NIST AI RMF MEASURE 2.3 asks that performance or assurance criteria be "measured qualitatively or quantitatively and demonstrated for conditions similar to deployment setting(s). Measures are documented." The clock turns that into four metric families, each logged with its test conditions.

1. Task metrics under field conditions. Detection or grasp success, false accept and false reject rates, broken down by the conditions that vary on site (lighting, speed, part variant, operator).
2. Autonomy metrics. Interventions per operating hour, safe-state entries, time to recover after an intervention. These describe how the product behaves as a system, not how the model scores.
3. Delivery metrics. The DORA metrics define change lead time as "the amount of time it takes for a change to go from committed to version control to deployed in production" and failed deployment recovery time as "the time it takes to recover from a deployment that fails and requires immediate intervention." For a robot cell, "production" is the physical unit, so these metrics expose whether an update can reach a second unit and be rolled back.
4. Operational and economic drivers in physical units. Installation hours, operator training hours, exception-handling minutes per shift, downtime hours, compute power draw in watts. The pilot supplies physical quantities and ranges; the sponsor's finance model converts them.

The ML Test Score by Breck and colleagues offers a scoring discipline worth borrowing. The authors "present 28 specific tests and monitoring needs" across four sections. "For each test, half a point is awarded for executing the test manually, with the results documented and distributed," and "a full point is awarded if there is a system in place to run that test automatically on a repeated basis." Crucially, "the final ML Test Score is computed by taking the minimum of the scores aggregated for each of the 4 sections."

The statistics of a short clock

A ninety-day pilot has a finite number of trials. That number bounds what the pilot can claim.

When no failures are observed, the rule of three applies. As Newman in JAMA summarizes Hanley and Lippman-Hand, "if 0 events are observed in N trials, the upper limit of the 95% confidence interval (CI) for the frequency of events is approximately 3/N." For independent trials, with C the confidence level, p the true failure rate and N the number of failure-free trials, the exact bound and its approximation are:

Inverting it gives the number of failure-free independent trials needed to claim a failure rate below a target p:

These are arithmetic consequences of the formula, not field results. The assumption to watch is independence: fifty picks in one shift, under one lighting condition, with one operator, may be correlated and provide limited coverage of the operating envelope; justify independence and the population to which the failure-rate bound applies. Correlated trials can overstate confidence.

The same logic scales up brutally. In Driving to Safety, Kalra and Paddock calculate that "fully autonomous vehicles would have to be driven hundreds of millions of miles and sometimes hundreds of billions" to demonstrate reliability for fatalities and injuries, and conclude that developers "cannot simply drive their way to safety." A factory envelope differs from public roads, and those vehicle figures do not transfer directly. Compare the available independent trial count with the required failure-rate bound; where trials are insufficient, the pilot needs additional evidence (safety architecture, fault injection, deterministic bounds) and an explicit statement of residual uncertainty.

Rollout mechanics: canaries for physical units

The Google SRE workbook defines canarying as "a partial and time-limited deployment of a change in a service and its evaluation", advises that a canary "should be sizeable and last long enough to be representative of the overall deployment," and notes that "canarying is an A/B comparison." In a Physical AI pilot the canary is one unit, cell or shift, compared against a control unit or the existing process over the same period. The SRE advice to run "only one canary deployment at a time" translates directly: one change per evaluation window, or the evidence cannot be attributed.

Human authority and operator trust

The operator is part of the product. Lee and See, reviewing trust in automation in Human Factors, write that automation can be problematic when "people fail to rely upon it appropriately", and that "trust guides reliance when complexity and unanticipated situations make a complete understanding of the automation impractical." The pilot must therefore observe whether operators over-trust uncertain outputs or override everything. Both are product failures invisible to model metrics.

Research evidence: what the sources support, and what they do not

The sources above are of different kinds and support different things.

The evidence base for Physical AI pilots specifically is thin. None of these sources measures how long Physical AI pilots take, what share of them reach production, or whether a fixed clock improves outcomes. The ninety-day default is a governance choice, not a research finding.

Business and decision analysis

Decision criteria

NASA's decision-analysis guidance gives three rules the clock adopts directly: "Criteria should be objective and measurable," "Criteria should also permit differentiating among options or alternatives," and "An option that does not meet mandatory criteria should be disregarded." It also states: "Uncertainty matters in a particular analysis only if there is a non-zero probability that uncertainty reduction could alter the ranking of alternatives." That sentence is the selection rule for which assumptions to test first.

The decision table

A partner is a way of executing a verdict, not a verdict of its own, so it is recorded inside the Go or Redirect recommendation with its terms. A stop decision can be a successful outcome of the clock. It prevents more spending on an unsupported thesis.

Cost drivers, without currency

The pilot should report cost drivers as physical quantities and ranges. The sponsor's finance function converts them.

Organizational ownership

The NIST AI RMF GOVERN 2.1 states that "roles and responsibilities and lines of communication related to mapping, measuring, and managing AI risks are documented," and MANAGE 2.4 calls for mechanisms, with responsibilities "assigned and understood," to "supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use." The clock maps this onto six named roles. One person may hold several, but the evidence owner should not be the person whose work is judged.

The method: the 90-day decision clock, week by week

Days 1–15 — Frame the decision (about weeks 1–2)

Start with the decision the sponsor actually needs to make. Examples include whether to authorize productization, narrow the operating envelope, change a supplier, collect different field evidence, or stop.

Write:

• the buyer and product outcome;
• the prototype behaviour already observed;
• the conditions under which it was observed;
• the unresolved customer, technical, safety, operational and economic assumptions;
• the decision options;
• the evidence that would distinguish those options.

Do not start by expanding the backlog. A long backlog can conceal an undefined decision.

Two more week-1 tasks keep the clock honest. First, run a premortem. Klein describes it in Harvard Business Review: "team members assume that the project they are planning has just failed," which makes it safe for knowledgeable dissenters to name weaknesses. Second, compute the trial budget (defined below) and decide whether ninety days can produce the reliability claim the product needs. If it cannot, change the claim now, not on day 85.

Exit criterion: a signed decision charter and evidence register. Signed means accountable ownership, not agreement with a predetermined answer.

Days 16–45 — Attack the decisive assumptions (about weeks 3–6)

Prioritize the assumptions that can reverse the recommendation. A product may depend on operator acceptance, a latency bound, a supplier interface, field data access, installation time or a safe degraded mode.

Design the smallest credible test for each one. Keep observed results separate from estimates. Record test conditions, failures and missing evidence.

An illustrative vision-product test might compare behaviour across the actual camera, lighting variation and line speed while observing how an operator handles an uncertain result. The purpose is not to produce a persuasive demo. It is to discover whether the product promise survives contact with the field.

Each assumption moves through a small set of states. The state is recorded in the evidence register, with its owner and the date of the last change.

stateDiagram-v2
    state "Open" as Open
    state "Test designed" as Designed
    state "Under test" as Testing
    state "Supported in stated conditions" as Supported
    state "Refuted" as Refuted
    state "Blocked by missing access" as Blocked
    state "Escalated to decision sponsor" as Escalated
    [*] --> Open
    Open --> Designed
    Designed --> Testing
    Testing --> Supported
    Testing --> Refuted
    Testing --> Blocked
    Blocked --> Testing
    Blocked --> Escalated
    Refuted --> Escalated
    Supported --> [*]
    Escalated --> [*]

A refuted decisive assumption goes to the sponsor immediately, not at day 90. That is the mechanism that lets the clock stop early.

Exit criterion: a versioned evidence pack with methods, observations, limitations and owners, and every decisive assumption in a terminal or escalated state.

Days 46–75 — Test the whole product candidate (about weeks 7–11)

A model test cannot prove a product. Connect the candidate to the surrounding system and workflow.

Review:

• sensors, compute, connectivity and hardware variation;
• deterministic controls, human authority and safe state;
• deployment, identity, observability and rollback;
• installation, training, acceptance and support;
• data lineage, access, retention and change approval;
• supplier responsibilities and exit;
• product economics as ranges.

Use a realistic field or bounded test environment. Label simulated evidence as simulated. Label an illustrative dossier as illustrative. Neither becomes customer production evidence through repetition.

Three drills matter more than additional trials: install the candidate on a second unit from documented steps; push one change through a canary unit and roll it back; and force the safe state from each fault the safety concept names. Each drill yields a measured quantity (engineer-hours, recovery time, safe-state entry time) for the decision table.

Exit criterion: a productization dossier and a list of residual risks with named owners. NASA's risk guidance applies here: "Risk status will need to be monitored periodically at a frequency identified in the risk plan" (NASA SE Handbook 6.4). A residual risk without an owner and a review frequency is not a residual risk; it is an unresolved assumption.

Days 76–90 — Make the decision (about weeks 12–13)

Reassemble the evidence around the original options. Do not score the pilot by activity completed or demo quality. Ask which option the evidence supports.

The recommendation should be one of:

• Go to the next Product System gate with acceptance criteria.
• Redirect the product boundary, operating envelope or value proposition, including moving a capability to a partner where ownership should sit elsewhere.
• Sequence behind one named item of unavailable evidence, with its owner and expected date.
• Stop because the opportunity or product case no longer justifies investment.

NASA's guidance asks for a report "which fully documents the decision needed, assessments done, recommendations, and decision finally made." The clock's version is short.

Exit criterion: a two-page decision memo, evidence appendix, residual-risk owners and next review date.

flowchart LR
    A["Days 1-15 Frame"] --> G1{"Charter signed and trial budget feasible"}
    G1 -- "yes" --> B["Days 16-45 Decisive assumptions"]
    G1 -- "no" --> R["Reframe claim or stop"]
    B --> G2{"Any decisive assumption refuted"}
    G2 -- "yes" --> S["Escalate to sponsor now"]
    G2 -- "no" --> C["Days 46-75 Whole product candidate"]
    C --> G3{"Drills done and risks owned"}
    G3 -- "no" --> P["Sequence: defer pending named evidence"]
    G3 -- "yes" --> D["Days 76-90 Decision review"]
    D --> M["Decision memo and next review date"]

The day-90 review

The review tests the recommendation against the charter; it is not a project presentation.

sequenceDiagram
    participant EO as Evidence owner
    participant PO as Product owner
    participant SL as Safety lead
    participant OP as Site operations
    participant DS as Decision sponsor
    EO->>DS: Evidence register against original charter
    PO->>DS: Recommended option and envelope
    SL->>DS: Safety concept status and open findings
    OP->>DS: Install, training and exception-handling measures
    DS->>EO: Challenge any estimate presented as observed
    DS->>PO: Decision and acceptance criteria for next gate
    DS->>SL: Residual safety risks with owners and dates

Two original devices for the clock

The following are Hyperion reasoning devices derived from the sources above. They are not field evidence, and no outcome claim attaches to them.

Device 1: the demonstrable reliability bound

Before any trial runs, compute the best reliability claim the clock could support if everything goes perfectly. Let r be the number of independent trials per field day the site can support, and d the number of field days available in the clock. Applying the rule of three to the total trial count gives the best-case bound, written B_best:

B_best is the lowest failure-rate bound, at 95% confidence, that zero failures in every available trial could justify. If the value case needs a lower failure rate than B_best, no execution of this pilot can prove it by trials alone. The honest options are to narrow the claim, extend the clock with a recorded reason, add evidence of a different kind (architecture, fault injection, deterministic bounds), or record the reliability claim as a residual risk with an owner.

Device 2: weakest-link gate readiness

Borrowing the minimum-of-sections rule from the ML Test Score, score each of the four product layers from the technical deep dive on the same scale (0 for untested, 0.5 for tested manually with documented results, 1 for tested repeatably), sum within each layer, normalize by the number of checks in that layer, and take the minimum:

Breck and colleagues chose the minimum on the stated ground that "all four sections are important." For a Physical AI pilot the argument is stronger: an excellent perception score does not compensate for a delivery layer that cannot roll back, because the failure of either stops the product. The decision memo reports all four layer scores and the minimum. A Go recommendation with a low minimum is a contradiction the sponsor should reject.

Worked hypothetical example

This example is hypothetical. It describes no real company, site or result.

A team pilots a camera-based surface-inspection station on one production line. In week 1 the site operations owner offers 20 field days of observed trials, and the evidence owner decides that, to keep trials reasonably independent, only 30 parts per field day will count, spread across shifts, lighting and part variants. Assume the 600 counted trials are independent inspections of known-defective parts drawn from the target defect population. With zero missed defects, the approximate 95% upper bound on the missed-defect probability, B_best, is about 3/600, or 0.5%.

The value case, written by the product owner, assumes a missed-defect rate below 0.1%. The clock cannot demonstrate that by trials. In week 2 the charter is amended: the pilot will demonstrate a bound near 0.5% in the stated envelope, test whether a second-stage human check catches the residual, and log operator minutes per shift spent on that check. The tighter rate becomes a residual risk owned by the product owner, with a post-pilot data-collection plan.

At day 90, the layer scores (on the 0 to 1 normalized scale) are: perception 0.8, control 0.6, delivery 0.3, product 0.7. The minimum is 0.3, driven by a rollback drill that took far longer than the change it reverted. The recommendation is Redirect: go ahead only within the narrower envelope, with the delivery layer as the named acceptance criterion for the next gate.

Implementation checklist

What invalidates the clock

The process fails when:

• the decision is rewritten after evidence arrives;
• a deadline is treated as permission to skip safety or field evidence;
• simulation is presented as field acceptance;
• an optimistic estimate is copied as an observed result;
• nobody can stop the work;
• the team hides an unresolved assumption inside a future phase;
• the output is automatically described as "production-ready."

The remedy is governance, not more velocity. Keep the decision charter stable, maintain the evidence register and make changes visible.

Adjusting the timeline honestly

Shorten the clock only when the decision is narrow and the evidence environment already exists. Extend it when evidence depends on seasonal operation, hardware lead time, regulated review, safety validation or access that cannot responsibly be compressed.

If the clock changes, record why, what evidence is blocked and who owns the next date. An extension without that record is drift.

Where it fits

The decision clock can be used at any Physical AI Product System gate. It is especially useful between Discovery and Productization, where technical feasibility must meet customer, field, safety, operating and economic evidence. Related: the prototype-to-product decision gate, the safety case evidence template and, on intervention metrics, Reliable Autonomy: From Robot Demonstration to Sustained Duty.

The Physical AI Product Decision Review uses the same principle: create one written, evidence-backed recommendation before another consequential product investment is authorized.

Limitations and open questions

• No outcome evidence for the clock itself. No source cited here measures whether a fixed ninety-day clock improves Physical AI pilot outcomes. The default is a governance choice.
• Independence is hard to establish. The trial-budget device assumes independent trials. On a real line, trials can share operators, lighting, wear and software versions. How to discount correlated trials is a judgement the evidence owner must document.
• Rare-event safety needs more than trial counts. The rule of three and Kalra and Paddock both show that low failure rates need very large trial counts. Where the available trials cannot support the required bound, safety acceptance must rest on additional architecture, analysis and conformity evidence, which this paper does not detail.
• Regulatory detail is summarized, not analysed. The Machinery Regulation statements come from the EU-OSHA summary; specific obligations require the Official Journal text.
• The four-layer scores are ordinal. The minimum is a decision aid, not a physical measurement; two teams can score the same evidence differently.
• Stop authority may be split. The six roles assume one sponsor who can stop the work. Where that authority spans organizations, consider on day 1 whether part of the work should be executed through a partner.

FAQ

Is ninety days long enough to reach production?

Not necessarily, and the clock does not claim it is. Ninety days is a default for reaching an evidence-backed decision. Production may follow later, or never, depending on what the evidence supports.

What if the evidence needs more than ninety days?

Extend the clock with a written reason, the blocked evidence and the owner of the next date. If one named item is missing, the correct verdict is Sequence, not an open-ended extension.

Can simulation results count toward the decision?

Yes, labelled as simulated. Research such as Tobin and colleagues shows simulation-trained perception can transfer under stated setups, but transfer to the actual hardware and site still has to be tested before the evidence supports a field claim.

How many trials are enough?

It depends on the failure rate the product case needs. With zero failures, N independent trials support a 95% upper bound of roughly 3/N. Compute this in week 1 and amend the claim if the clock cannot reach it.

Who should own the stop decision?

A named decision sponsor with the authority to end the work. The NIST AI RMF calls for documented roles and for mechanisms to supersede, disengage or deactivate systems inconsistent with intended use; the clock makes that authority explicit before any evidence arrives.