Skip to content

Industrial model evaluations · AI-run desk study · 19 November 2026

When a valid alarm label misses the alarm

Four open-weight models, each run from an 8-bit GGUF file, were given the same maintenance, alarm and routing tasks on a cloud CPU guest using four threads. Three of them also classified inspection photographs. On twelve simulated alarm histories, Olala v1.0 mini and Ministral 3 3B each answered CLEAR every time: both scored 6 out of 12 and missed all six active alarms. This record shows the counts, how they were produced and what they support.

All industrial model evaluations

Models and files tested

Every local result on this page comes from these files, pinned by repository revision and SHA-256 in the results file. Table columns use the short names. Links open the file at the revision the study used. Qwen3.5-9B was run from the GGUF in the unsloth repository; the weights it was converted from are in Qwen/Qwen3.5-9B. The hosted trials did not expose their weights or quantisation.

A score of 6 out of 12 that missed every active alarm

Each of the twelve alarm cases gives a short history of threshold crossings and resets, then asks whether the latched alarm is ACTIVE or CLEAR. Six cases require ACTIVE and six require CLEAR. With the output constrained to those two labels, Olala v1.0 mini and Ministral 3 3B answered CLEAR twelve times. Each scored 6 out of 12 and labelled all six active alarms CLEAR.

Ministral 3 8B answered ACTIVE on eleven of twelve cases. It labelled all six active alarms correctly and only one of the six clear cases. Qwen3.5-9B labelled four of the six active alarms and five of the six clear cases correctly.

Every constrained reply contained an allowed label, and each model still made at least one state error. An acceptance test for a safety-relevant state has to count the two kinds of error separately, and a latched state is better computed with explicit rules than asked of a model.

Alarm outcomes by actual state, constrained output

  • Correct label
  • Active alarm labelled CLEAR
  • Clear state labelled ACTIVE
ModelAlarm was active (6 cases)Alarm was clear (6 cases)
Olala v1.0 miniQ8_0-safe26 of 6 labelled CLEAR0 of 6 labelled ACTIVE
Ministral 3 3BQ8_06 of 6 labelled CLEAR0 of 6 labelled ACTIVE
Ministral 3 8BQ8_00 of 6 labelled CLEAR5 of 6 labelled ACTIVE
Qwen3.5-9BQ8_02 of 6 labelled CLEAR1 of 6 labelled ACTIVE
Each cell summarises six saved decisions on simulated alarm histories, grouped by outcome. Mark positions do not identify individual cases. Constrained-output profile, 32-token limit.

Which model to trial for which job

Trial means a comparison on your own cases. Each row gives the result observed here and the check it motivates. No row is a statement about how a model will perform on your plant. Before shortlisting any model, check the requirements that bind your deployment: origin, licence, offline operation, support and data path.

  • Answer questions about sensors and faults

    Next trial or engineering checkInclude Qwen3.5-9B in a read-only comparison. Give the technician the asset evidence before a diagnostic decision.

    What was observed, and what to testQwen3.5-9B answered 52 of 100 questions correctly, Ministral 3 8B 45, Olala v1.0 mini 41 and Ministral 3 3B 35. Always answering A scores 32. The questions run both ways, from a fault to its sensor and from a sensor reading to the fault. Compare by asset, rotate the options and keep back site cases the model cannot have seen.

  • Apply a maintenance bulletin you supply

    Next trial or engineering checkTrial Qwen3.5-9B with the current revision identified in the prompt. Evaluate finding the right revision separately.

    What was observed, and what to testQwen3.5-9B returned the required disposition on 11 of 12 cases in both profiles, including all three HOLD cases. Its one error held a case that should pass. Retrieval was not measured. Test superseded documents, missing readings and citation of the revision used.

  • Request one read-only lookup

    Next trial or engineering checkInclude Ministral 3 3B, Ministral 3 8B and Qwen3.5-9B in a single-lookup trial. Measure deadline misses on your device and reject extra calls before dispatch.

    What was observed, and what to testAll three requested the right function with the right identifier in 8 of 8 cases, at median 2.6, 5.7 and 10.1 seconds on this guest. Olala v1.0 mini passed 2 of 8 and requested two calls in each of its six failures. No function was executed. Test wrong identifiers and ambiguous requests.

  • Explain a latched alarm state

    Next trial or engineering checkCompute the state with the approved latch and reset rules. If an assistant explains the result, test its explanation separately.

    What was observed, and what to testWith constrained output, Olala v1.0 mini and Ministral 3 3B each labelled 6 of 6 active alarms CLEAR. Ministral 3 8B flagged 5 of 6 clear cases. Qwen3.5-9B labelled 2 of 6 active alarms CLEAR and flagged 1 of 6 clear cases. Interlock operation and restart permission were not measured.

  • Triage photographs of parts

    Next trial or engineering checkInclude Qwen3.5-9B in a photo-triage trial on your own parts. Keep the release decision with the inspector.

    What was observed, and what to testOf 16 photographs of defective parts, Qwen3.5-9B labelled 3 NORMAL, Ministral 3 8B 7 and Ministral 3 3B 8. Of 16 photographs of normal parts, they labelled 1, 5 and 4 DEFECT. All three of Qwen3.5-9B's misses were among the four defective photographs of one circuit-board product. The study ran Olala v1.0 mini on text only. On your line, label the defect types and measure misses and unnecessary rejects at the production mix.

  • Check a coordinate transform or a warehouse route at a desk

    Next trial or engineering checkUse geometry code for coordinates and a planner to validate routes. The long Olala v1.0 mini profile is a candidate for a desk draft if minutes per request are acceptable.

    What was observed, and what to testWith medium reasoning and a 12,000-token allowance, Olala v1.0 mini passed 10 of 12 transforms and 10 of 12 routes, at median 216 and 102 seconds per request. With a 256-token allowance all four models scored 0 of 12 on both tasks. The other three models were not run with the long allowance, so a choice among the four at that allowance remains open. Robot motion was not measured.

  • Compute production quantities

    Next trial or engineering checkCalculate the value in code. Check the inputs and units before using model-written explanatory text.

    What was observed, and what to testQwen3.5-9B passed the same three carriage-distance cases in both profiles and missed the other nine calculations. Each other model returned 0 of 12 in both profiles.

  • Choose a model for an offline CPU box

    Next trial or engineering checkInclude Olala v1.0 mini in an on-device timing trial if it meets the task's accuracy requirement. Measure time to an accepted answer, peak memory and disconnected operation on the intended box.

    What was observed, and what to testGeneration on this four-thread guest ran at 17.0, 8.9, 3.8 and 3.6 tokens per second for Olala v1.0 mini, Ministral 3 3B, Ministral 3 8B and Qwen3.5-9B. Tokenizers differ between models. Device fit, energy and operation without a network were not measured.

  • Translate technical notes

    Next trial or engineering checkChoose candidates through a blinded review in your language pair. Include terms where a mistranslation changes the required action.

    What was observed, and what to testA single AI grader marked eight translations per model, between English and French, German or Spanish. That supports no choice of model. In one case the grader failed a translation that turned a measured deviation into an out-of-tolerance finding. Test identifiers, units, negation and conditional instructions.

  • Draft a bilingual shift handover or extract a maintenance record

    Next trial or engineering checkEvaluate this workflow on paired shift notes with a bilingual reviewer, and check each extracted field against the source. The choice of model remains open.

    What was observed, and what to testAll eight handover sources were in English and asked for a short summary. Bilingual handover and structured extraction were not measured.

Outside this study: live actuation, forecasting from sensor waveforms and any decision about the sovereignty of a deployment. No recommendation follows for these.

Results by protocol

Counts are strict passes over attempts. The scorer checked the submitted answer, so a correct label inside explanatory prose could fail. Where replies reached the output limit, the cell says how many. A cut-off reply is a result about that serving profile and not a measure of what the model can reason.

Public multiple-choice questions, one output token

TaskOlala v1.0 miniQ8_0-safe2Ministral 3 3BQ8_0Ministral 3 8BQ8_0Qwen3.5-9BQ8_0
Sensors and faultsalways A scores 32/10041/10035/10045/10052/100
Return to originalways No scores 145/250126/250105/250105/250165/250

All four models received the same raw prompts in one option order. A grammar allowed one option letter for the sensor and fault questions, and Yes or No for return to origin.

On the navigation questions, always answering No scores 145 of 250. Both Ministral models answered Yes every time and scored 105. Olala v1.0 mini scored 126. Qwen3.5-9B scored 165, yet recognised only 32 of the 105 cases that do return to the origin.

On the sensor questions, 18 questions favoured Qwen3.5-9B alone over Olala v1.0 mini and 7 favoured Olala v1.0 mini alone. The difference of 11 points has an approximate paired 95% interval of 1.3 to 20.7 points for this sample and protocol. It describes sampling uncertainty across the balanced question groups, using paired answers on the same items. Olala v1.0 mini against always answering A was 9 points, with an interval of −7.0 to 25.0. Neither interval measures transfer to a plant.

Local chat tasks, up to 256 output tokens

TaskOlala v1.0 miniQ8_0-safe2Ministral 3 3BQ8_0Ministral 3 8BQ8_0Qwen3.5-9BQ8_0
Latched alarm0/1212 cut off4/120/121 cut off9/12
Maintenance bulletin decision3/120/120/1211/12
Production arithmetic0/125 cut off0/120/123/12
Read-only tool call2/88/88/88/8
Coordinate transforms0/1212 cut off0/129 cut off0/125 cut off0/12
Shortest legal route0/1212 cut off0/120/120/12

Each model's own chat template, temperature zero, seed 42. Reasoning was switched off for Olala v1.0 mini and Qwen3.5-9B in this profile.

As a secondary check, the scorer read the final label of each completed reply. On the alarm cases it found the correct label in 10 of 12 Ministral 3 8B replies, where the strict check accepted 0 of 12.

Constrained output, 32-token limit

TaskOlala v1.0 miniQ8_0-safe2Ministral 3 3BQ8_0Ministral 3 8BQ8_0Qwen3.5-9BQ8_0
Latched alarm6/126/127/129/12
Maintenance bulletin decision8/125/127/1211/12
Production arithmetic0/120/120/123/12
Coordinate transforms0/121/120/120/12
Shortest legal route1/120/120/120/12

A grammar restricts the reply to the answer type, never to the correct answer. This profile also changes the request path and the output allowance.

What happened to each of Olala v1.0 mini's spatial requests

  • Accepted
  • Completed, wrong
  • Output cap reached
ProfileCoordinate transformsShortest legal route
Local guest · direct answer · 256-token cap0 accepted, 0 wrong, 12 at cap · median 16.3 s0 accepted, 0 wrong, 12 at cap · median 16.2 s
Local guest · medium reasoning · 12,000-token cap10 accepted, 2 wrong, 0 at cap · median 215.7 s10 accepted, 0 wrong, 2 at cap · median 101.5 s
Hosted Olala-v1.0-mini-7A1B alias · provider reasoning defaults · 2,048-token cap8 accepted, 0 wrong, 4 at cap · median 5.5 s9 accepted, 0 wrong, 3 at cap · median 5.1 s
Twelve coordinate transforms and twelve routes per profile, Olala v1.0 mini only. The two local profiles use the same weights and cases; reasoning mode, context and output cap change together. The hosted service runs a different serving stack on different hardware and did not expose its exact weights or quantisation. Its timings include the service request.

Inspection errors by product

ProductMinistral 3 3BQ8_0Ministral 3 8BQ8_0Qwen3.5-9BQ8_0
Defective, labelled NORMALNormal, labelled DEFECTDefective, labelled NORMALNormal, labelled DEFECTDefective, labelled NORMALNormal, labelled DEFECT
Candle0 of 44 of 40 of 43 of 40 of 40 of 4
Capsules2 of 40 of 42 of 40 of 40 of 40 of 4
Circuit board 14 of 40 of 44 of 40 of 43 of 40 of 4
Circuit board 42 of 40 of 41 of 42 of 40 of 41 of 4
Total, four products8 of 164 of 167 of 165 of 163 of 161 of 16
32 photographs from the public VisA dataset: four of defective parts and four of normal parts for each of four products. The unit is a photograph; individual defects and defect types were not scored. Qwen3.5-9B's image backend logged token-position warnings while processing images and returned a label on all 32 requests. The three models ran as Q8_0 files with BF16 vision projectors. The study ran Olala v1.0 mini on text only, so it is not shown.

Speed on a four-thread cloud guest

MeasureOlala v1.0 miniQ8_0-safe2Ministral 3 3BQ8_0Ministral 3 8BQ8_0Qwen3.5-9BQ8_0
512-token prompt, tokens/s105.146.223.018.1
128-token generation, tokens/s17.08.93.83.6
Sensor question, median s1.11.62.55.3
Tool request, median s5.02.65.710.1

Mean of three warm repetitions per workload on the 8-bit files listed above, no GPU. Task latency is the median over all requests, including failed ones, with prompt caching on.

A stable total can hide different answers

On twenty hosted sensor questions with a 12,000-token allowance, the hosted Olala-v1.0-mini-7A1B alias scored 8 of 20 in the original option order and 8 of 20 with the options rotated, but was right in both orders on only 4. The hosted Qwen3.5-9B endpoint scored 5 and 4, and was right in both orders on 3. The two orders reused the same 20 questions. Neither hosted endpoint exposed its weights or quantisation. Olala v1.0 mini reached the output cap on 1 of 40 requests and Qwen3.5-9B on 5 of 40. The Ministral models were not part of this replay. Rotating the options belongs in any multiple-choice acceptance test.

Document tasks, graded by the evaluating AI

The evaluating assistant also graded eight translations and eight English handover summaries per model against their synthetic source notes. Each grade is one AI judgment, and review by a bilingual reader and a plant engineer is pending. In the direct profile the tallies were 5, 6, 8 and 6 of 8 for translation and 3, 1, 5 and 7 of 8 for handover, for Olala v1.0 mini, Ministral 3 3B, Ministral 3 8B and Qwen3.5-9B. With the same 1,536-token limit and its default reasoning setting, Olala v1.0 mini received 7 of 8 for translation and 7 of 8 for handover. They are reported for completeness and support no choice of model.

How it was done

  1. Inputs frozen first

    100 sensor and fault questions from IBM Research's FailureSensorIQ (five positive and five negative per asset, ten assets), all 250 Navigate questions from BIG-Bench Hard, 68 synthetic factory and robotics cases, 16 synthetic documents and 32 photographs from the VisA inspection dataset. Each case has an identifier and an expected answer held by the scorer.

  2. Models pinned

    Olala v1.0 mini 7A1B (final) as OVH's mixed 8-bit GGUF file, Ministral 3 3B Instruct 2512 and Ministral 3 8B Instruct 2512 as Mistral AI's 8-bit GGUF files, and Qwen3.5-9B as Unsloth's 8-bit conversion of the published weights. Each file is identified by its SHA-256. The three image models used their matching vision projector files. The study lists Apache 2.0 for the four model releases. Check the exact files, service terms and data path for your deployment.

  3. One runtime, several profiles

    Local runs used OVH's fork of llama.cpp at revision 55c22dca, four CPU threads and no GPU, on a cloud guest reported as an AMD EPYC virtual machine with a four-vCPU quota and 16 GiB of memory. Short text tasks used a 2,048-token context, temperature zero and seed 42. The public questions allowed one output token, the chat tasks 256 and the constrained tasks 32. Document replies had a 1,536-token cap. Image classification used a 4,096-token context, a 12-token cap and a two-label grammar.

  4. Exceptions recorded

    Qwen3.5-9B's prompt-cache budget was cut from 8,192 to 512 MiB after the guest began evicting model pages during the navigation questions. The published navigation counts come from the complete replay, and the 40 answers saved before the change all agreed with it. Only Olala v1.0 mini received the long local spatial replay, with medium reasoning, a 32,768-token context and a 12,000-token cap. A separate hosted sensor replay gave Olala v1.0 mini and Qwen3.5-9B a 12,000-token cap under their endpoint defaults.

  5. Scored by program

    Routes are checked move by move and any shortest legal route is accepted. A tool request passes only with exactly one call to the approved read-only function and the right identifier. Each tally counts one saved answer per case, model and profile. Expected answers for the 68 synthetic cases were recomputed by a second solver.

  6. Checked afterwards

    A second assistant recounted the headline scores from the saved per-case records and matched the 149 files in the evidence archive to their recorded hashes. It did not rerun inference or re-grade the documents.

OVH's own published comparison uses the Ministral Reasoning variants. This study ran the Ministral Instruct variants, on different tasks. Neither set of results overturns the other.

Download aggregate results (JSON)

The file holds the score tables with their protocol labels and cut-off counts, the runtime revision, references to the model and projector files, and the dataset links. Per-case answers, prompts and photographs are not included. The questions and photographs stay with their publishers.

Why these tasks matter for Physical and Industrial AI

Each task stands in for a decision someone takes on a plant or around a machine. The desk version removes most of what makes the real decision hard, so each row also names the gap.

  • Sensor and fault questions

    Decision it stands in forWhich signal a technician checks first for a suspected failure

    What the desk version leaves outThe asset's history, the site's instrumentation and the cost of a wrong first check

  • Latched alarm

    Decision it stands in forExplaining the final alarm state to a technician

    What the desk version leaves outReal signal timing, missing samples, and the site's approved interlock and restart procedure

  • Maintenance bulletin decision

    Decision it stands in forWhether a lot or an asset is held under the current revision

    What the desk version leaves outFinding the right revision among superseded documents

  • Read-only tool call

    Decision it stands in forAn assistant fetching a work order or an asset record

    What the desk version leaves outExecution, permissions and what happens after a wrong call

  • Photo inspection

    Decision it stands in forAccept or reject at an inspection station

    What the desk version leaves outYour parts, your lighting, your defect types and the cost of an escape

  • Transforms and routes

    Decision it stands in forChecking a robot frame or a vehicle route before it is used

    What the desk version leaves outObstacles that move, units and frame conventions, and motion itself

  • CPU throughput

    Decision it stands in forWhether a model fits an offline box at the edge

    What the desk version leaves outThe actual device, sustained load, memory and energy

What the study teaches about acceptance tests

  1. Count each kind of error separately. A total of 6 out of 12 hid six missed alarms.
  2. Compare with the trivial answer. Three of four models scored below always answering No on navigation.
  3. Record the serving profile with the result. The same Olala v1.0 mini weights went from 0 of 24 to 20 of 24 spatial answers when the reasoning mode, context and allowance changed.
  4. Report task latency beside task success. The fastest generator was not the fastest at completing a tool request.
  5. Rotate the options and split by product. Stable totals concealed different answers, and all three of Qwen3.5-9B's missed defects fell in one product group of four photographs.
  6. Use the exact model variant. A vendor's score on one variant says little about another variant on your task.

Limits

  • The factory and robotics cases are synthetic, eight to twelve per task, and reuse templates. They are not a representative sample of plant operations.
  • No robot was moved, no real-time deadline was enforced and no raw telemetry was forecast.
  • The hardware, the hosted endpoints and the operator are not independently attested.
  • Reasoning was switched off for Olala v1.0 mini and Qwen3.5-9B on the short tasks. Only Olala v1.0 mini received the long local spatial replay, and only Olala v1.0 mini and Qwen3.5-9B the hosted sensor replay.
  • Document grades come from the same AI that ran the study.
  • Thirty-two balanced photographs cannot price escapes or false rejects on a production line.
  • Speed was measured on one virtual machine and does not size an edge device.

Revisions

  • 19 November 2026: Study run and saved results reviewed.

Qualify one workload on your own hardware

A desk study narrows the candidates. The decision needs your cases, your device and your acceptance limits. Hyperion scopes that as a Product Decision Review or inside a Product Leadership Mission, with fees by quotation.

Discuss a qualification