Direct answer. If you lead an industrial product and need to adapt a language model to engineering documentation, classify the failure first. In my analysis, a missing or outdated fact points toward a retrieval or search intervention. A wrong format, terminology habit or workflow behaviour is a candidate for parameter-efficient fine-tuning. A failure where nobody can define correct output, or where an error is hazardous, calls for neither: narrow the scope, use deterministic logic or add a human gate. I recommend funding the cheapest intervention that passes a held-out test. Treat any provider's training platform as an option to evaluate, not a default.
Method and boundary. This piece analyses two supplied public sources: a vendor product page and library documentation. It contains no owned R&D, no client work and no benchmark results. The decision table and acceptance criteria are my proposed method, not vendor claims and not measured outcomes. The supplied sources do not evaluate retrieval, so every retrieval statement below is analysis for you to test.
Separate the knowledge problem from the behaviour problem
This distinction is my own framing. Ask what the model got wrong.
- It lacked a fact, such as a current specification or a revised work instruction. My proposal: test search or governed retrieval over a versioned document set first, and measure whether answers match the current revision.
- It had the fact but behaved wrongly, such as a wrong output structure, misused vocabulary or ignored procedure order. Adapting model weights becomes a candidate once correct context has been supplied and the error persists.
- No one can say what correct looks like. Neither approach helps. Define acceptance first.
What the sources say about adaptation
The Hugging Face documentation describes PEFT as a library for adapting large pretrained models without fine-tuning all of a model's parameters, because that is prohibitively costly PEFT. It says PEFT methods fine-tune only a small number of extra parameters, which decreases computational and storage costs PEFT. It also says this yields performance comparable to a fully fine-tuned model PEFT. That is a documentation claim, not a result on your task. Require your own comparison on your held-out set.
Mistral describes Forge as training models across the full lifecycle, from pre-training and synthetic data generation to post-training with reinforcement learning Build AI models that know your enterprise.. Its page lists LoRA to specialize models without heavyweight retraining Build AI models that know your enterprise.. It names industrial domain adaptation, training on engineering documentation, standards, vocabularies and decision frameworks, as a use case Build AI models that know your enterprise.. These are vendor descriptions. The extract gives no benchmark, licence terms or data-retention scope. Obtain those in writing before treating Forge as a candidate.
The page also says inference can run across private cloud, on-prem or Mistral compute with control over data residency and infrastructure Build AI models that know your enterprise.. Deployment location, data control, model licence and capability are separate questions. This page does not settle the licence question.
Complexity order (my proposal)
Full-lifecycle training sits far up the complexity ladder. For most industrial documentation assistants I would test in this order: deterministic logic or search, an evaluated existing model, governed retrieval, lightweight adaptation. Pre-training is exceptional.
Decision table (proposed method, not vendor evidence)
| Failure observed | Simplest intervention | Held-out evaluation | Data / control requirements | Rejection condition |
|---|---|---|---|---|
| Answers cite outdated or missing facts | Search or governed retrieval over versioned documents | Questions whose answers changed between document revisions; score source correctness | Document ownership, revision control, access rights | Wrong-revision answers above your agreed threshold |
| Correct facts supplied, wrong format or terminology | Prompt and output-schema changes on an existing model | Fixed task set scored against a format checker | Style guide, labelled examples | No gain over the baseline prompt |
| Persistent behaviour gap after prompt and retrieval fixes | Parameter-efficient fine-tune (for example LoRA via PEFT) | Held-out cases never used in training, plus a regression suite on prior capability | Curated labelled pairs, training-data rights, versioned adapters, rollback | Gain within noise, or regression on prior-capability cases |
| Rare edge cases dominate risk | Rules, interlocks or human review, not a model | Edge-case replay with expert sign-off | Defined hazard cases, named reviewer | Any unreviewed safety-relevant output |
| Requirement or ground truth undefined | Neither: stop and specify | Not testable yet | Product owner decision | Cannot write a pass/fail case |
Acceptance and stop criteria (recommendations for your team)
- Baseline first. Record the existing model's score on the held-out set before changing anything.
- Freeze the test set. No training or retrieval tuning on it.
- Escalate one rung at a time. Move from retrieval to fine-tuning only when the failure log shows a behaviour error after correct context was supplied.
- Stop when improvement does not exceed run-to-run variation, when training data rights are unclear, or when you cannot roll back to a known-good version.
- Require evidence from any supplier on licence, retention, residency and evaluation, since the supplied extracts do not provide it.
If model output could influence machinery safety or another regulated function, take the question to your safety and compliance owners and check the current primary regulatory text. This piece does not source or interpret any regulation.
Trade-offs
Retrieval adds an index to maintain and a failure mode in search quality. Fine-tuning adds dataset curation, versioning and regression burden. Full-lifecycle platforms add vendor dependency on top. Choosing neither costs the least and is often correct when the requirement is vague. This recommendation does not apply if you already hold a labelled behaviour-gap dataset and a frozen test set showing prompting and retrieval fail. Evidence like that would justify moving to adaptation sooner.
Next step
If you face a live choice between retrieval, adaptation or neither, a Product Decision Review is the fitting engagement for one consequential choice. Bring your failure log, a draft held-out set, data-rights status and your deployment constraints to a fit conversation at Hyperion Consulting.
