Topic
Factorybench
Category
Topics
2 issues matching filters
Factorybench
- Field NotesJul 13, 2026
Wiring a language model to the alert pipeline: the work order it can draft, and the diagnosis it still cannot
Issue 11 read the AI-in-manufacturing pitch from the floor and left one bounded test open: put a small local model where the pitch insists it belongs, on the alert the stack already produces, and measure what it adds. This issue runs that test on paper. When a gated flag fires, a small model turns the raw alert and the machine's manual into a serviceable first-line work order, the document task the FactoryBench results say it can actually do. Ask the same model in the same loop to name the fault from the signal and it walks straight into the 17.7 percent task it cannot. The value is real and the boundary is exact, and the design that earns the model its place is the one that never lets it cross the line.
Issue 11 set the franchise beat, the intersection of AI and manufacturing read from the floor, and closed by naming the first test: take the alert the reference condition-monitoring stack already produces and wire a small local language model to it, then model what the model adds rather than what a vendor demo claims. This issue runs that test. The finding is that the loop contains two tasks the agentic pitch keeps fused and that the model handles them at opposite ends of its competence. The first task is document retrieval. When a gated anomaly flag fires, the model receives the alert, the affected asset identifier, and the relevant section of the maintenance manual, and it drafts a first-line work order, naming the asset, quoting the applicable procedure, listing the parts, the tools, and the lockout and tagout steps, and leaving the priority and the assignment to a human. Published work on retrieval-augmented generation over maintenance and equipment documentation reports that small models do this reliably, because the task is retrieving and reformatting text the manual already contains rather than reasoning about the machine. The second task is diagnosis, deciding from the vibration signature what is actually wrong, and that is the exact task FactoryBench measured the best current models at 17.7 percent on, because reading an industrial time-series signal and retrieving a protocol document are separate competencies these models conflate. The same small model in the same alert loop can draft the paperwork and cannot name the cause, and the danger is that a confidently phrased wrong cause is indistinguishable in tone from a right one, so a model allowed to guess the fault will file plausible fiction into the maintenance record. The issue models the economics from the same edge-inference benchmarks issue 11 cited, a one-to-three-billion-parameter model returning single-digit to low-double-digit tokens a second on an owned edge box, which is fast enough to draft a work order in the seconds after a flag and far too slow to reason in a real-time loop, at a hardware cost that amortizes to a few dollars a month and adds nothing to the per-query bill because there is no per-query bill. The design conclusion is a single rule: the model may quote the manual and may never infer the cause, and the pipeline enforces the rule structurally by giving the model the alert and the document and withholding the raw signal it would need to hallucinate a diagnosis from. The numbers in this issue are modeled from published third-party benchmarks and vendor specifications, cited inline.
Retrieval Augmented Generation·Small Language Models·Work Orders·Cmms·Factorybench·Edge Inference - Deep DiveJul 6, 2026
The agentic-AI factory pitch meets the floor: the 95 percent that never reach the P&L, and the benchmark the best models fail
2026 is the year every industrial vendor sells agentic AI to manufacturing, and the adoption curve under the pitch is real. The trouble is the three numbers the pitch does not put on the slide: the MIT finding that 95 percent of generative-AI pilots return nothing measurable to the income statement, the benchmark where the best current model scores 17.7 percent at diagnosing a machine from its own sensor data, and the pilot that costs ten thousand dollars a month and five hundred thousand at production scale. This issue reads the wave from the floor rather than the keynote, and finds that the AI which actually pays is small, local, and narrowly scoped, with one asterisk the small-model camp does not advertise either: the eight-millisecond edge demo is a single-pass detection number, not a reasoning agent's, and conflating the two is the same trick one altitude down.
The series changes altitude and sets its beat for the issues ahead: the intersection of AI and manufacturing, read from the floor by a publication that has built the small version and can therefore read the large pitch. The first ten issues built a reference condition-monitoring stack, an Isolation Forest scoring vibration on one inexpensive VM, and that build is the credential for the question this issue turns to. The pitch is stated at full strength first, because the honest broker steelmans before it counters. Adoption is genuine: about a third of manufacturing operations are AI-augmented today by Rockwell's 2026 survey, Deloitte's 2026 Manufacturing Industry Outlook expects agentic adoption to roughly quadruple from about 6 to 24 percent even as only one in five reports being equipped to scale it, and the vendors at Hannover Messe and CES 2026 are selling agentic copilots, industrial foundation models trained on CAD and sensor data, and physical-AI robot cells. Then three numbers are set against it. MIT's NANDA study found that for 95 percent of companies, enterprise generative-AI pilots showed little to no measurable P&L return, and Gartner projects 60 percent of AI projects abandoned through 2026 for lack of the data foundation the first ten issues were spent building. FactoryBench, published in 2026, is the first benchmark to test whether a model can read industrial machine time-series rather than retrieve a manual, and the best current models, Claude Sonnet 4.6 and GPT-5.1, scored below 50 percent on structured reasoning and 17.7 percent on root-cause decision-making, direct evidence that the foundation models do not yet understand a machine from its signal. And pilot economics invert, a ten-thousand-dollar pilot becoming five hundred thousand at fifty times scale. The constructive turn is the small, task-specific model run locally, which Dell and Gartner are calling the real 2026 story, and which is the shape of the stack this series already built. But the issue adds the asterisk the small-model camp leaves off: the eight-millisecond, two-thousand-dollar edge figure describes a single-pass detection workload, not a generative reasoning agent, and on published edge-inference benchmarks a one-to-three-billion-parameter reasoning model returns single-digit to low-double-digit tokens a second, fine for drafting an offline work order and far short of a real-time agent. The honest synthesis is to match the model class to the task and to refuse both camps when they swap one spec sheet for the other. The numbers in this issue are modeled from published benchmarks and vendor specifications, cited inline.
Agentic Ai·Industrial Ai·Small Language Models·Foundation Models·Physical Ai·Factorybench