Imaging and Fixturing Decide the Project Before Any Model Exists
Most quality-control vision projects that fail were lost before a single image was labelled. The defect was not reliably visible in the picture, and no amount of model architecture recovers information the sensor never captured. This is the least glamorous part of the work and the part that decides the outcome. A scratch on a brushed metal surface is invisible under diffuse illumination and obvious under darkfield lighting at a grazing angle. A dent in a stamped panel disappears in a flat image and appears immediately under a deflectometry setup or a structured light pattern. The lighting choice is not a preference; it is the measurement itself, and it should be made with real defective samples in hand, before anyone writes a training script.
Resolution follows from the smallest defect you must catch, and the arithmetic is simple enough to do on a whiteboard. As a working rule we want three to five pixels across the smallest feature that has to be detected. If the specification says a 0.3 mm crack must be caught, that is roughly 10 to 17 pixels per millimetre at the part surface, which fixes the sensor and lens pairing for a given field of view. A 100 mm wide field at 15 px/mm needs about 1,500 pixels across, so a 2 to 5 megapixel sensor is usually adequate. Asking the same camera to cover 400 mm and still resolve 0.3 mm is asking for a project that fails quietly at acceptance.
Depth of field and motion blur are the two constraints teams discover late. Stopping the lens down to gain depth of field costs light, which costs exposure time, which on a moving line costs sharpness. A part travelling at 0.5 m/s smears 0.5 mm during a 1 ms exposure, which is larger than the defect you were trying to see, so either the exposure comes down and the illumination goes up, or the part is stopped. Triggering has to be deterministic as well: a photoelectric sensor or an encoder pulse, never a software timer, because 20 ms of jitter in trigger timing is enough to move the feature of interest out of the field of view entirely.
Fixturing is what makes all of the above repeatable. If the part can rotate 15 degrees or sit 8 mm closer to the lens between cycles, your training set is modelling part pose rather than defects, and you will need several times more data to compensate for it. Mechanical repeatability of a millimetre or two, a consistent background, and a shroud that excludes ambient and seasonal daylight are worth more than any change of model architecture. Our advice here is blunt: spend the first two to three weeks of a vision project on imaging trials with real defective parts, and be prepared to conclude that this defect cannot be imaged at an acceptable cost.
How Much Labelled Data You Actually Need
The question every buyer asks first is how many images are needed, and the honest answer is that the number is set by defect variability rather than by a rule of thumb. A defect that always appears in the same place, at the same scale, with the same appearance — a missing rivet on a fixtured assembly — is a nearly deterministic pattern and can be learned from a few hundred examples. A cosmetic defect that varies in size, contrast, orientation and background texture across three material suppliers is effectively several different problems wearing one label, and each variant needs its own coverage. Quoting a single number before seeing the parts is how integrators end up renegotiating scope in month three.
The ranges we typically see are still worth stating, with that caveat firmly attached. Low-variability defects on a well-fixtured station: roughly 150 to 500 labelled examples per class, plus a few thousand images of good product. Moderate variability, such as scratches on a single finish: 800 to 2,000 per class. High-variability cosmetic defects spanning suppliers, lots and finishes: 3,000 to 10,000, and at that point the project is often better served by an anomaly model than by a classifier. Good product is cheap to collect and you should gather far more of it than feels necessary, because the good-product distribution is what actually defines the boundary the model learns.
Collection strategy matters more than raw volume. A thousand images from one shift on one machine is a narrower dataset than three hundred images sampled across four weeks, two shifts, three material lots and both cameras, because the second set contains the variation the model will genuinely meet. Keep the defective parts physically; you will need them for acceptance testing and again after every retrain. And record metadata alongside each image — station, lot, shift, timestamp, operator disposition — because that is what lets you diagnose a failure six months later instead of guessing. The wider discipline of getting a dataset fit for modelling is covered in data readiness.
The Rare-Defect Problem and Why Accuracy Is Meaningless
On a mature production line the true defect rate is often between 0.1% and 2%, and on some processes it is measured in parts per million. That single fact invalidates the metric most people reach for first. If one part in 500 is defective, a model that predicts OK for every single part is 99.8% accurate and completely useless. Accuracy is dominated by the majority class and says nothing about the only decision that matters. We have sat in review meetings where a supplier presented 99.6% accuracy on a line with a 0.5% defect rate, which is consistent with a system that caught nothing at all, and nobody in the room had done the arithmetic before the meeting.
The metrics that carry information are recall at a fixed precision, and precision at a fixed recall. State the operating point explicitly: at 95% recall on cracks, precision is 62%, meaning we reject about 1.6 good parts for every genuine defect caught. That sentence is auditable and a quality manager can act on it. Report the confusion matrix in absolute counts per shift rather than percentages, because four escapes per week and thirty over-rejections per shift is a conversation a plant can have, while an F1 score of 0.74 is not. Report the counts the numbers rest on as well: a recall figure computed from eleven defective samples is noise wearing a decimal point.
Validation set design is where rare-defect projects quietly go wrong. Randomly splitting a dataset that contains multiple images of the same physical part leaks the answer across the split, and the reported score becomes fantasy. Split by part, by lot and ideally by time: train on the first six weeks, validate on the next two. Keep a held-out set of physical defective parts that never enters training at all, and run them through the deployed system at acceptance and after every retrain. If your defect count is in the low tens, use cross-validation, report a confidence interval, and be honest with the customer that the interval is wide and will stay wide for months.
False Positives Versus Escapes: Setting the Threshold From Cost
Every vision system has one knob that really matters, and it is the decision threshold. Turn it one way and you catch more defects while rejecting more good product; turn it the other and the reverse happens. There is no setting that avoids both errors, so the only sensible way to choose is to price them. An escape — a defect that reaches the customer — costs the return, the rework, the freight, the credit note, the corrective action report and some quantity of relationship damage. An over-rejection costs the scrapped or reworked part, the operator time to re-inspect it, and a slice of line throughput. These are different numbers, and in most plants both of them are already known.
Do the arithmetic explicitly rather than leaving it implied. Suppose a part costs 40 TL to make and a scrapped part is a total loss, while an escape costs 2,500 TL all-in once returns handling and the customer claim are counted. The cost ratio is roughly 60 to 1, so you should be willing to accept around sixty over-rejections to prevent one escape, which pushes the threshold hard toward recall. Now change the part to a 900 TL machined casting with an escape cost of 3,000 TL and the ratio collapses to about 3 to 1, where the same threshold would be indefensible. The threshold is a business parameter that happens to live in a configuration file, and quality should sign it off, not the person who trained the model.
This is why optimising for F1 is the wrong default. F1 weights precision and recall equally, which encodes the assumption that a false positive and a false negative cost the same, and on a production line they essentially never do. Worse, the F1 optimum can sit at a point the plant will not tolerate in practice. There is a human threshold as well: if the system over-rejects noticeably more than the operators' own judgement does, they will begin overriding it, and once overriding becomes routine the system is effectively off even though the PLC still reports it as running. We treat a sustained operator override rate as a primary health metric, on the same dashboard as uptime.
The way out is often to stop forcing a binary decision. A three-way output — pass, reject, route to manual review — lets you run high recall on the reject path while keeping the scrap rate tolerable, provided the review lane has real capacity. Size that lane from the numbers: if a line runs 3,000 parts per shift and the uncertain band captures 1.5%, that is 45 parts for an inspector to look at, which is realistic. If it captures 12%, it is not, and either the threshold moves or the imaging improves. Designing the uncertain band well is usually worth more than another two points of model accuracy, and it is far cheaper to obtain.
Edge Versus Central Inference and the Millisecond Budget
Cycle time sets the budget, and the budget is smaller than people expect. A line running 120 parts per minute gives you 500 ms per part, and the model does not get all of it. Subtract trigger latency, exposure and sensor readout of 5 to 30 ms, image transfer, any preprocessing, and then the time the PLC needs to actuate the ejector before the part passes the reject station. On fast lines the realistic inference window we design to is 15 to 60 ms, and on slower assembly stations 200 to 500 ms. Write the whole budget down as a table before choosing a model, because the model choice is a consequence of that table rather than an input to it.
That budget is what usually rules out a central GPU. A round trip to a server in the same plant costs 2 to 10 ms on a healthy industrial network, and considerably more once a 5 megapixel image is moving across it; a round trip to a cloud region adds 20 to 80 ms before the model has done any work at all. Add jitter, a switch under load, or a firmware update on a network appliance, and a system that comfortably met its budget in testing starts dropping parts on a Tuesday afternoon. For anything above roughly 30 parts per minute we default to inference on the line, on an industrial PC or an embedded accelerator sitting next to the camera.
Edge inference has its own price and you should quote it honestly. Models must be quantised and compiled for the target accelerator, which changes numerical behaviour slightly and means you validate the deployed artefact rather than the training checkpoint. Updates need a rollout mechanism that does not stop the line, and you have acquired fleet management across dozens of devices. The same compression logic that produces deployable vision models is what makes small models practical at the edge generally, discussed in small language models. Where images cannot leave the plant at all for contractual or regulatory reasons, the choice stops being an optimisation and becomes an architecture constraint, which is the subject of on-prem and sovereign deployment.
Drift: The Line Changes Even When Your Model Does Not
A vision model is a fixed function pointed at a process that will not hold still. The supplier of the raw material changes and the surface finish shifts with it. A new material lot has slightly different reflectivity. LED illuminators lose output over thousands of hours, so mean image brightness drifts downward across a year. A camera window collects a film of coolant mist. Someone opens a roller door in summer and daylight reaches the station. A maintenance technician replaces a lens and sets focus by eye. None of these events change a line of your code and all of them change your inputs, which is why a model that passed acceptance in March can be quietly unreliable by September.
Monitoring for drift does not require deep instrumentation, but it does require that someone actually looks. Three signals cover most of it. First, reject rate on a control chart, per station and per shift, because a step change in reject rate is almost always a process or imaging change rather than a genuine quality event. Second, the distribution of model confidence scores; a distribution that shifts its centre or grows a second mode is telling you the input population has changed. Third, plain image statistics — mean brightness, contrast and a focus measure per station — which catch lamp ageing and a dirty window well before the model's own outputs do.
The cheapest control of all is a golden sample run. Keep a set of ten to twenty physical parts, some good and some with known defects, and pass them through the station on a fixed schedule — once per shift on a critical line, weekly elsewhere — logging the scores. When the score on a known defect drifts by more than an agreed margin, you have a signal that precedes any customer complaint by weeks. Plan retraining as a cadence rather than as an emergency: two to four times a year is typical, tied to lot and supplier changes. The general practice around monitoring a deployed model is covered in observability and evaluation.
The Labelling Operation and Inter-Annotator Disagreement
No model is more consistent than its labels, and on borderline cosmetic defects human labels are considerably less consistent than anyone expects. Give the same fifty marginal images to two experienced inspectors from the same plant and agreement in the 70 to 85 percent range is normal; on scratch severity near the acceptance limit we have seen it lower still. That disagreement is not carelessness. It reflects a specification that says no visible scratches without defining visible under which lighting, at what distance, over what length. The model inherits the ambiguity and then gets blamed for it at acceptance, when its errors turn out to cluster precisely where the two inspectors disagreed with each other.
The fix is a written defect standard with pictures, produced before labelling starts and owned by quality rather than by the vision team. For each defect class it should carry a definition, a measurable threshold wherever one exists — length over 2 mm, depth over 0.1 mm, area over 3 square millimetres — and a boundary gallery: five images that are acceptable and five that are not, chosen from the hardest cases rather than the obvious ones. Then measure agreement. Have two annotators independently label an overlapping sample of 200 to 300 images and compute agreement per class. Classes falling below roughly 80 percent agreement need the standard rewritten, not more training data thrown at them.
Run labelling as an operation with a queue, an adjudication path and an audit sample, rather than as a task someone squeezes in between shifts. Disagreements go to a named adjudicator whose decision updates the standard, so the standard improves as the corpus grows instead of ossifying. Budget the effort honestly: bounding-box labelling of a straightforward defect runs somewhere around 10 to 30 seconds per image once the annotator is trained, while pixel-level segmentation of a diffuse cosmetic defect can take 2 to 5 minutes. A 5,000-image segmentation set is therefore several weeks of work, and it is usually the largest single line item in a first vision project.
Anomaly Detection When You Have Almost No Defect Examples
On a new line, or on a process good enough that defects are genuinely rare, you may have thousands of good parts and eleven defective ones. Supervised classification is not available on those numbers, and waiting a year to accumulate examples is not a plan anyone will fund. Anomaly detection inverts the problem: train only on good product, learn what normal looks like, and flag deviation from it. In practice this means an embedding of image patches from good parts and a distance or density measure against that reference, which gives you both a score and a heat map showing where the deviation sits. It puts a usable system into the plant in weeks rather than after a year of defect collection.
The limits are real and you should state them up front rather than at acceptance. Anomaly models detect that something is unusual, not what it is, so classification and disposition still need a human or a later supervised stage. They are sensitive to exactly the variation discussed earlier: an unfixtured part, a new material lot or a dirty lens all read as anomalous, and false positive rates of 3 to 10 percent are common in the first weeks before the normal set has been broadened enough. They also miss defects that are structurally similar to normal variation, which is why a subtle uniform discolouration is a poor fit. Expect to retrain the normal model each time the process legitimately changes.
The pragmatic pattern is staged. Deploy anomaly detection first, use it to harvest candidate defects that operators then confirm or reject, and let it build the labelled set a supervised model will eventually need. After three to six months you typically have enough confirmed examples for the two or three defect classes that actually dominate your scrap, and you move those to a classifier while the anomaly model keeps catching the long tail. Vision-language models are also becoming useful for describing an unusual region in words for an inspection report, a capability discussed in multimodal AI, although we would not put one on the reject decision path today.
PLC and MES Integration: What Happens on a Reject
The model producing a score is perhaps a third of the system. What follows it is where projects meet reality. The verdict has to reach the PLC inside the cycle, over a deterministic industrial protocol, with an explicit timeout and a defined behaviour when the verdict is late. Decide what late means before you deploy: on most lines a missing verdict must pass the part and raise an alarm, because stopping a line on a network hiccup is a far more expensive failure than one uninspected part. On safety-relevant characteristics the default has to be the opposite. Write that rule down and have the plant sign it, because it will be exercised within the first month.
Traceability is the quiet requirement that appears in the final week if you let it. For each part the system should record the identifier, timestamp, station, verdict, score, the threshold in force, the model version and a reference to the stored image, and it should push a result to the MES against the work order. Two questions decide the storage design: how long images must be retained, commonly 30 to 180 days for good parts and permanently for rejects, and whether an auditor or a customer will ever ask you to reproduce a specific decision. If they will, then model version and threshold must be recoverable per part, not merely per deployment.
Then the human path, which is routinely designed last and should be designed first. There must be a defined way for an operator to override a verdict, that override must be logged with a reason code, and the override rate must be visible on a dashboard, because it is the earliest indicator that the system has lost the line's trust. Uncertain parts need a physical destination — a review lane, a marked bin — not just a flag in a database. And keep automation scoped: a vision result can trigger a downstream workflow such as notifying quality or opening a nonconformance, and the operational cautions for that kind of automation are the subject of agentic workflows in production.
How We Run a Vision Pilot
We scope a first vision project to one station, one part family and two or three defect classes, and we resist every attempt to widen it. The reason is not caution but measurability: a single station gives you a baseline reject rate, a known escape history and an operator population small enough to train properly, so at the end there is an unambiguous answer about whether the thing worked. A typical schedule is two to three weeks of imaging trials and fixture design, four to six weeks of data collection and labelling running in parallel with model work, and two to four weeks of shadow running before the system is permitted to act on anything.
Shadow mode is non-negotiable in our sequence. The system scores every part and records a verdict while the existing inspection continues to make the actual decision, which gives a direct comparison on real production and, more importantly, surfaces the disagreements so they can be adjudicated against the defect standard. Acceptance criteria are agreed before that phase begins and written as an operating point rather than as a metric: recall at a stated precision on the held-out physical defect set, an over-rejection rate the line is willing to absorb, an end-to-end latency ceiling in milliseconds, and a defined behaviour on timeout. If a criterion cannot be measured in shadow mode, it is not a criterion.
We also say no reasonably often, and we would rather say it early. If the defect cannot be imaged repeatably at an acceptable cost, if the specification is subjective and quality is unwilling to write a defect standard, or if the line's true defect rate is so low that a full year of collection would still leave twenty examples, we prefer to report that at the end of the imaging trial than to deliver a system that gets switched off after four months. In a meaningful share of the enquiries we look at, the correct outcome of phase one is a better fixture and a rewritten inspection standard, with the model deferred to phase two.
Our computer vision engineering work sits mostly around the model rather than inside it: imaging trials, fixture and lighting specification, the labelling operation and the standard behind it, threshold selection from the plant's own cost figures, edge deployment and update mechanics, PLC and MES integration, and the drift monitoring that keeps the system honest after handover. We are equally direct about the cases where a mechanical fix, a simple gauge or a better process control chart solves the problem more cheaply than any model would. A vision system is a maintained asset with a running cost, not a delivered artefact with a warranty.