What is image segmentation, and how does it differ from detection?
Image segmentation separates regions at pixel level. In a factory, that can mean identifying the surface covered by a scratch, chipped edge or damaged seal. Object detection draws a box around an object or defect. Classification assigns a label such as acceptable or defective to a whole image or an already isolated part. These approaches answer different questions: is there a problem, where is it, and exactly which region does it cover?
Ultralytics' image classification documentation describes an image label and confidence score for tasks that do not need localisation. Its object detection documentation describes boxes, classes and confidence scores, while its segmentation documentation describes masks or contours belonging to individual objects. These are output formats. They do not define your factory's rejection policy, and the extra detail in a mask does not automatically make the final decision more accurate.
Start with the action rather than a model name. The deciding requirement is which part an operator must remove, where a robot must reach or what measurement a quality specialist must report. Unnecessary detail increases annotation and validation work. Insufficient detail makes an apparently correct prediction impossible to act on. If the system only needs to send one isolated part for review, establish what pixel-level defect annotation would add before commissioning it.
A decision table for outputs and annotations
Write the inspection rule in one sentence: which physical object goes to which action under which condition? Then use the table to identify the smallest sufficient output. The annotation format must support that output. A system trained with image labels cannot simply be treated as a reliable area-measurement instrument because its visualisation highlights a region.
| Rejection decision | Sufficient output | Annotation work |
|---|---|---|
| Any defect on one part | Image or part class | Acceptable, defective and uncertain labels |
| Remove a defective part from a tray | Part box and class | A box for each part |
| Show a missing bolt position | Object position or empty socket | Boxes and a position rule |
| Reject according to defect area | Defect mask and calibration | Pixel regions with defined boundaries |
| Scratch crossing a critical zone | Mask and critical-zone overlap | Defect boundary and zone definition |
Another option is detecting several parts and classifying each crop. Evaluate both failures to locate the part and failures to recognise its defect. Known positions in a fixed tray may remove the need for a separate detector, but you still need an explicit response to a shifted part or an empty position. The output choice follows the physical arrangement and the rejection rule together.
Validate annotation rules with a small experiment
Do not build the representative sample entirely from clean images. Include glare, blur, tiny scratches, chipped edges, partial occlusion and cosmetic marks close to the acceptance boundary. Ask two trained annotators to work independently on the same images. Record time per image, rework and disagreement on the final accept or reject decision. This gives you a measured estimate of the additional cost of masks.
The annotation instructions should define defect classes, how to draw a boundary, what to do with invisible regions and how to handle uncertainty. If experts disagree about where a scratch ends, a detailed mask can encode precision that the quality rule does not actually have. Clarify the rule first and send unresolved images for human review. Two masks with poor overlap may still yield the same rejection decision; also check the reverse situation.
Keep the part ID, product variant, batch, camera, capture time, lighting settings and annotation version with each image. Consecutive frames of the same physical part must not be scattered across training and test sets. Seeing almost identical views of one object in both sets can make performance look better than it will be on independent production. Split using independent units such as physical parts and production batches.
Turn a mask into an operational measurement
If rejection depends on area or distance from an edge, define the full chain: correct the image, locate the part, segment the defect, measure it and apply the rule. Do not report a pixel count directly as square millimetres. Camera distance, lens distortion, viewing angle and the surface plane affect the conversion. A validated pixel scale may work for a fixed flat surface; one scale may not cover angled surfaces or parts at different heights.
Define the critical zone in the part's coordinates. If the part moves in the image, fixed screen coordinates no longer represent the same physical region. Route measurements close to the decision threshold for review, accounting for calibration and annotation uncertainty. A confidence score is neither a measurement of dimensional error nor a judgement of whether a defect is physically acceptable.
Record the part ID, model and rule versions, measured area and final action with every decision. This lets you trace a wrongly removed part back to its mask, threshold or line identity match. Do not automatically treat an operator's correction as a training label; have an expert check it first. If accumulated corrections change the quality rule, run a separate acceptance evaluation for the new version and compare it with previous results.
When defect examples are scarce, PatchCore's approach trained on normal images can supply candidate suspicious regions. Do not equate its anomaly map with a validated defect mask for size-based rejection. Verify the candidate through expert review or a separate measurement. An initial pilot that routes images for inspection can be easier to manage than an unsupported claim of precise automatic measurement.
Hypothetical example: a broken latch in a twenty-part tray
The following figures are hypothetical and illustrate the selection process. Suppose a tray contains 20 moulded parts, and the rule rejects a part if a chip around its latch exceeds 2 mm². Image classification can identify a defective tray but cannot tell the removal mechanism which part to pick. Part detection can identify the right position, but the real area of an irregular chip cannot be inferred from the area of its enclosing rectangle.
For 1,000 pilot images, suppose annotation takes an average of 20 seconds for an image label, 60 seconds for boxes and 180 seconds for masks. That is approximately 5.6, 16.7 and 50 hours, respectively, before expert checking. If every chip causes rejection, boxes might be sufficient. Here, the area condition provides a concrete reason to consider the additional mask annotation effort rather than treating it as an automatic upgrade.
Assume a corrected image of a flat surface has a validated scale of 0.05 mm per pixel. A defect mask covering 1,200 pixels represents 1,200 × 0.05 × 0.05 = 3 mm². Under the same conditions, 700 pixels represents 1.75 mm². Verify that the scale applies to both axes and route borderline parts to the predefined review band. This calculation does not establish that the camera sees the entire physical defect or that calibration error is zero.
Test the model and the final part decision together
Compare candidates on the same held-out production images. Mask overlap and box quality help diagnose failures. The main acceptance outputs are rejectable parts missed, acceptable parts wrongly removed and parts sent for review. Agree targets with the quality team before looking at the acceptance results, including the denominator used for each rate.
- Independent examples: Reserve images from different days, batches and shifts. Keep repeated frames in the same group so one physical part cannot leak between evaluation sets.
- Difficult slices: Report tiny defects, occlusion and reflective surfaces separately. Include the number of actual defects so a strong percentage based on very few examples remains visible.
- Measurement checks: Compare areas with a validated reference measurement. Examine changes in the decision around the threshold, where small boundary errors can change the physical outcome.
- Line timing: Measure from camera trigger to removal of the correct part, including the identity match. Fast inference does not prove that the actuator receives the right decision in time.
- Review workload: Calculate uncertain image counts, operator time and the cost of false removals together. A model can reduce missed defects while creating an impractical review queue.
Missing images, parts that cannot be found and processing that finishes too late belong in the test plan too. Apply a review or stop path agreed with the process owner rather than silently producing a pass decision. Repeat the same acceptance set after changes to the model, threshold or camera arrangement. Keep threshold tuning on validation data, leaving the acceptance set independent.
Common mistakes
- Assuming the most detailed output is automatically the most suitable method and never measuring annotation time.
- Using a defect's box area as its true surface area, or converting a mask to dimensions without calibration.
- Hiding disagreements about boundaries and forcing uncertain examples into acceptable or defective labels.
- Using a strong average score to conceal missed tiny defects or removal of the wrong physical part.
- Tuning thresholds on the test images and presenting those results as an independent acceptance evaluation.
How Hatson Tech builds this
We start image recognition and computer vision development with the rejection rule, camera conditions and an annotation experiment. We connect the chosen output to the final action and its acceptance criteria. Our guide to image processing and quality control in production covers image collection and line integration; our article on anomaly detection without defect labels complements the plan when defective examples are scarce. Defining a pilot through which part is removed on which evidence gives the data, model and operations teams a shared basis for acceptance.