The Sensor That Read 0.05 mm Differently Every Time
We used a laser displacement sensor to measure part height. The spec said “0.01 mm resolution.” On the bench, the reading varied: 10.00, 10.05, 9.98, 10.03 — a spread of 0.07 mm. The customer needed 0.02 mm. The problem: resolution is not repeatability. The sensor’s resolution (the smallest digit it displays) was 0.01 mm. But its repeatability (how consistently it reads the same part) was 0.05 mm. We needed a sensor with repeatability under 0.02 mm (not just resolution). We switched to a higher-grade sensor (repeatability 0.005 mm). The readings stabilized. The mistake was confusing resolution with repeatability.
Sensor resolution and repeatability are different specs. This article clarifies the terms.
The Specs You Need to Understand
Resolution
Resolution is the smallest increment the sensor can display. A 0.01 mm resolution sensor shows readings in 0.01 mm steps (10.00, 10.01, 10.02). It’s the display’s fineness, not the accuracy. A sensor can have 0.01 mm resolution but repeat to 0.05 mm (the display flickers).
Repeatability
Repeatability is how consistently the sensor reads the same (stable) target. Measure the same part 10 times. The spread (max – min) is the repeatability. It’s the spec that matters for a checking gauge. A sensor with 0.005 mm repeatability gives 10.00, 10.01, 10.00, 10.01 (tight spread).
Accuracy (Linearity)
Accuracy is how close the reading is to the true value. A sensor can repeat to 0.005 mm but be offset by 0.1 mm (always reads 10.10 instead of 10.00). Accuracy is the absolute error. For a comparative check (is the part in tolerance?), repeatability matters more than absolute accuracy (you zero the sensor on a master). For absolute measurement, accuracy matters.
| Spec | Meaning | For What |
|---|---|---|
| Resolution | Smallest displayed increment | Not the key spec (display only) |
| Repeatability | Consistency on the same target | Checking gauge (in/out) |
| Accuracy (linearity) | Deviation from true value | Absolute measurement |
Step 1: What Do You Need?
For a checking gauge (is the part in tolerance?), you care about repeatability. You zero the sensor on a master (known good part). Then you measure production parts. You want the reading to be consistent (the same part reads the same). A repeatability of 1/10 of the tolerance is standard (if the tolerance is 0.2 mm, use a 0.02 mm repeatability sensor).
For absolute measurement (a CMM, a dimension report), you care about accuracy (the sensor reads the true value).
The sensor rule: For a checking gauge, spec repeatability, not resolution. The sensor that read inconsistently had 0.01 mm resolution but 0.05 mm repeatability. Choose a sensor with repeatability under 1/10 of your tolerance. Resolution is just the display’s digit.
Step 2: The Gauge R&R (Repeatability and Reproducibility)
Before trusting a gauge, do a Gauge R&R study. Measure 10 parts, 3 times each, 3 operators. The gauge’s variation (repeatability) should be under 10% of the tolerance. If it’s 30%, the gauge is too noisy (the readings don’t mean anything). The sensor that varied 0.07 mm (on a 0.02 mm requirement) would fail Gauge R&R.
Step 3: What Affects Repeatability
The sensor’s repeatability depends on:
- Target surface: A shiny/reflective surface (for a laser) scatters the beam. A rough or angled surface (article 108) degrades the reading. Use the right sensor for the surface.
- Lighting: Ambient light (sunlight, other lights) interferes with optical sensors. Shield the sensor.
- Vibration: A vibrating mount (article 48) makes the reading noisy. Isolate the sensor.
- Temperature: The sensor drifts with temperature. Warm it up (30 min) before measuring.
- Target distance: The sensor must be at the right working distance. Out of range, the reading is noisy.
Step 4: Signal Filtering
Most sensors have a digital filter (averaging). Turn on the averaging (e.g., average 10 readings) to reduce noise. The reading smooths. But it adds delay (for a fast process, too much averaging slows the response). Balance noise vs speed.
A Sensor Checklist
- Do you need checking (repeatability) or absolute (accuracy)?
- What is the tolerance? (Sensor repeatability under 1/10?)
- Did you check the repeatability (measure the same part 10 times)?
- Is the target surface compatible? (Shiny? Rough?)
- Is the sensor shielded from light/vibration?
- Is it warmed up? (30 min?)
- Is the target at the working distance?
- Is averaging (filtering) on?
- Did you do a Gauge R&R?
- Is the reading stable? (Not flickering?)
The Bottom Line
Sensor resolution and repeatability — resolution is the display’s digit; repeatability is the consistency. The sensor that read 0.05 mm differently had 0.01 mm resolution but 0.05 mm repeatability. For a checking gauge, spec repeatability (1/10 of the tolerance). Zero on a master. Isolate from light and vibration. Do a Gauge R&R. The sensor that measured reliably wasn’t the highest resolution — it had the best repeatability.
