Gauge selection: choosing the right instrument for the tolerance

A gauge is chosen by arithmetic, not by what is nearest the bench. The tolerance band sets two limits: the resolution has to be fine enough to see a tenth of the band, and the expanded uncertainty has to be small enough to leave three quarters of it for the process. Get those two numbers right and the instrument picks itself.

Most arguments about inspection start here. A supplier measures a flange with a caliper and reports it conforming; the buyer measures the same flange with a height gauge and reports it out. Both instruments work, both are in calibration, and the disagreement is 0.04 mm on a callout that never said which method applied.

What follows sets out the two ratios, the instruments that fit each band, the drawing defaults that decide the band when nothing is written, and the point where inspecting harder stops helping.

The two ratios that choose the instrument

The first rule is the rule of ten: the instrument should resolve at least one tenth of the tolerance band. On a ±0.5 mm call the band is 1.0 mm, so a resolution of 0.1 mm is enough and a 0.02 mm vernier is five times finer than required. On a ±0.05 mm call the band is 0.1 mm, and the resolution has to be 0.01 mm or better.

The second rule is the test uncertainty ratio: the expanded uncertainty of the measurement should be no more than one quarter of the band. On the same ±0.5 mm call, 1.0 mm divided by 4 gives 0.25 mm of permitted uncertainty, and any hand tool clears it. On a ±0.1 mm call the band is 0.2 mm, the permitted uncertainty is 0.05 mm, and a caliper at ±0.02 mm sits right on the limit.

The two rules point the same way but they are not the same test. A gauge can resolve 0.001 mm and still have an error of 0.02 mm; the display is not the accuracy. That is why the register carries both the resolution and the maximum permissible error, and why the selection decision uses the error.

Worked through on four common callouts, the arithmetic looks like this. A ±0.5 mm band of 1.0 mm needs 0.25 mm of uncertainty, which a caliper meets by a factor of six. A ±0.2 mm band of 0.4 mm needs 0.1 mm, which a caliper still meets. A ±0.1 mm band of 0.2 mm needs 0.05 mm, where a caliper is marginal and a micrometer is comfortable. A ±0.05 mm band of 0.1 mm needs 0.025 mm, and now only a micrometer, a bore gauge or a CMM will do.

Resolution is not accuracy

The digital caliper is the most over-trusted instrument in a sheet metal plant. A 0–150 mm caliper with a 0.01 mm display typically carries a maximum permissible error of ±0.02 mm, and that figure is for a new instrument measured under laboratory conditions with a defined measuring force. In a press shop the same caliper picks up jaw wear, a loose slide, a dropped corner and an operator who closes it on the part with two hands.

The practical consequence is that a caliper is a 0.05 mm instrument, not a 0.01 mm one. That is fine on the ±0.5 mm flanges that make up most of a fabricated assembly, and it is not fine on a bearing seat, a shaft hole or a bend allowance check where 0.1 mm changes the fit.

Micrometers behave better because the anvil and spindle give a defined contact and a friction thimble gives a defined force, but they only measure what fits between the anvils. A 0–25 mm micrometer cannot check a 40 mm flange, and a deep-throat micrometer is needed to reach a flange mid-panel.

What the drawing asks for when it says nothing

Most fabricated parts carry a general tolerance note in the title block rather than a tolerance on every dimension, and that note decides the gauge. ISO 2768-1 sets general tolerances for linear and angular dimensions that carry no individual tolerance, in four classes: fine, medium, coarse and very coarse.

For the medium class, a dimension between 30 mm and 120 mm carries ±0.3 mm, and one between 120 mm and 400 mm carries ±0.5 mm. Fine class drops those to ±0.15 mm and ±0.2 mm. The difference matters: the same 200 mm overall length is a 0.4 mm band under fine class and a 1.0 mm band under medium, and the instrument that suits one is not automatically right for the other.

If the drawing carries no class at all, the acceptance limit is undefined, and no gauge choice can be justified against it. That is a drawing defect, and it is worth raising before the first article rather than after the first rejection.

Where the measurement itself moves the number

Three effects push a reading away from the true size, and none of them is fixed by buying a better gauge.

The first is temperature. Carbon steel expands by about 11.7 µm per metre per kelvin, so a 1,000 mm part measured at 25 °C instead of the reference 20 °C is 58.5 µm longer than its 20 °C size. On a part with a ±0.5 mm callout that is 12% of the band; on a ±0.05 mm callout it is more than the whole tolerance. Long parts off the 4,450 mm brake bed are measured after they have settled to room temperature, and the waiting time is part of the inspection plan rather than an inconvenience.

The second is measuring force. A caliper closed firmly on a 0.8 mm sheet deflects it and reads low by 0.02–0.05 mm. A point micrometer with a defined anvil and a friction thimble reads the thickness rather than the squeeze.

The third is the datum. A height gauge reading a bend height from the bench measures from wherever the part is sitting, including a burr, a spot weld or a twist. The same dimension read in a checking fixture from the drawing datum is a different number, and usually the correct one.

The instruments, matched to the band

The table has one trap in it. The CMM is the most accurate instrument on the list and the least useful for lot screening. A programme that takes four minutes a part cannot check a day's production, and using it that way turns inspection into a bottleneck that the line routes around.

Bend angle is missing from the list because it is measured differently. A digital protractor reads to 0.1° on the outside face of a flange, which sounds generous until the number is converted into a length: 1° across a 100 mm flange moves the tip by 1.75 mm, and across a 2,000 mm panel it moves it by 35 mm. Angle tolerance on a long panel is therefore not a gauge question but a bend allowance and machine compensation question, and a protractor on the bench is a check on the setup, not a substitute for it.

Inspecting all of it or some of it

Gauge selection and inspection quantity are separate decisions, and the volume usually settles the second. At 1,200 parts a month, a checking fixture that takes 6 seconds a part inspects the entire lot in 2 hours. An 80-piece sample on a CMM at 4 minutes a part takes 5.3 hours and tells you less about the parts that shipped.

That inversion is why attribute gauges matter more than they look. A go/no-go fixture answers the only question the line needs answered, and it answers it for every part. The variable measurements then come from a first article and a capability run, where the question is how much and the sample is small enough for a slow instrument.

Sampling still has a place. ISO 28590 explains the ISO 2859 series and how to choose between the sampling schemes, and for a stable process the sample is cheaper and adequate. The condition is that the process has been shown capable first, which is a CMM job, not a caliper job.

What Ganesh runs

Parts run from 0.8 mm to 40 mm, with ±0.5 mm typical sheet metal tolerance and tighter features held by agreement and marked on the drawing. The 12 kW fibre laser with its 40 ft × 16 ft bed holds hole position closely enough that the checking fixture, not the CMM, is what the line uses for position; the 200 T, 5-axis brake with the 4,450 mm bed sets the bend heights that the height gauge then checks against a datum.

The first article carries the instrument and its calibration status against each characteristic, and the system behind it is ISO 9001:2015. PPAP and first article inspection are available where the customer's plan asks for them, and the gauge question is settled at the quotation stage so that the inspection cost is in the price rather than in the argument after the first lot.

What to put on the drawing and the inspection plan

1. The general tolerance class, so that untoleranced dimensions have a defined band.
2. The critical characteristics, marked, with the gauge or method that will be used to check them.
3. The datum scheme, because a dimension without a datum is measured from whatever is convenient.
4. Where an attribute check is acceptable and where a variable reading is required.
5. The acceptance rule at the limit: whether a reading inside the tolerance but within the guard band is accepted, reviewed or rejected.
6. The inspection stage: in-process, final, or both, and who owns the record.
7. For thin sheet, the anvil type and the measuring force, because flat jaws and a firm hand read low.

When a better gauge will not fix the problem

If the process is not capable, a finer gauge only tells you so more precisely. A bend that varies by 0.6 mm across a lot will fail a 1.0 mm band whatever instrument is used, and the fix is in the press brake setup, the tooling or the material, not in inspection.

The same applies where the failure is not dimensional. A powder coating that chips at the bend, a weld that cracks at the toe, a hole that burrs on the exit side: none of these shows up on a caliper, and a plant that measures only dimensions will ship all three. The inspection plan has to name the failure it is looking for before it names the instrument.

And where the tolerance is tighter than the process can hold, the honest answer is a design conversation. A ±0.05 mm position on a 3 mm CRCA bracket formed on a press brake is a machining tolerance on a forming process, and no gauge closes that gap. Send the drawing with the critical characteristics and the tolerance class, and the engineering team will confirm what the process and the inspection can hold before the price is fixed. Request a quote.

The plant and quality hub carries the rest of the inspection system. The calibration register guide covers how the instruments stay in tolerance, the bending tolerance guide sets the limits a formed part is measured against, and pressed parts are quoted through the stamping capability.