What's new
What's new

question on precision level specifications

metalmagpie

Titanium
Joined
May 22, 2006
Location
Seattle
I'm looking at a level that looks good to me. It's spec'd at ".05 M/M".

What does that actually mean? Is it a good level, or more like a builder's level?

metalmagpie
 
I suspect something got lost in translation. Normally a "length per length" spec for a level would mean that one division on the bubble/scale corresponds to a rise or fall of the 1st length over the 2nd length. But 0.05 M/M would mean if you set the level on a meter stick or 3 foot straightedge and lifted the end of the stick/straightedge 50mm (almost 2 inches), the bubble would only move 1 division. Which is a really crummy precision.

I have sometimes seen foreign specs where M/M is an abbreviation for mm. In this context, I'd interpret that as "over the length of the level, whatever that is, a rise or fall of 0.05mm is one division on the bubble". If that's what it is, that seems like a pretty good precision.
 
I'm looking at a level that looks good to me. It's spec'd at ".05 M/M".

What does that actually mean? Is it a good level, or more like a builder's level?

metalmagpie

What he said. But who made it?

Anybody can CLAIM a specification.

My Lufkin is marked .0005 in/ per FOOT, the Scherr the same.
The 3" block levels .0005" in 10 inches - near-as-dammit 25 cm, FWIW.

I THINK I've seen the markings you describe on a Polish "VIS". Which are very durable and well-regarded goods. I have one of their squares, but not their level.

A maker in France makes another good one - I think Weiler actually uses it, re-branded?

And then there is Japan.
 
I have levels that are speced 0.000'2" in 10". (And checked with a surface plate and shim stock they are at least close to that.) That would be 0.000'8" in 40" or 0.020mm / M (So over 1 meter 20microns out of level would move 1 full line on the level.)
 
I have levels that are speced 0.000'2" in 10". (And checked with a surface plate and shim stock they are at least close to that.) That would be 0.000'8" in 40" or 0.020mm / M (So over 1 meter 20microns out of level would move 1 full line on the level.)

Two seconds is an angular dimension, no? Thus it would always be 2 seconds over whatever length. Now .020 mm/M is an actual ratio of lengths. Calculating arctan(2E-5) and converting it to hours minutes seconds gives 4 seconds which is double the angle originally specified.

I figure 2 seconds of arc is equivalent to .0001" in 10".

Where am I wrong?

metalmagpie
 
Last edited:
Two seconds is an angular dimension, no? Thus it would always be 2 seconds over whatever length. Now .020 mm/M is an actual ratio of lengths. Calculating arctan(2E-5) and converting it to hours minutes seconds gives 4 seconds which is double the angle originally specified.

I figure 2 seconds of arc is equivalent to .006" in 10".

Where am I wrong?

metalmagpie

You aren't wrong.

The confusion comes from the symbol " being used for arc-seconds OR inches.
 
Last edited:
I suspect something got lost in translation. Normally a "length per length" spec for a level would mean that one division on the bubble/scale corresponds to a rise or fall of the 1st length over the 2nd length. But 0.05 M/M would mean if you set the level on a meter stick or 3 foot straightedge and lifted the end of the stick/straightedge 50mm (almost 2 inches), the bubble would only move 1 division. Which is a really crummy precision.

I have sometimes seen foreign specs where M/M is an abbreviation for mm. In this context, I'd interpret that as "over the length of the level, whatever that is, a rise or fall of 0.05mm is one division on the bubble". If that's what it is, that seems like a pretty good precision.

I am writing this for all the forum members: I usually write my answers so anyone reading the thread can figure it out.

In machine lingo mm means millimeter and M is meter or 39" (most say 40" as the math is easier). 0.05 mm = 0.0019" or .002" per division - as Sf said, this level in not a super precision level. Like a Starrett 98-12 level with Accuracy 0.005 per division In per Ft I have used them when I level a lathe and if you eyeball the bubble on a line. Making it close enough Because lathe accuracy is .0002" in 12", Lets say your lathe bed is 36" long, so you are allowed .0006" in 36". The levels I use in Scraping machines - Starrett 199Z level with the accuracy of .0005"/12" (per division) or a King-Way level that is .0003" in 12". For the heck of it I looked at a Carpenters level and the one I saw was .006" in 12" . I hope that explains it.

In Machine rebuilding and leveling the rule of thumb is - Conventional or a regular lathe like a South Bend or a Bridgeport mill we scrape the machines to .0002 / 12" and on precision machines, a tool room lathe like a Mori Seiki, a surface plate that we call "Super Precision" we shoot for .00005" in 12".
 
I can hit 2 tenths per foot in a regular heated shop environment if careful. Be darned if I can hit 50 millionths/foot. Or measure it on a scraping project. If you’re doing a precision machine rescrape shooting for 50 millionths/foot how are you printing it?
 
One arcsecond subtends 0.001 inch at a a distance of roughly 206 inches, a bit over 17 feet. Two arcseconds would subtend 0.001 inch at roughly 103 inches.

One arcminute subtends 0.001 inch at a distance of 3 7/16 inch, very nearly.

I'll also mention that the English-speaking world has historically graduated level vials in 1/10 inch increments, while the rest of the world generally graduated level vials in 2 millimeter increments. This means that specifying the sensitivity of a level vial in "tilt per division" terms is ambiguous unless the length of division is included in the citation.
 
One arcsecond subtends 0.001 inch at a a distance of roughly 206 inches, a bit over 17 feet. Two arcseconds would subtend 0.001 inch at roughly 103 inches.

I agree with the above.

One arcminute subtends 0.001 inch at a distance of 3 7/16 inch, very nearly.

Again, right on.

I'll also mention that the English-speaking world has historically graduated level vials in 1/10 inch increments, while the rest of the world generally graduated level vials in 2 millimeter increments. This means that specifying the sensitivity of a level vial in "tilt per division" terms is ambiguous unless the length of division is included in the citation.

Hooray! I think you have just identified the source of my long-standing confusion.

Thank your for an excellent post!

metalmagpie
 
It's hard to scrape iron when it's cold. I try to keep the shop at 68 F when I am scraping and rebuilding. Temp is so important when scraping. Years ago I was scraping a machine inside a guys shop and at night he turned the heat down from 65 to 50 F. I did not know that and told him I could not scrape accurately with the temp change. So he had us build a plastic room around the area and put in an electric heater. He also had us near an overhead door and in the MN winter when it can be -20F opening an overhead door and getting a blast of cold air on one side of the machine and not the other you have to quit scraping. Sunlight shinning through a wonder or a heater blower blowing on one side can change geometry. Ink spreads OK at 68 F. One more thing is as long as you keep the temp the same all the time you should be OK. If your shop is at 50 F or 80 F as long as it stays the same because iron expands the same. Scraping .00005 can be done with thing blue and rubbing it a lot to get the shinny mirror high spots. I have been teaching rub a part on a plate or way - to count to 10 or more, not a 1 or 2 slide rub. The more you rub the ink changes colors. Stan, look at your class booklet or my DVD as I explain that in them.
 
As I understand it 0.05mm is roughly equal to 0.002". I would assume, unless it is stated otherwise on the level, that the degree of accuracy is 0.002" in the length of the level. Not bad for general shop work but not really good enough for machine installation and rectification work. Not for me anyway. Of course in the unlikely event of your level being a metre long that would be another story.

Regards Tyrone.
 
Yeah, Richard, I do remember your teaching, and can replicate results on a simple project like a surface plate, straight edge etc. Although I’d hesitate to claim flat within 50 millionths, I am getting flat to the tolerance of a new single A 30” x 48” surface plate. And greater than 40ppi. That’s not hard just takes awhile.

It’s on machine parts that I’m talking about, that I can’t put on the surface plate, where I can get to 2 tenths per foot reliably, but better than that seems to be by chance only. Shop is radiant floor heated, with two ceiling fans (hopefully) distributing heat evenly, and not taking any measurements till door to outside has been shut for at least two hours.

Even with lots of rubbing and thin blue I’m not getting a print off a straight edge reliable and hinged well enough to hit 50 millionths/foot.

Apologies to the OP for straying from his topic...

L7
 
Of course in the unlikely event of your level being a metre long that would be another story.

Bit over a meter do yah? 42 1/4" long by 4 1/16", actually.

Not sure how "likely" that was "back in the day" nor why that length unless something specific to a flour mill needed it?

Mostly it earns it keep in the present-era lending-out its huge sole plate as a surface to reference "other stuff" off-of.
 
It's worth adding that European and Asian precise block levels having vial sensitivities of 0.02 millimeter per meter per 2 millimeter division are fairly common. This 0.02 millimeter per meter "trigs out" to a skosh over 4 arc seconds per division, but the vial radius is very close to that of a 5 arc seconds per 1/10 inch division . . . because the 1/10 inch division is very nearly 25 percent longer than a 2 mm division.

Incidentally, there have been several academic studies showing that 2 mm vial graduations can be reliably divided by eye to 1/4 division. while 1/10 inch graduations can be reliably read visually to 1/5 division.
 
Metalmagpie --

My guess is that the "0.05 M/M" citation is intended to mean "0.05 millimeter per meter", which trigs out to 10 arc seconds. That's probably per 2 mm division, so it would be somewhat less sensitive than a Starrett 199, but generally comparable.

John
 
Metalmagpie --

My guess is that the "0.05 M/M" citation is intended to mean "0.05 millimeter per meter", which trigs out to 10 arc seconds. That's probably per 2 mm division, so it would be somewhat less sensitive than a Starrett 199, but generally comparable.

John

"If only.." we could get a brand & model.. or a photo.. someone here - or several "some ones" ... probably HAS <whatever>. High, Low, or Medium grade & precision, most any maker.

.... and we'd not have to "guess".
 








 
Back
Top