Fadal intermittently losing home position during program...

aarongough · Sep 4, 2020

Hey guys,
I have a 1995 Fadal VMC15 that is intermittently losing its home position in the middle of a program. This morning when starting the machine up I checked that the slides were aligned with the cold start markers, cold started the machine and then had it move to home. All went as expected there.

Running the first program of the day I hear a 'brrrt' and turn around to see it cutting a nice slot through the middle of one of my fixtures before finally the endmill gives up and explodes... :crazy:

Machine did not fault out.

I move it back to the cold start position and re-cold start it, and then tell it to move to home. However rather than moving in the positive direction it tries to move in the negative direction. My normal home position is with the table all the way forward, but it's trying to move all the way back instead...

The machine is actually commanding this move as well, the control shows -203.20mm in the 'distance to go' if I hit feed hold during the homing move... Turning the machine off, and then cold-starting it again consistently produces the same behaviour. Really weird that it has the length of the move right but it's going in the wrong direction. It's like a bit flipped and now it thinks the home location is a negative number rather than a positive one.

This is the second time it has happened. First time I did a little investigation, couldn't find anything wrong and figured it must have been a stray cosmic ray flipping a bit. All was well after re-setting the home position, and it stayed 'fixed' for a bit over 2 months with no issues until today.

Any thoughts on causes here? I'm just wondering if anyone has seen this symptom before and knows a likely cause... I hate intermittent faults!

I will try the following but would appreciate input!
- Test voltages on ATX supply in cabinet, maybe replace 'just in case' because I know intermittent or fluctuating voltages here can cause some very weird issues
- Test control battery voltage, maybe the battery is bad and causing some weird issue? Strange if it's causing problems when power is on though...
- Pull out all the control cards, clean them with compressed air, clean the edge connectors and backplane, re-seat boards
- Double check control to servo card cables and connectors

This machine was previously a graphite machine so I know there will be an electrical gremlin or two over it's lifetime, but I have already cleaned it *very* thoroughly, also previously replaced all contactors and relays just to be sure and it has otherwise been performing flawlessly since then...

Thanks guys!
-Aaron

Ianagos · Sep 4, 2020

aarongough said:
Hey guys,
I have a 1995 Fadal VMC15 that is intermittently losing its home position in the middle of a program. This morning when starting the machine up I checked that the slides were aligned with the cold start markers, cold started the machine and then had it move to home. All went as expected there.

Running the first program of the day I hear a 'brrrt' and turn around to see it cutting a nice slot through the middle of one of my fixtures before finally the endmill gives up and explodes... Machine did not fault out.

I move it back to the cold start position and re-cold start it, and then tell it to move to home. However rather than moving in the positive direction it tries to move in the negative direction. My normal home position is with the table all the way forward, but it's trying to move all the way back instead...

The machine is actually commanding this move as well, the control shows -203.20mm in the 'distance to go' if I hit feed hold during the homing move... Turning the machine off, and then cold-starting it again consistently produces the same behaviour. Really weird that it has the length of the move right but it's going in the wrong direction. It's like a bit flipped and now it thinks the home location is a negative number rather than a positive one.

This is the second time it has happened. First time I did a little investigation, couldn't find anything wrong and figured it must have been a stray cosmic ray flipping a bit. All was well after re-setting the home position, and it stayed 'fixed' for a bit over 2 months with no issues until today.

Any thoughts on causes here? I'm just wondering if anyone has seen this symptom before and knows a likely cause... I hate intermittent faults!

I will try the following but would appreciate input!
- Test voltages on ATX supply in cabinet, maybe replace 'just in case' because I know intermittent or fluctuating voltages here can cause some very weird issues
- Test control battery voltage, maybe the battery is bad and causing some weird issue? Strange if it's causing problems when power is on though...
- Pull out all the control cards, clean them with compressed air, clean the edge connectors and backplane, re-seat boards
- Double check control to servo card cables and connectors

This machine was previously a graphite machine so I know there will be an electrical gremlin or two over it's lifetime, but I have already cleaned it *very* thoroughly, also previously replaced all contactors and relays just to be sure and it has otherwise been performing flawlessly since then...

Thanks guys!
-Aaron

Maybe not the solution you are looking for but just keep your home at xyz zero and then just add a g53 x0 y9.5 z0 at the end of your programs to bring the table up to the front of you.

I’ve been having some weird electronics issues aswell. My y axis amp faults but after a reset or 2 it then works perfect.

I also had my power supply go bad. I took my power supply out and washed it in warm soapy water and that actually fixed it. But the rest of the cabinet is still filthy I’m considering pulling everything out and washing all the cards and spraying the inside of the cabinet out. My only fear is that I lose the labels on the wires as most are already peeling off.

David Carlisi · Sep 4, 2020

Check the woodruff keys that are on the servo motor shafts or maybe the lead screws. We have a 1995 VMC 5020A and it was doing something very similar. Turned out that one of the woodruff keys was hammered, almost sheared. Good thing about Fadal machines, they are fairly easy to work on. We cut the next size woodruff keyway and put a new key in it, and it's been fine ever since.

Bobw · Sep 4, 2020

Many moons ago, the first Fadal I ever played with. CS, and then send to home..
It would seize up. So we had to roll her forward, then set it manually.

Out on my own. Lets just keep it simple. Home is the machine home. Its one
less thing that can possibly go wrong, one less thing to think about. And its
really not hard to post your code so the table comes back to the front of the
machine.

Why it does it... No idea. And quite honestly, my simple fix of just keeping
home at the CS position has worked just fantastic for many many years.

D Nelson · Sep 4, 2020

I think the DC machines have resolver issues. The dust from the brushes of the motor get into the resolver and make it act up runaways
Don

Sent from my iPhone using Tapatalk Pro

Tedger · Sep 5, 2020

So it's just the Y axis? Swap control cards with the X axis and see if the fault follows.

macds · Sep 5, 2020

Is this a new program, or one youve run before?
Any possibility a g10 or g51.1 got in there?

Cole2534 · Sep 5, 2020

D Nelson said:
I think the DC machines have resolver issues. The dust from the brushes of the motor get into the resolver and make it act up runaways
Don

Sent from my iPhone using Tapatalk Pro

That was my guess. Any time these machines have random run aways it seem to have been a resolver or encoder issue.

I'm not super well read on the matter, just seen a couple of these pop up in Fadal FB group.

Sent from my SM-G981V using Tapatalk

aarongough · Sep 8, 2020

Thanks for the feedback guys! Just an update, today I cold started the machine and re-set the home position. This fixed the issue immediately, but it's clear that there's an underlying cause which I will continue to look for...

Investigation/remediation so far:
- Checked voltages coming from ATX power supply in control cabinet. They were all as expected within about 200mv, all were very slightly high. I believe this is ok, but I might get a spare to keep on hand.
- Pulled all the control cards and cleaned them thoroughly. I have cleaned them before but it was just with compressed air. This time I washed them all down with 99% isopropyl alcohol. The IPA turned a little dark so there was clearly still some graphite or other gunk on there, but it really wasn't much. Dried all the boards off with compressed air and thankfully everything fired back up. Not necessarily guaranteed when cleaning electronics of this vintage!
- Checked the backup battery voltage, 3.6v as it should be.
- Cleaned all the edge connectors and the backplane connectors with IPA and compressed air
- Saved all my offsets, tool lengths and programs to a USB and then zeroed the memory in the control. Re-loaded everything.
- Ran the memory tests in the diagnostic panel which all came back 'OK'.

The reason I'm suspecting a control/electronics issue is because of the way the home position switched from +203.2mm to -203.2mm. Numbers like this are stored on disk/RAM with a single bit that indicates whether the number is positive or negative. If that bit flips or is lost then you will see exactly what I'm seeing where the number 'changes sign'. This could be because of bad RAM that intermittently loses or flips bits. Sometimes RAM will go bad in very specific areas which would behave exactly like I'm seeing.

I'm not discounting other causes, and I will check out the Y axis mechanically and electrically when I have a bit more time!

Both times this happened it was right after firing up the machine and cold-starting it in the morning. In the meantime until I am satisfied that the issue is fixed I will simply run a program that would run into the machine hard limits (softly) before I run anything else, just to make sure the home position is where I expect it to be.

In the meantime I plan to run the memory test overnight and see what the results are. Sometimes you have to run RAM tests for very long periods of time before seeing an error... Will post an update tomorrow.

aarongough · Oct 1, 2020

Just to follow up on this:

Running the RAM test overnight showed no errors... The machine has been running ok since the last post, UNTIL TODAY. :angry:

Just had the exact same issue happen again, it crashed into the fixture in EXACTLY the same spot. However this time it destroyed the endmill so quickly that the machine did not fault out, I think it hit a chunk of carbide left embedded from the last crash... I hit feed hold, then sent the machine home and it went exactly where it was supposed to, front of the Y axis... Ran my program that was supposed to test for this happening and it ran perfectly fine, moved to the machine limits in all directions without issue.

Tried running the program again and it started the toolpath in the exact same WRONG place.

Turned the machine off, turned it back on and cold-started it, and suddenly it thinks the home position is with the Y all the way back again. However all the coordinates on the screen are still exactly reflective of what the machine is actually doing. It's not like the coordinates are wrong. For lack of a better way to explain it, it's like the machine is deliberately moving the wrong way lol.

Re-set the home position again and run the 'problematic' program, runs without issue in the right location... :nutter:

I'm honestly thrown for a loop by this one...

snowshooze · Oct 1, 2020

Oh.. battery is good.

aarongough · Oct 1, 2020

snowshooze said:
I am suspicious of the battery on board 5.
Mess with it, you may loose all programs, parameters, back-lashes... and have to do a clear memory and re-load everything.
But mine was forgetting programs... battery was 1.17 volts instead of 3.6
Had to start out ground zero and put everything back.
No big deal unless you haven't done it before.
All better now.

Did you just test the battery in-circuit? Or did you have to de-solder it to test?

I checked it in-circuit, but with the board out of the machine and got 3.6v previously...

Haha, just saw your edit

snowshooze · Oct 1, 2020

aarongough said:
Did you just test the battery in-circuit? Or did you have to de-solder it to test?

I checked it in-circuit, but with the board out of the machine and got 3.6v previously...

Haha, just saw your edit

No, you got 3.6 where you need it.

aarongough · Oct 1, 2020

The thing that's super weird to me is that although I can't predict exactly when the fault will happen, I can predict now that it will be the EXACT same each time... Seems like if it was a tachometer or resolver issue that it would be at least a little random? Not really sure though how those errors normally manifest.

Omega · Oct 1, 2020

Check the wires inside the seal tight conduit that runs down to the motors, sometimes the
wires will develop small cracks over time. It could be that the problem reoccurs at the same place in the program because the table is in the same position,conduit flexes.. wires short, and your off to crazy town..

pcasanova · Oct 1, 2020

thinking out loud but my 3016 had an issue after ho at shutdown the z would jump off the mark, had to reset each time on power up. Tech I know said replace the icecube relay. there's 2 but only like $12 so just replaced and problem went away. Those relay seem to be involved in the cs/ho position so wondering if you maybe swapped the two and see if an issue goes somewhere else.

snowshooze · Oct 1, 2020

aarongough said:
The thing that's super weird to me is that although I can't predict exactly when the fault will happen, I can predict now that it will be the EXACT same each time... Seems like if it was a tachometer or resolver issue that it would be at least a little random? Not really sure though how those errors normally manifest.

Yeah, resolver could be an issue here.
Especially if there are a bunch of hours, or everything is dirty.

wrrocket · Oct 2, 2020

If the machine was still feeding at a constant/commanded speed it is unlikely to be an encoder/resolver issue. Those are responsible for the speed control feedback on the axis, so if something was going wrong you would be likely to see jerky motion, or a runaway/rundown. It's unlikely to repeat in the same location. I'm not entirely sure what failsafes the control has going on under the hood, but I'd imagine the control would fault out if it suddenly got a reported 400"+ movement in a few milliseconds. Swapping the axis motors and control cards would be a good way to rule this issue out. As if it followed the axis it would be an issue with your motors/control card. If it didn't it would point to something in the controls/other systems.

Intermittent issues like this are the absolute worst to diagnose. I had an issue with a system that would randomly have an issue about every hour to two weeks; and it took my almost two months to finally figure it out, with it being a broken pin internal to a mil-spec connector.

My best bets for you would be something in the homing electronics, if it actually has mechanical components for it and not just software based. The program you are doing might have the exact right motion for it to cause a short in a dodgy cable/connector. I would expect if this was the case for it to slowly start happening in more instances.

Your guess of RAM or something on the control board going wonk is also a good choice, as usually stuff like current position is stored in the same memory locations. I would also expect it to happen in more situations than just your one program.

If you happen to have a spare control board or one from another machine, swapping them over might not be a bad way to diagnose it.

It could also just literally be a bug in the code for the machine, that you program happens to have conditions that exacerbate it.

Whatever troubleshooting you do it's best to do one at a time, no matter how painful that is. If you don't it's hard to actually pin down what's wrong. At the very least it happens close to the start of the program and not 2 hours in, which does speed up your troubleshooting.

Johnhudson · Oct 3, 2020

20 plus years ago had kinda the same issue. Are problem was heat.
Solution was extra fan in cabinet.

aarongough · Nov 11, 2020

Just had a really lovely 90 minute phone conversation with Dave DeCaussin over at FadalParts.net... I originally called to inquire about the NxGen control upgrade, but we ended up talking about a lot of Fadal related stuff! Very cool to speak with someone that was involved in the original manufacture of my machines!

I asked him about the issue I've been seeing with this machine and he immediately suggested the battery as the likely cause. Apparently there is a capacitor that runs in concert with the battery, the cap supplies backup power for the first 2 days or so after the machine is powered off, and then the battery takes over.

Testing the battery properly is apparently difficult because it should be tested with an actual load (albeit small) rather than just a voltmeter. He said there are cases where the battery will test fine with a voltmeter but when any amount of actual load is applied the voltage sags an unacceptable amount which can cause memory corruption... He also said that testing the battery in-circuit shortly after the machine has been powered up will likely not provide a good reading either.

I'll likely just pull the battery and backup cap off the board and replace them both to see if that solves the issue. They are inexpensive parts so it's worth the shot! Will follow up with results, just thought I'd relay the above info for future reference.

-A

Fadal intermittently losing home position during program...

Stainless

Stainless

Stainless

Diamond

Stainless

Aluminum

Hot Rolled

Diamond

Stainless

Stainless

Stainless

Stainless

Stainless

Stainless

Aluminum

Hot Rolled

Stainless

Plastic

Aluminum

Stainless

Similar threads