Ghost in the Machine [again]

Fanuc 18T - prox 1997 vintage.

15 yrs ago (?) I had a ghost in this control. It would take the back end of this program and flop it over onto that program from random progs in the library. It was cornfusing, and a hassle, but it never did it to a program that was in use. Just an occasional prog in the library. So - no actual crashes.

Hardinge was clueless, but they referred back to Fanuc, and came back at me to dump all my prog's and reload.

So I did, and for 15 yrs it's been fine from that time on...

My library was not completely full at that time, but it was maybe 75% full?
The library has been completely full for some time now.


Now, I have since then had issues with an 18iT dooing the very same thing - if I let the library get too full. Dump it out and start fresh, and all is right with the world. This control has done this for many yrs, and will still doo this today if I fill the library up. But again - no worries as other than botched progs in the library, and possibly locking up the control, it has never caused an issue with a running program. It just makes a fella scratch his head when trying to set up the next job sometimes as you scroll through the prog - and it just doesn't make sence...


But recently - the original 18T (not i) that has NOT caused any grief in the last 15 yrs, has all of a sudden learned a new trick. It will sometimes decide not to bother running the toolchange (index turret) macro, yet thinks that it did, and continues on...

I am sure that it is NOT a mechanical issue as the turret is never mis-indexed. (not seated) and it is always at the exact same place in the prog and crashed exactly the same way. Now it may be a cpl hundred cycles between crashes, It may be several hundred. It may be less than 100, but the outcome is the same.

The CRT shows that the proper tool number is called up in the BRH corner of the CRT. Meaning that if T3 is s'posed to be up, the CRT reads T303 in the BRH corner, yet the turret is still in position 7 let's say... (It is not good)

I have put it in MDI and cycled an indexing prog with M99 to see if I could make it mess up between the two positions that it was missing the index on, but of course you can't make it doo it while you're watching...


It did it just now in a different place that the last cpl times on this same prog, but that appears to be b/c it did it during a Top Cut macro. But otherwise, it always does it at exactly the same place in any given program.

It seems to me to obviously be that the control is blowing through the macro that calls the tool change. Now - how that happens is way above my pay grade. But I am relatively sure that is what it is dooing.

This control has been "on" for most of it;'s life. Running most of it, but ON most all of the time. My mind was wondering on about what's the scoop, and I had a flashback to the computer "hairs" (?) or whatnot - that grow over time on circuit boards. Seems like they have addressed this to some degree over the years - maybe (Winders still crashes, so .....) but I think that I have ruled this out due to the NON-randomness of the issue. It doesn't doo it often, but when it does it - THIS is when it will doo it. Make sense?


I have completely dumped all progs and macro's. Left shut down overnight, over weekend, reloaded only the progs needed, and still there. I thought that it was gone at first - as it didn't miss-hit for a few days strait, but - it's back like a bad penny... (whatever that means)



Has anyone had any similar issues? And if so, what did you doo about it?

I have not been getting many hours on it lately as I don't bother to leave it run overnight - in hopes that shutting it down overnight will help it to be able to run through the day w/o crashing. ... having mixed results with that....

On the current job it likes to run the live cross drill that is in 4th position into the bar - thinking that it is a TNMG in pocket 3. I am down to my last collet nut for those holders, and it seems that my contact @ Gosiger must be MIA as I have not gotten any replies from my e-mails to her...

I have made a post-it note to CALL there tomorrow.


I have never dumped the params and such out.
That would prolly be next on my list to try.
However I might have to send it out to have the ladder dumped and reloaded. ???


Otherwise I will likely hafta git a new or at least used control for this.


---------------------------

I am Ox and I approve this h'yah post!

I can't help you with the program scrambling issues. However, I don't think this is related.

Lathes don't typically have a "macro" for the tool change. The ladder would handle those duties. Do you have a hard copy of the ladder?

What have you done to rule out a mechanical problem? Have you changed the turret encoder? Had a look at the turret limit switches?

I rule it out b/c it's never been mis-indexed.
It's never been between lock points, and if it were, then the "UNCLAMP" prox would pick it up and alarm out.


It appears to simply not even try to index at one point, and then it is off that count until it crashes into something.


"Macro" , "Ladder" , whatever ... I understand that it is not a 9000 series macro, but I can't see a G76 macro either, and I know that it exists somewhere... so ...

... one way or t'other - it's simply NOT cycling through the required PLC cycle that lifts the turret, indexes it X amount, and locks it back down. It's simply skipping the whole process, yet is updating the T value in the CNC.

I guess that it could be argues to be a PLC issue, but if that was the case, then it would skip the process at any toolchange called randomly. But since it is always / only one in particular in any given program, I hafta think "CNC".

I have no clue how it decides which toolchange point will be the one that crashes tho. It hasn't done it enough for me to find the least common denominator as of yet (from program to different program) and I don't see how I can wait long enough to get to that point.

Why doo I need a hard copy of the ladder?
It's a Fanuc - I can see and watch it!



-----------------------

I am Ox and I approve this h'yah post - but I doo not approve of this ghost!

Ox, the worst of the chip "whisker" problems would happen regardless of whether power was on or not. It's certainly not impossible for an elderly (for a computer) machine control. But, as you no doubt know, there are lots of other things that can go out with machine age...

Ox said:

Has anyone had any similar issues? And if so, what did you doo about it?

Click to expand...

Yes. Long enough ago we could still put a logic probe onto the circuitry and single step it with paddle switches to find a 'sticky bit' that turned a machine language op code into sumthing other than programmed.

More recently with T86 and descendents.

It used to be that all-hands could do that. And did.

Doubt it is so now. Too many layers between man and machine have been added for all but a few specialists to still go there. Wiser and cheaper to replace the memory or PCB and just go for coffee.

That may no longer be practical, either, on a machine the age of this one.

But 'sticky bit' is what I'd have looked for first, flawed logic gates next.

Can you do memory stress testing?

How about selectively chilling sub-sections of it to see if it behaves when cold(er) so the truant part can at least be narrowed-down.

Software-bug-wise, how about adding NOPs to the code so that an errant JMP in a not-often called routine doesn't go to the same place it once did?

Between those two, you should be able to ascertain if it is hardware or software.


Bill

Monarchist said:

Yes. Long enough ago we could still put a logic probe onto the circuitry and single step it with paddle switches to find a 'sticky bit' that turned a machine language op code into sumthing other than programmed.

More recently with T86 and descendents.

It used to be that all-hands could do that. And did.

Doubt it is so now. Too many layers between man and machine have been added for all but a few specialists to still go there. Wiser and cheaper to replace the memory or PCB and just go for coffee.

That may no longer be practical, either, on a machine the age of this one.

But 'sticky bit' is what I'd have looked for first, flawed logic gates next.

Can you do memory stress testing?

How about selectively chilling sub-sections of it to see if it behaves when cold(er) so the truant part can at least be narrowed-down.

Software-bug-wise, how about adding NOPs to the code so that an errant JMP in a not-often called routine doesn't go to the same place it once did?

Between those two, you should be able to ascertain if it is hardware or software.


Bill

Click to expand...

That was kinda my "Vibe" if its not -puter and not mechanical there is still a ton of "sticky" electronics in between where -puter thinks that everything is fine-ish and there are no apparent mechanical problems. Definitely the phrase in my mind was also "sticky" :-)

BTW OX "man" that was a really excellent description of the problems you are having...

cameraman said:

BTW OX "man" that was a really excellent description of the problems you are having...

Click to expand...

Agreed. Were it otherwise, *I* wudda just gone for coffee.

Those memories were seldom 'fond' ones...

OX, the difference between 7 and 3 is 4, bit number 4 might be the problem and the control thinks that it's on tool 3, when it's on tool 7, and never does an index.

If the turret encoder is a BCD encoder and not a gray code style, then you could have a flaky wire for the 4 bit.

Hardinge did a BCD encoder on the CHNC series. On the interface board in the back there were 4 LEDs and you read the turret position by BCD decoding the 4 LEDs to make sure they matched the actual station facing the spindle.

* BCD - Binary Coded Decimal, 1 wire per bit, 1-2-4-8-16
* Gray code - the A/B 2 bit pattern used on typical shaft encoders to denote direction and count.

Perry Harrington said:

OX, the difference between 7 and 3 is 4, bit number 4 might be the problem and the control thinks that it's on tool 3, when it's on tool 7, and never does an index.

If the turret encoder is a BCD encoder and not a gray code style, then you could have a flaky wire for the 4 bit.

Hardinge did a BCD encoder on the CHNC series. On the interface board in the back there were 4 LEDs and you read the turret position by BCD decoding the 4 LEDs to make sure they matched the actual station facing the spindle.

* BCD - Binary Coded Decimal, 1 wire per bit, 1-2-4-8-16
* Gray code - the A/B 2 bit pattern used on typical shaft encoders to denote direction and count.

Click to expand...

Good one! 'physical' check can be faster than software if this applies. Needs dooin' anyway.

Otherwise a 'trace' can show the values appearing on I/O ports too.

So can a simple port-fetch (p@) in Forth.

If you know the address and preconditions to allow the fetch, of course. Some I/O is more complex than others.

Bill

I would definitely start with the PLC, there has to be some logical method of bypassing all of the tool change logic and my guess is the "it is already there" rung in the ladder. There obviously isn't enough error-catching in that rung of logic or there is an input issue / stuck bit / something that is letting that rung of logic bypass all of the rest of the physical tool change logic. The chances it's entering the actual tool change part of the logic (physically moving stuff part) and then skipping several physical inputs is next to zero. This has to be occurring in the bypass loop part of the code.

I would start there. See what conditions in that rung allow it to activate.

>>>>I am sure that it is NOT a mechanical issue as the turret is never mis-indexed. (not seated) and it is always at the exact same place in the prog and crashed exactly the same way. Now it may be a cpl hundred cycles between crashes, It may be several hundred. It may be less than 100, but the outcome is the same.<<<<

Random is seldom random

try running a dummy program that just does that sequence of tool changes and see if it screws up, the combination of a poorly written PLC and a bad switch[Hey why bother monitoring the toolchanger, if it says its at the spot, it is there, what could possibly go wrong?]

maybe a double tool change, T1, empty tool position,T7 see if that eliminates the crash. If it doesn't it may point to a more general issue, not a switch on the toolchanger, but something that is able to say 'ignore toolchanger commands'

I'm afraid that much of what y'all have posted here is over my _ experience level, but I can prolly doo some things with some hand holding. A Fanuc service tech is 3 hrs away in Cleveland, so I am hoping to locate the trouble w/o help if possible.

This is my #1 machine that runs all-the-time, so I need this to get fixed, and soon...


Reading the last post reminded me of one other thing that may effect some of your thoughts:

When this first started - I added a second tool change line in the code. But pretty sure that it didn't make no never mind....


As for the "turret encoder", there isn't such actual unit. I know what you're talking about as my "Big Bertha" has such a unit on it, but this one just uses the encoder off the index motor. To doo a Turret Align it looks for a prox to go ON and then slows, and when it goes OFF it just counts via the 1850 value and sets down. Once it sets down - the motor is relaxed. From this point on - I am pretty sure that the motor is working in INCR mode from here to there.

I know the ins and outs of this turret assy very well as I have been in there many times (mostly related to a crack-up) and modified it heavily. My mods are a cpl yrs old at this point. I guess we had it apart and put new seals in it .. maybe 6 months or so ago, but ...

So how would this encoder update effect your responses?

I wouldn't have a clue how to "stress the memory" either.

I could maybe find the turret index by-pass loop like was mentioned earlier and take some pics of it... ???



EDIT:

BTW - I don't think that the "4" feature (position 7 - position 3) means anything. (and that suggestion my have went by the wayside with the encoder update) but in the program running now - it is s'posed to move to T3, but is in position 4.


Last known correct positioning would have been T11 (cut-off) for sure. Then it should have indexed to T1 (trigon) used as a feed stop. (ends the program) Then index to T3 to a TNMG to start the program. So on this program - it is crashing on the first tool, but that's not normally the case. (normally = I have prolly only ran 3 or so diff parts during this time-frame)

But this last crash (which thankfully only cost me a 1/2 SMD) was to have called up T11 for "end prep" for the new bar during the bar change routine, but instead had T4 (live tool) again. First time that I noticed it dooing it during this routine.


-------------------------------

I am Ox and I approve this h'yah post!

OK - I shot pics of most all of the pages of the ladder that had Turret 1 involved.

This machine does NOT have a Turret 2, which was an optional "lower axis" mechanism, which would have went in place of the tailstock or sub-spindle. This machine has a sub...

My first batch of pics were pretty fuzzy as I had to shut the flash off to get the pics, but that induces a long exposure time, and my hands aint that stable, at least after the first few pics especially... So I had to go back and try aggin... What I needed was a tri-pod, but since I am not "into" pics, I don't have one. So I chopped up a perfectly good 2by6 (used - stacked outside in the snowbank) and sat the camera on that, and steadied with the other hand to control distance to the screen, and viola !!! instant good pics!

So, some of these are still from the original batch, and if you need a clearer shot of some page(s), let me know...

So I just started a new page for this, and I can just delete the page when done...

I recommend looking at line 1222-1225. It is labeled as something interesting, but it is way off on it's own, so ???
Also - sorry that Photobucket has them scattered out of order. I don't know why they did that?

18T by Chad Oxender | Photobucket

Thanks for any help that y'all have to offer.
I will start to check on the worst case scenario now ...


------------------------

I am Ox and I approve this h'yah post!

I'll be honest, I don't know much of how to read ladder logic, but with all those symbolic names I'm a bit lost. I usually gander at a wiring diagram to get started at solving such problems. I'd sure like to see all the switches and stuff that go to the turret. But as Tony said, it sounds like it's shortcutting the change because some criteria is already met.

What model and year is this machine?

You said they are plugged in 24/7, are you running the RPCs that long, or'd you get utility power?

"All Switches"

The only electronics involved with this turret is:

Prox 1: This is the one that catches the "align" cam as it comes around.
Once the turret is HOMED it is not (should not anyhow) be relevant anymore.

Prox 2: This is the "Turret Clamp" prox. It simply tells the PLC that the top plate is seated or not.

Sol 1: Actuates the hydr valve to lift or clamp the top plate.

.. and then the servo motor that indexes it.


1997 Hardinge Conquest T51

Yes, my RPC runs 24/7/360. (60hp)
Hopefully this machine is running - at least part of the night...
It should be running 24's on the job(s) that it has right now...

I only shut the shop down if I am gunna be gone for a cpl days or more, and it is seldom that I am gone for more than a day - provided that I am able to take Sunday's off. Although this spring I did bug out for a cpl quick trips (documented by some tattle tales on this h'yah forum..) and one trip sledding this winter. I'll likely not be gone like that 'till snow flies again now...



-----------

I am Ox and I approve this h'yah post!

How are you sending programs to the control? On my Deckel Dialog, if the PC sends a bad character, the control won't detect it, but the control will puke at some seemingly random time later, often partway through a program. Fortunately, the control just goes dead and drops the main contactor to the drives and spindle, and there's no physical crash. The Dialog control is all in firmware, so a hard reboot after a puke will restore back to factory conditions without having to reload parameters.

Initially - all programs are hand punched in at the keypad.
Generally that is the end of story as a full library is about all that I need for that machine (125 programs).

But in the case that I need to back-up, or in the case a week'r so ago - I dumped them to thumb drive via a unit from CalMation. (Prolly the same unit that Shop Floor Automation sells too)

Then - I reload from the thumb drive/232


I haven't loaded from the 'putor in a LONG time. It never worked well for me, and so I just expanded the memory and life was good after that.



Either way - this has happened all of a sudden on the last 3 programs that I have ran. Both - before I dumped them, and since reloading. The same exact issue.
It's not a random bad bit of incoming code.

I only have my Macro's and the few programs that I have ran since in the Library. It is essentially empty currently.


--------------------------

I am Ox and I approve this h'yah post!

Ox said:

Initially - all programs are hand punched in at the keypad.
Generally that is the end of story as a full library is about all that I need for that machine (125 programs).

But in the case that I need to back-up, or in the case a week'r so ago - I dumped them to thumb drive via a unit from CalMation. (Prolly the same unit that Shop Floor Automation sells too)

Then - I reload from the thumb drive/232


I haven't loaded from the 'putor in a LONG time. It never worked well for me, and so I just expanded the memory and life was good after that.



Either way - this has happened all of a sudden on the last 3 programs that I have ran. Both - before I dumped them, and since reloading. The same exact issue.
It's not a random bad bit of incoming code.

I only have my Macro's and the few programs that I have ran since in the Library. It is essentially empty currently.


--------------------------

I am Ox and I approve this h'yah post!

Click to expand...

A 'sticky bit would not then have to be onboard. Could be in the plug-in storage device's memory. And no . nothing really 'random' about it. It either rears its head or NOT - temperature can be a player - but in the same marginal area and bit if/as/when it bytesyerazz at all.

One can rule that out - or confirm it - by storing and reloading the same goods as modules simply onpassed one direction, the other, or both in a different order. Or even only one at a time.

Which puts them into a different physical area of memory.

Which SHOULD show up as a fault, almost certainly of a different kind and place, in 'whatever' was laid where the present truant had formerly sat.

Or just using a different 'stick' for the transfer if you have more than one as works.

Rule-out that outboard storage device first, 'coz it should be fastest and cheapest to get to the end of.

If the change dasn't move.. then back to the onboard goods..

Bill

This started on prog's that had been in the library for some time already. "Stick" not involved to this point, or - if it had - it would have been at least 2 years ago.

This is showing up in every program since it started. Not just one point of one program.

It just did it again, and this time was during the bar change routine as the last time. Again - this is about as good of a crash as I have had - as it only takes out a 1/2 SMD.

I just dumped everything again, and will be reloading just the current part program and the few macro's required.
Control is OFF in the meantime.

I can open the cabinet door where the control lives and stuff a fan on it - just to see if that helps any. But it hasn't seemed to me to be a heat issue - meaning that it's only been the last cpl of days that it's been warm here really... But then - it seems to not crash in the AM's if left off overnight either.... So - AM temps - both ambient and machine warm-up are in that scenario as well, so ... .... I'll stuff a fan on it!

-----------------------------------------

I am Ox and I approve this h'yah post!

cameraman said:

BTW OX "man" that was a really excellent description of the problems you are having...

Click to expand...

It has happened a few times already - that I will sit down and write up a big schpeal like this on here _ for whatever issue may be at the time - only to figger it out on my own - simply from laying it all out here at one time, and trying to draw maps and explain as best that I can. These posts never get loaded. I never hit the SEND button (or whatnot).


-------------------------

I am Ox and I approve this h'yah post!