Using security systems to debug automation

We've got many, many automated production systems. Sometimes you have issues you just cannot seem to solve and need some help.

We have a system that uses ink printers to mark parts. Parts come to it via conveyor system, they are oriented via Cognex camera system and a rotary table, then marked. Afterwards they go to an automated inspection system that verifies the mark is correct and in the correct place and orientation. It's a fairly common process in high volume production. All of this is designed and built in-house. We've had an issue with this particular system in that we get one printed backwards occasionally.... and by occasionally I mean you get ONE somewhere between 30,000 - 1,000,000 parts. Lately, it's been more toward the 1/50,000 range. It is extremely difficult to troubleshoot something like this.

Management is blaming the camera system, but we've got about as robust a camera program as you can have. Scores are excellent for both the "good" and the "bad" (we check for both and compare the two). You can't really record images because there is no set orientation for the parts coming into the system, you would fill up large hard drives in short order trying to record images. And, even if you did that, you will only see what the camera sees, not what the machine is actually doing.

So what I did was order a 1080P wireless security camera system. I've got a camera looking at each orientation device (mechanical system - this thing has 2 stations, processing 2 production streams) and then I put a 3rd camera watching the Cognex cameras. (both cameras open on split-screen on a computer - basically I'm recording the computer monitor.) This way I have synchronized time-stamped video from both the mechanical system and the vision system.

So yesterday it produced another bad part. I was able to find it on video and see that the camera did it's thing correctly, but there is some logic bug in the machine. I sent video clips to our Electrical Engineer who programmed it yesterday evening. The video will really help him troubleshoot quicker because he can see exactly where in the sequence (and what everything else was doing) when this occurred. I'm sure it's some kind of timing issue resulting in this happening, but there is also the possibility that it is a copy/paste issue triggered by the timing. (I.e. copying code and miss changing a single tag in the copied logic.) Actually the tag issue is quite likely since this is a dual machine serviced by a single AB plc.
It will be interesting to see what he finds when he digs into it. I'll try to update this post when he finds it.

The whole point is though, that a $400 security system may save you a lot of heartache when troubleshooting and automated system.

Very clever. Thanks for the post.

The built-in slow-mo recording on an iPhone is pretty handy as well. I've used it to catch anomalies in video timecode (hours, minutes, seconds, frames) that was intermittently misbehaving. Or screen glitches in video displays.

Chip

Hey Tony,

Good use of technology indeed!

We have leveraged the 8 camera security systems from Costco and installed them in paper converting machines for similar reasons in addition to giving operators a clear view of trouble areas that have become increasingly difficult to monitor when in production with greater deployment of safety guarding and LOTO procedures.

It has also been handy for properly handling warranty claims. (Bent Z-axis quill caught being a result of trying to remove a large part from a machine with the z-axis fully extended deep within the part with a broken / seized tool . . . the operator was doing burnouts in reverse with the part chained to the forklift mast - all on camera)

Just a follow up on this. This is still in place, but we had to add a 4th camera. The EE still couldn't figure out what the logic was doing wrong, so we set up a 3rd computer with AB software on it and he wrote a couple of bum rungs of logic that just displayed the pertinent signals and put the 4th camera on this screen. So now we had time stamped video of the cognex cameras, 2 on the machine and the actual logic rungs. I don't remember specifics of what he found when it did it again but he has apparently fixed it. It has been running for 8 months now with no more failures.

Something like this would have been impossible to fix without these extra eyes recording it.

Yep.
We used old baby monitors too.
Sometimes you just can’t see what’s happening.
Now you can get cheap thermal cameras too, one more tool in the box.

Hard to go wrong with those DVR systems.

They're so cheap now that I'm considering hooking up 4-8 cameras to every single machine to monitor the screen, the cutting area, the ATC, and the APC. The time saved in troubleshooting an issue can pay for the system in one incident. It could also easily prevent finger pointing among employees and with the machine builder if something goes wrong, and let everyone get on with their lives.

Orange Vise said:

Hard to go wrong with those DVR systems.

They're so cheap now that I'm considering hooking up 4-8 cameras to every single machine to monitor the screen, the cutting area, the ATC, and the APC. The time saved in troubleshooting an issue can pay for the system in one incident. It could also easily prevent finger pointing among employees and with the machine builder if something goes wrong, and let everyone get on with their lives.

Click to expand...

Oh the old finger pointing.... I had an operator press reset followed by cycle start in the middle of a program one time. Anyway, machine wasn't set to rewind, led to the machine drive the Z into the part and fixture, (large double column). Thankfully there wasn't much damage. Upon investigation, I pulled the operator history off the control, there is was Reset, followed by cycle start three seconds later. Upon questioning him, he swore, and still to this day swears he didn't press reset. But the machine doesn't lie.... As for video we also had an operator dump an upper frame out of the rotator while flipping it. His claim was that the part was fully clamped. Upon investigation we found the sliding clamp all the way open. Which would have led to a faulty clamp, which would have led to the part falling out. However through proper use of the cameras on the machine you would have seen the issue, and it wouldn't have given clamp confirmation. Anyway, we reviewed the security cameras (which were put in for an entirely different reason....) and found that immediately after the part fell out of the machine, he operated the clamp slide lever opening it up fully. Anyway, he denied it even after showing him the film, which clearly shows what he did before and after the incident. Thankfully nobody got hurt. Those security cameras were originally put in on the down low, unbeknownst to our maintenance crew in order to get a handle on what they were doing to our robots during "Preventative Maintenance". Which almost always left at least one cell crippled each week. The cameras were put up in the rafters and were there for well over six months before we decided to let the cat out of the bag after a safety incident happened. Should have seen how scared the night crew was once they found out they had been filmed and documented extensively for months. Needless to say, we had long before eliminated a few people and modified the pm procedures, with care attention on what not to do. It ended up being a mixture of things, but mainly lack of procedures being completed as indicated, and lots, I mean lots of driving into things and rough housing with the arms while working on them, and zero understanding of how to maintain arm cable management.

Oh the good ol days.

I've been asked to add process monitoring cameras to any new machines or lines I design. I'm just finishing up a line of 5 robotic cells with various processes (pick & place, dispensing adhesives, screwdriving) so I want to add a camera to each cell. I've never looked into these things before so I'm a bit confused by the lack of information. I'm used to machine vision where I know the camera's imager size, pixel size, the characteristics defining the lens, etc so that I can do the math to understand field of view at a given distance, focal length, depth of field, etc. With these security cameras all I see is generic information like the angle of coverage and the maximum distance. For example, if the camera says it's good out to 100 feet, how do I know it will be in focus at 3 feet? Maybe I'm overthinking it and any of these cameras will work find inside the machine but I don't have time for trial and error at the moment. Is there a good brand and/or reseller I can work with so that I can specify cameras that will have the correct resolution and decent focus at the distances I need inside my cells?

Also, the brand Hikvision keeps coming up in my searches. Are they decent quality or Chinese junk?

You are unlikely to find that information in consumer level gear.
But then consumer level equipment gear probably isn’t the right choice for long term industrial monitoring.
Short term, debug things like that, sure.
Not for permanent long term use...if only for support.

SICK builds solid robust camera systems...you can get the cameras without the vision monitoring.