Automated Data Collection Systems

[Stupidav 1]

There are many different Data Collection packages out there, I have seen a few and used even less. We are starting to look into them to find the right one for us. I have seen ABs PlantMetrix, and seen Honeywells LETS, and used PI, and was not thrilled with it. We have AB, Mitsubishi, Automation Direct, Seimans, GE, Giddings and Lewis, and Wintress (Honeywell) controllers to collect data from. I need some ideas, directions, pointers, and developers (AB - PlantMetrix, Honeywell - LETS, ect). Any advice / opinions are greatly appreciated.

[Ken Moore 23]

We're using OSI-PI as our plant historian, connects to almost anything. It's very expensive, but worth it in my opinion. You can build HMI type graphics for management to view the plant and there lot's of excel add-ins for doing intense data manipulation.

[TimWilborne 108]

Ken, do you have a link as far as what it can connect to and how?

[paulengr 44]

I'm sure Nathan will be bringing it up but Inductive Automation's data collector is very good. Most HMI's come with some sort of data logger built-in. This does pretty good for the first few hundred to thousand points or so. The next tier up is a data historian which is an entirely different beast. Eventually you will get into MES, and I'll talk a little more about that. Many low end HMI's will log to CSV files, dBase, or some such. Avoid these. OK for troubleshooting and quick reports but if you want to get seriously into data logging, avoid at all costs. It is not worth the pain and suffering. The capability of these things isn't much better than what you could do with your choice of programming languages, an OPC server, and about one day of work. That being said, I have worked in a plant where they used Labview as the HMI and logged the data into Access database files (MDB). The plant was very small (26 people), so it worked more or less most of the time. The front end was written in Access and did fairly well. If you plan on doing everything in Excel, stop right now. In fact, don't even bother going any further. Excel does do a fairly decent job of data analysis but you will turn data analysis into a serious clerical task, every time. I have gotten very creative with Excel spreadsheets, even writing fairly extensive VBA queries (feeding from SQL databases no less), to preserve the "interface" that managers are used to. But let me tell you, this is a very crummy way to go in the long run. It is better to get all the pain of setting up a particular chart, calculation, etc., up front, and then it's once and done, forever. The next level up is a SQL based system. Most better HMI's will log directly to SQL. FactorySQL is a pretty good SQL data logger without an HMI that will set you back under $5000. Wonderware InSQL is NOT a SQL data logger in spite of the name and the advertising. If you buy it, you will be sorry, from personal experience. It's really a poor data historian that is "SQL compatible" (so are all the historians). A data historian is a high speed data collection system. The ideal target for a historian is a process plant where you want to do trending on a massive scale. With collectors you can frequently tell it to essentially just "log everything" and sort through the data later. Therein lies the problem...how to get information out of so much data. They have "calculators" which scan through the data and calculate sums, averages, etc. They also have various interface programs for showing trend lines, browsing data, dumping results to Excel, etc. Some historians that are on the market include Wonderware InSQL, PI, AB PlantMetrics, and GE Fanuc's iHistorian (the old Intellution historian broken out of iFix). There are a couple things to keep in mind. They will NOT interface with any other plant data. No matter what they say, they are NOT SQL compatible except in that they frequently sort of shoe horn an interface into them. You CANNOT log data or merge it with other data. Even compared to SQL, I always feel like I'm working at "arm's length" with the data. Any time I want to do something that they didn't specifically design for, it turns out to be a lot of work. It is bad with the SQL loggers. It's even worse with historians because they maintain control over the data and they limit your interfaces and capabilities. If you just want to try one, GE Fanuc will let you run iHistorian with either a 10 or 100 point limit on an unlimited license forever for free (no strings attached). I just don't remember how many points(tags, whatever you call them) the free one gets you. I really do NOT recommend taking the recommended approach of "log everything". This works well in a process-oriented situation but not for discrete or batch manufacturing. In those plants, what I find works best is to do a little bit of process-oriented logging for things like tank levels or plant air pressure. Outside of that, it mostly seems to work best to dump one record containing all relevant parameters at a time at the end of a batch or beginning of the next one. Simply counting the number of records gives you production totals and rates. You can still do averages, totals, efficiencies, etc., based on this data. And the data takes up far less space and you don't need calculators to convert the data to usable form. Just use triggered logging to log the record. On top of that, there's MES (Manufacturing Execution System). You basically configure the parameters of the model built into the MES and then it will automatically calculate things like production rates, downtimes, KPI's, efficiency metrics, etc. It is truly a nice system because 99% of the things that people want data loggers for outside of the process industry (and frequently even within process industry plants) is already prebuilt. They tend to be extremely expensive ($20K+). They also require a Historian and won't generally play well with others. MES systems include GE Plant Metrics, PI (hard to figure out the names of the packages and their salesmen never call me back), and AB PlantMetrics. For what it does and the cost, it is usually much less expensive and just about as time consuming to roll your own, however. So unless you are working in an operation where $50K for a software package is no big deal, it is very hard to justify buying vs. building your own. If I can convince you that SQL is the way to go, please consider mySQL before messing with the alternatives (Microsoft SQL, IBM DB, Oracle). The price can't be beat (free), and with the last major release, it has all the same capabilities and features of the competition. They've fixed 99.9% of the bugs, and the speed, the speed. You get answers to queries in fractions of a second. If you are manipulating say 10-1000 data points, you can easily do this in Excel. Once you get into "buckets" of data (thousands of data points), Excel very quickly becomes unwieldy. Heck, it can only do 30,000 rows which is a "small" table in the SQL world. You need to understand SQL and you are going to have to think about what kind of front end you want to use. The two hurdles with SQL is that thinking in database terms is sort of "inverted" compared to traditional programming. It's called "programming in the large" because instead of say adding up all the rows one at a time, you say "sum this set" and add various filters to define what that set of data is. I always think of "programming in the small" as building something brick-by-brick. Programming "in the large" is more like slicing and dicing to whittle it down to what you want. With Excel I could find out the average plant air pressure over the past month while logging once per second after making about 80 work sheets or writing a VBA macro to run the calculation. Even if I logged once per minute, I'd still be looking at 2 worksheets even to get all the data in one place. In SQL, at that rate of data collection, the result would come back in about 1-2 seconds at most. With a historian, the result would come back in about 0.1-0.2 seconds at most. That gives you an idea of the comparisons. Since it's a completely different interface, you've also got to think about the front end. Excel is one such front end. Access is another (just set the SQL database up as a remote database). Many HMI's offer some sort of SQL database. There are lots of "report" programs on the internet. A canned one for sale is Crystal Reports. Inductive Automation's FactoryHMI is an HMI that is 100% SQL based. Every major programming language interfaces to it. If you want a canned one, GE's Proficy Portal is another. There are plenty of trending programs and/or add-ons that will talk SQL. Even better, in the system I have running, there are already several IT systems that also use SQL. My data logger populates some of their data. At the same time, I'm using many of their tables for "standards", production schedules, etc. The only real problem we have at this point is that if either one of us makes a change, everything is so intermixed that we have to be careful that we don't break some other part of the system accidentally. We've largely overcome this by defining views on the data and always accessing the other guy's data via a view. This layer of insulation makes it easier to maintain everything going forward but allows flexibility in changing the underlying data structures. Just be careful not to use "*" (wildcards). What I can recommend is that you sit down with the folks involved and write up something to start with. Say a sketch of a report, a trend chart, etc. Come up with how far back you realistically want to look in terms of history. This gives you a place to start. Build/design a system to implement those exact functions. Everything will naturally expand from there. Involve multiple people in implementation if possible because these things tend to get huge quickly when it starts to take off. The best ones to involve are the "analysts" that already do a lot of data manipulation. If AB PlantMetrics happens to fit your data model, great. Download and read the manual carefully. You will have to decide for yourself if it is worth the money or an extra week of your time to develop exactly the same thing minus the extras you don't really need for less money. Or if their model even comes close to fitting your plant. Also consider the fact that every user has to have a license with that one. Every MES system has that same blossoming costs issue because the market is small and because they are selling you on doing a lot of up front work. And remember that any metric or plant process that does NOT fit the model is simply not going to work with the MES system. In my case, I used their system as a model for how to do certain things but we went and developed our own "model". It is simply a couple dozen SQL queries and nothing more. The underlying data logger is a simple one that triggers either periodically or based off a machine cycle (like a batch/cycle counter). One of the big tricks that you will have to overcome is converting things like "machine running" or "downtime" into mechanical rules that make it possible to generate reports. People frequently "know it when they see it". Usually if you start with something simple, it doesn't take too long before people point out the flaws. Then add additional rules (the minimum possible) to cover the exceptions and nothing more. Before long, you will have a good working system.

[Ken Moore 23]

Not readily available, here's a link to the website: http://www.osisoft.com/ We currently collect data via RSLinx OEM (OPC) for our AB processors, and use a Kepware OPC server for most of the other brands of PLC's. We are using an API application for our DCS interface. You can also connect to any OBDC compliant database, we're using this feature to collect QC data. For batching operations you can configure batch "triggers" so that batch start/stop times can be collected, then trends of the same products can be overlayed to detect problems. A starter system would run somewhere in the 50-100k range.

[TimWilborne 108]

Thanks, I might be PMing you for some advice in the near future

[Ken Moore 23]

No problem, glad to help.

[Stupidav 1]

Thanks to all of you, in particular paulengr. Going by the list that we have developed, automatically calculate things like production rates, downtimes, KPI's, efficiency metrics, etc, I believe we are looking at a MES system.

[entertron 0]

We will be releasing a newsletter at the end of the month on a software product from one of our customers that deals with event capture software relating to the factory floor. I believe this is what you are looking for. If you email me, I will send you the press release for the product in advance of the newsletter release.

[Nathan 7]

Paul - I must have missed this post. It sounds like an ideal application for Inductive Automation software. As usual, your advice is sound - I only offer a few points to fill in some gaps. FactorySQL can work as a simple drag and drop datalogger, or be used as a tool to work into "historian" and "MES" type applications. The addition of "Historical" SQLTags should bring FactoryPMI on par with any other historian (as a "historian") out of the box.

[Mike Atwell 0]

Have you heard of the I/Gear Data Transport Utility? It can be used to read data from different types of processors (via a third party OPC Server like Kepware) and log it to any destination (SQL Server, Excel, CSV file, etc.). It can also manage notification of production events (downtime, out of range values, alarms, etc.) through e-mail and others that you configure. Do you have some specific examples of what sort of data you want to collect? Also, as a matter of disclosure, I work at I/Gear.

[paulengr 44]

I have set one up 3 different times for 3 different companies. Each time was more successful than the last because I applied the mistakes that we made early on to each one. Before you dive into an MES system, there are a couple things you need to understand ahead of time: 1. It turns out that most plants really only use a few metrics. But these tend to be very plant specific. The only one that is "generic" in any sense is OEE. Even if you don't call it that, it's the one that you always eventually end up with. 2. "True" MES systems require a LOT of up front work configuring their model to work with your system. Usually, you will find that you will have to have some special PLC routines to support it, and some special programming in both the front ends and often some SQL programs (stored procedures and views) as well. In the end, MES systems make things very easy but I always end up finding limitations that make it much harder to do something. All the development time that I saved from the MES program gets spent on working around it's limitations. So I stopped using them. Same with the "true" data historians. 3. Download the manuals for PlantMetrics from AB. They are free. You can also download the manuals for Fanuc's Plant Applications. Read them thoroughly. While you are reading it, understand what they are and what they do thoroughly. With both of them, I believe you can get "demo CD's" that are actual running copies. The best way to try them is to set up an experimental system. This may take up to a week or two to do the first time. By the time you get done with this, you'll understand what it is they do and you can know whether or not the price justifies the time savings (if any). 4. The achille's heel with a lot of this stuff is really the front end interfacing. In this area, there are three routes I know of. As long as it's just trend lines, most HMI's do a fantastic job of it. You can do production rates. KPI's, downtimes, etc., don't really work out. Trending systems work best for continuous processes like chemical plants that produce stuff and not individual things. The other metrics work better on things rather than stuff. If you need things, then take a good look at FactoryPMI. Heck, download it and set it up and run it, too. Also take a look at Proficy Portal from GE. This gives you a great front end and you supply the SQL programming (the "MES" side of things) within it's limitations. If you're not afraid of programming, another approach would be to develop your reports in Netbeans or in a reporting program such as Crystal Reports or it's free equivalent (Japer Reports). BIRT is also a consideration but I think it's still under development. 5. One of the problems with MES in general is that when it comes to for instance downtime, most people "know it when they see it" but can't really identify what is downtime and what isn't. You are going to spend more time with arguments about definitions than any other problem. It helps to have a very mechanical definition that you can program to. With a machine that is either in automatic or it's not, this is very easy. If you are trying to watch an operator in a semi-automatic system, this is much harder. Usually a simple definition like "machine doesn't cycle after 5 minutes" works. If you are not a 24/7 operation, you will also need to figure out how to figure in shifts, overtime, etc. Either someone has to be willing to type in the data so that the calculations can be done (many plants don't have the discipline for this) or else make some more guesses and assumptions. You will also run into problems getting people to identify downtime reasons, defects, etc. This stuff usually takes several iterations over a period of a few years to get it working well. It takes years to get this stuff truly off and running, and several iterations before you get good results.

[kaiser_will 65]

Some design constraints to consider when designing a data collection system: 1. Since this is a database, who will be the "dad"? Too often projects lead to data collection databases that IT or Engineering "inherit" from project manangers, and the integrator finds out down the road that these people have a preferred database system that your application should interface with. Sometimes it is as easy as having a new table added to an existing database to pull in all your tags. 2. What is the desired use for the data? Today, most customers want automated shift report generation. If that is the case, then the scope of your solution might be...collect event-driven data from the control system, produce shift reports (automated emails of shift reports to a user list?), archive shift report totals, build metrics from past report performance (i.e., average cycle time, etc.), provide customer access to customize reports or build new reports. The typical bottom-line goal is to produce reports with binary metrics (i.e., Line 1 is producing at or above goal, etc.) that quickly identifies shifts (i.e., increased setup time due to source part design changes). 3. How much data will your system produce? The number of tags, timed interval of tags, and how long you must keep the data all lead to how big the database (and the license) will be. Too often customers ask us to capture every darned tag with ms intervals only to find nobody has the facilities or understanding to interpret the data. All managers know they need data but many have little or no concept of what the heck to do with the mountain of information. 4. Does the solution need to be modified in the future? Such as to add more machines, new or existing, mofidy email user lists, modify shift report metrics (add new calculations). 5. Take into consideration support (who will help the customer when the system fails) and maintenance (what steps are needed to archive data, how should the data be archived, is old data even needed if shift reports are archived) for your solution. 6. How will the data collection system communicate with field devices? Possibly, your collection system will need to pull Ethernet/TCP-IP, DH+, Serial/DF1, etc., data. The data server PC might need special or duplicate communication cards. New hardware and software solutions are continually coming to market all the time that are addressing the roadblocks we have all had to endure, such as how to capture small amounts of data without having to fork out huge $ for SQL server licenses.