Ethernet is Fun......Not

[enigma 0]

You guys have been helpful in the past and I have an issue that has come to light today that has me burning the midnight oil. I have a canning line that has 3 SLC505's controlling the conveying lines. I didn't do the install but the guy before me liked to play with ethernet communication and I believe got carried away.These processors communicate with each other via "read" messaging. No "writes" Anyway the company hooked up a data collection system that is on their business ethernet and communicates with the plant ethernet through a gateway address. Now since they did this we are getting random communication errors "634" on the main panelview. The panelview communicates with all 3 PLC's. When they disconnected the data collection system the problems seem to have gone away. Question: Is it possible that the data collection system was slowing down the ethernet causing the communications to fault??? Is there a way to visually (thru a program) see what's happening??? Edited 1 Dec 2007 by enigma

[BaxtersDad 0]

How are the 5-05's interconnected, CAT-5 to a hub or switch? A good switch would possibly help. Sounds as if your collection software may be polling way too much, as they are want to do... Just a thought.

[mracer 2]

This brings up a question that I have. What is a good switch to be using? I have looked at the B&B switch's. Are there others? Racer

[Nathan 7]

Baxters brings up 2 relevant, but separate issues. 1. Hub versus switch. The hub sends all signals over all ports. It's a multiport repeater. Switch does not. 2. "Good" switches. I like Cisco equipment. Good switches should be pretty much commoditized by now. Ask the industrial experts for "good" harsh environment ones.

[enigma 0]

Lot's of infor to digest. The guy used off the shelf Linksys switches, The kind you buy in Futureshop. I have been using a switch by MOXA in the new installs. It's industrial strength. He is using 4- 5 msg reads per plc and using not bit 15 of the msg command to trigger the read. Each control block length is 51.

[buster 0]

bit 15 is the message enable bit. as soon as the message is done it resends again and again

[Stupidav 1]

There are several possibilities, I doubt that the problem is with the switches. I lean towards the "Data Collection System", or the program communication coding. Many older and some custom Data Collection Systems used DDE to collect the data, which can cause a heavy load on the network, vs. OPC systems. The other thing is what type of PanelView is it, and what is the reading and writing setup. You mention that they are communicating by the only time that the enable would not be true is during the first scan of the program. If you have a copy of one of the programs that would shed a lot of light on your the situation.You are better off with "Not" bit 13 the DN (done) to control the communication. When you are having this issue, check the status of bit 12, my guess is that the BTRs and BTWs (or equivelant) are erroring quite regularly, which is highly possible if there is a high load on the network because it will send and attempt to resend again regaurdless of whether it is done or not. So by changing them to the done bit at least the messages will complete fully before the next attempt. Also with 4-5 BTRs and/or BTWs, each one should occur and complete before the next is started. If by chance bit 12 is a 0 all the time, then you can try increasing the CTS Time out in the communication settings on the PanelView. Although this is most likely just a work around, not a proper fix.

[paulengr 44]

The following is fairly long winded but here is the short version of what you can do: 1. Yes, you can monitor communication, in a "gross" sort of way. Just set up the channel status information on the PLC. You are kind of limited though on what you can do with an HMI. 2. Get rid of all CAT-3 and all hubs, if any, as a first step. 3. Check for corrupted packet errors and fix bad connections, cables, and switches, first. 4. Put all the HMI's and the PLC's on their own separate subnet so that you don't have cross-traffic issues. First off, the "all reads" is usually a good thing, with one small exception. There are roughly 3 AB Ethernet protocols: PCCC, CSP, and CIP. SLC's mostly implement the first two which are essentially variants on DF-1, the same protocol used by AB PLC-3, PLC-5, and SLC through their serial ports and over DH-485 and DH+. CIP (Ethernet/IP) is significantly different and has at least two different flavors of the basic protocol. In any case, the first packet is sent from the PLC that has the MSG instruction. It contains the command. The recipient will respond with either a response including data (for a READ) or else simply a "go ahead" response. For a write, a third packet (or packets) contains the actual data. There lies the rub...3 packets for a write, only 2 for a read. So most of the time for efficiency reasons, reads are best. A second reason for using reads is that it is easy to monitor the status from the receiving PLC via the MSG's status bits (.ER or .DN). A third reason is when it comes to programming, writes insert data into the memory of a receiving PLC. There is no corresponding documentation unless you create some on the recipient. So this can be very confusing to troubleshoot since you have to check ALL PLC's to see where the writes, if any, are going. There is one exception. When you do reads, the only effective way to check if some data on another PLC has changed is by polling it...so you set it up to constantly read a location, whether a change has actually occurred or not. With a write, the PLC that has the data controls when or if it transmits an update. For instance, if you have 3 production lines all connected to a merge point and those 3 PLC's need to monitor data from the merge point (such as a common scale shared by all 3 production lines), it will be more efficient for the merge point PLC (running a scale) to WRITE updates to the appropriate PLC rather than having all 3 constantly polling. Now, as to looking for what's happening...use those status bits! For instance, whenever you START a MSG (by whatever means triggers it), you can also trigger a high resolution timer with a long enough time limit that it never times out and copy the timer over to a buffer (trigger off the .DN bit to write the .ACC of the timer to a separate integer for viewing) so that you can monitor how long a MSG is taking. Second thing you can do is to set the number of retries to 1 and set up a counter that triggers off the .ER bit to record how many/how often packets fail. Third thing you can do, although the granularity sucks, is to make sure to set up status files on your communication channel and look through the Ethernet data to see if there are any problems. If you see any errors such as corrupted packets, collisions, etc., then you have a network problem that needs to be investigated and fixed first before proceeding. Although errors can and do occasionally happen, if you are seeing any collisions, errors, etc., in your Ethernet counters, you've got a physical/network problem that needs to be addressed first. With Ethernet, there are generally two problems you will run into: network problems, and bandwidth problems. A network problem would be for instance using CAT-3 or using hubs instead of switches. Both will cause network errors and generally limit you to about 10% of full bandwidth. Hubs are even worse because they tend to propagate bad packets all over the network. Another problem would be bad wiring or bad switches. As a first step if you haven't already, go through your entire plant and remove and destroy all CAT-3 cable. Remove and throw away all hubs, replacing them with switches. Yes, the B&B Electronics ones are fine, even the el cheapo dumb switches. I have several and they've worked flawlessly for a couple years now. The reason that Ethernet problems are so bad is because of the nature of Ethernet, and more importantly, TCP/IP. Ethernet's backoff algorithm which deals with collisions is somewhat flawed. It will limit your bandwidth to about 10-20% of full bandwidth and create very large delays and major jitter problems in packet transfers. TCP is even worse. Since these days, error packets are very rare, TCP actually uses packet dropping as a signalling method for measuring bandwidth. If TCP sees dropped packets, it automatically assumes that there's a bandwidth problem and will slow down communications, many times to 25-50% of full rate. In a "packet storm" situation, TCP has been measured at 1-10% of bandwidth even though the physical network was nowhere near that choked off. See the old Van Jacobsen papers (google his name) if you don't believe me. With modern TCP implementations, the situation has improved but you still won't get to the maximum theoretical bandwidth of 35% of capacity on a hub-based system. A bandwidth problem is somewhat more difficult to deal with. Especially PLC-5's and SLC's have limited communication processing capability. They can easily generate more MSG commands than they can deal with. They can also receive more packets than they can physically process...leading to various glitches and bugs to track down. Some of the older PLC's will actually outright crash if you hammer them with packets. At least the newer firmware is equipped with "broadcast storm" protection that will essentially just start throwing packets away randomly if this actually happens. In terms of MSG instructions on a given processor, carefully make sure that you never have more than 4-5 of them active at a given time on a given channel. If you have more than that, either use timers or the status bits to ensure that no more than that trigger at once. There is buffer space for roughly 10 MSG instructions per channel on those PLC's and if you exceed it, the PLC will start dropping packets on you. Second, be careful not to hammer it on the receiving end either. I don't recall the exact limits but if you are forcing it to process more than about 50 packets per second, you are in for a world of trouble. Carefully look at the polling rates of both your SCADA software and the inter-PLC communication. If you are getting over 50 packets per second, figure out ways to reduce the traffic. SCADA systems by the way use the exact same 2 or 3 packet READ/WRITE MSG instructions that inter-PLC communication uses. It's just a question of whether they optimize that traffic (does it combine reads to locations N7:0, N7:1, N7:2 into a single read or issue 3 separate ones?) or not. You will be well served by optimizing your own MSG instructions as well. If you can't easily dig out this data, you can use the channel data to figure it out. Just use a stopwatch and the status display on the Ethernet channel. Reset the counters and time for 1 minute. Check how many packets the PLC received in that time period and divide by 60 to get the packets per second. Generally speaking there's nothing wrong with Ethernet (actually, there are a lot of things that are right with it). It's just that when faced with the idea of a 10/100 Mbps raw bandwidth, many people assume that they have virtually unlimited bandwidth and start doing crazy things like setting up their HMI's with 10 ms (or less) polling rates. This spells trouble for both the PLC and the HMI. Finally, you can have a simple "cross traffic" issue. This is a common problem with "general" networks as you described. IT departments normally set up the network as a "data" network...it's all "best effort". All of the bandwidth is allocated on a first-come, first-served basis. So if all you have is a bunch of PLC's doing low-bandwidth inter-PLC communication and then Fred over in accounting starts doing a huge download for end-of-month, what happens? Essentially the PLC traffic grinds to a halt since Fred sucks down almost all bandwidth. A second issue is if things aren't set up right, "broadcast" traffic can eventually cause similar problems. The solution to this is to recognize that PLC's and their demands are fundamentally different from general "data" networks. The first step is as a minimum to make sure to install switches in such a way that the PLC's are on an isolated network. The way that a switch works is that when it first starts, it acts essentially like a hub. Any packet received on a port is broadcasted on ALL other ports. But, it also monitors the MAC (Ethernet) addresses to the attached devices. Once it recognizes where the devices are at, when it receives a non-broadcast packet intended for a specific destination address, it will send the packet out the correct port and only that port. If the switch is part of the general data network, then the PLC traffic is contending with everything else. But if you have all the PLC's off on their own little subnet, chances are that the switches themselves (even "dumb" switches) will take care of the problem. If this doesn't help, then you need to have separate physical networks connected by a router, or set up VLAN's, or set up QoS (priority queueing). These last two more advanced methods require managed switches but allow you to properly allocate bandwidth and resources over shared data links. In my plant, right now at least (no Ethernet/IP to deal with yet), I have all the HMI's and PLC's (and I/O if it's Ethernet) on isolated subnets, isolated by simple dumb switches. That way if IT does something goofy or not too bright like just rebooting a switch in the middle of a production run (hey, those users will never notice!), at least the PLC, HMI, and I/O doesn't crap out suddenly without explanation. At worst, it becomes an isolated island for a few minutes. As to monitoring traffic to/from an HMI, this is kind of tricky to do because the HMI doesn't usually give you a whole lot of information to work from. Generally speaking, the first thing to do is to acquire a low cost managed switch. I'm partial to Hirschmann myself (they have a few useful features that Cisco doesn't and are intended for the industrial environment), but I'd suggest you get at least one managed switch strictly for test purposes. Put it inline between the HMI and the network. You can monitor the traffic to/from the HMI directly from the switch to find out some basic information. If this isn't enough, download a copy of Wire Shark, put the switch inline between the PLC and the network, and turn on port mirroring in the switch. Now you can monitor ALL the traffic and drill down to discover what the problem is. It's not exactly the prettiest way to do it but it's effective. If all this sounds complicated, go to ctrlink's "Ethernet University" web site. Also head over to the ODVA's web site. Download and read every shred of information you can get your hands on. For explanations of CSP and PCCC, check out Lynn Johnson's protocol web site. He has spent more time than anyone I've met decoding the sticky details of PLC Ethernet protocols. Get a copy of the DF-1 protocol description, too. I know it has nothing at all to do with "Ethernet" but it does. Both CSP and PCCC and merely modified versions of DF-1. Understanding DF-1 leads to understanding PLC-5 and SLC Ethernet protocols. You can also look at Ron Gage's web site. He has complete source code (very readable) for doing PLC-5 and SLC reads and writes from a PC. With a copy of the DF-1 protocol manual in hand, it's pretty easy to understand the finer details of how the actual wire protocol works. The rest of the non-PLC specific stuff you'll get from the other sources I mentioned on Ethernet in general.

[Ken Roach 194]

I am flat-out impressed. Excellent treatise on Ethernet from an A-B PLC/SLC messaging perspective, and then some.

[Nathan 7]

Paul - kudos to your explanation. I don't intend on knocking it because it was very good. However, a few comments: 0. You are dead on in re-iterating the kill all the hubs - real world, that's a major problem that many of our readers here face. Replace Ethernet hubs with switches. 1. In #4 of your summary replace "subnet" with VLAN, if possible. Many of the problems that you've described (such as collisions) are a layer 2 problem. Subnetting is an IP feature (layer 3). VLANS also separate broadcast domains, one of the problems in your example, which are different from collision domains. 2. I wouldn't say that Ethernet is flawed based on the backoff algorithm. It has to work like that because you can't send two signals over the same wire. Using a modified exponential backoff has proven to be statistically "less chatty" than other models. The problem is that a hub has a single collision domain. You nailed the solution - use a switch. (I didn't read the paper - maybe I could be convinced otherwise on the "flaw", but we both agree what the practical user should take away from this...#0 if you still don't get it!) 3. Great comments related to limited PLC bandwidth and networking processing capabilities. Keep in mind that concurrent HMI/SCADA client polling affects the PLC in a similar way as dropping the update rate for a single node. In other words, the more concurrent clients you have, the more necessary it becomes to have a client/server model where a single node communicates with the PLC and aggregates the comm path, acting as a cache in the computer science sense. Direct writes aren't so bad - concurrent polling can kill. 4. Hopefully the Fred from accounting example has some, but limited merit on our modern systems. Specifically, if you know what to ask IT can help you resolve the problem. (We're talking infrastructure bandwidth). With a hub you're hosed - what can I say? If you're all plugged in to the same switch, Fred shouldn't be able to hurt you. As fast as he can download, he can't tax the switch fabric, which should be able to handle on the order of 10-100+ gigabits. Modern switches will do many simultaneous point to point 100 meg connections without breaking a sweat or pestering the other connections. However, if you have separate switches with slow uplinks between them (relative to Fred's download power, which is probably on the order of 20 megs/sec), he can take you down. Don't forget Fred's buddies if you have a remote server room that houses plant and corporate machines - that's where QoS needs to come into play. This is why the salty integrators so adamantly demand that Plant and IT networks are physically separated. Great Lesson! Edited 3 Dec 2007 by Nathan

[enigma 0]

Great stuff Guys and especially Paul. I will be going in today to see if I can detect what's happening. Everything is CAT-5. The switches are Linksys. The same ones you use at home. I know, don't ask. The data collection software is Factorypulse @ www.factoryware.com As long as they don't put the data collection system on line, there will be no problems as the line has run this way for years. I'll let you know if I find anything. Edited 3 Dec 2007 by enigma

[enigma 0]

Futher Update: The Data collection system is polling the plc every 100 ms.

[enigma 0]

Paul Another point that you brought up was isolated networks. The way IT attached was through the plant network switch using a gateway address. there is no physical separation such as a router.

[enigma 0]

Ok Some further info. 3 plcs 505's. 2 of these have 3 msg controls each, read only, contiuously updating. One has 5 msg controls read only, continuously updating. The factory pulse polls an integer file (Can Count) in the plc with 5 msg controls. which causes communication errors on that line when it is connected. The factory pulse polls another production line also but that plc has one msg control. It also has a HMI on the ethernet. There are no communication errors. Will one the following solutions reduce some of the over head??? Original Solution One Solution two Edited 4 Dec 2007 by enigma

[paulengr 44]

That my friend is probably your issue. Do you really need to poll that much? If you are polling a single data point, you're hitting the PLC with 10 packets per second. If you are polling 10 points and the HMI is generating 10 individual packets (if it doesn't know how to, or can't aggregate reads together), it will be hitting your PLC with 100 packets per second. And if you're like anybody else, you are probably monitoring much more than just 10 data points. Same concern about the HMI. I know it's nice to have "quick" response times but there are limits on how far you can or want to push this. The easiest way to decide on "proper" polling rates for SCADA systems is with some simple practical knowledge. The fastest human reaction rate is about 300-350 ms. So if a "push button" control operates at around 250 ms, it is plenty faster than the human reaction time anyways. When you consider a temperature reading, it is very unusual to measure changes faster than about once per second. With tank levels, frequently rates on the order of seconds to minutes is plenty fast. So use these types of ideas to set your polling rates to reasonable values. If you really think you need sub-second polling rates, often it is better to use a timer within the PLC to actually measure the time length of the event and to poll that from your SCADA system at a much slower rate (say once per second) rather than the raw value. ...remember, I said earlier that once you get into the hundreds of packets per second territory, you will start to experience various problems with the PLC. The issue isn't raw network bandwidth. It's limits on the ability of the processor to read and respond to the packets. With Controllogix Ethernet/IP I/O, the stuff is frequently listed with capabilities around 1000-1500 packets per second MAXIMUM. But it is based on newer platforms and capable of much higher throughputs than PLC-5 and SLC Ethernet interfaces. I don't know the exact packet per second limits, but I'd be very concerned about hundreds of packets per second rates. Edited 4 Dec 2007 by paulengr

[enigma 0]

I'd like to thank Paul and the rest that helped out. I have copied this thread for future use as there is so much good info here. I had the factorypulse increase the polling time from every 100ms to every 1 sec. We have run 4 shifts with no problems at all as far as communications go. I will still have to address the way the msgs are sent to reduce the overhead on the PLC as well as look at the HMI polling time.

[Ghettofreeryder 0]

I had a similar experience with 2 PLC5s talking DH+.. The programmer before I had both plcs reading, or writing, i cant remember, to one another. Once there was a significant amount of msgs being used, we ran into the same problem. To help alleviate some of the issue, we triggered the messages at certain intervals. We then moved all communication to one plcs, so the it writes data to the other plc, and reads data from it, with no message blocks on the other plc. This helped tremendously. I dont know if it will help with this issue, but it might be something to look at.

[enigma 0]

Unfortunetly I may have been premature. The problem has resurfaced starting on Friday last week. Disconnected the FactoryPulse and the proble is once again gone. This is a hair puller.

[Sleepy Wombat 27]

Paulengr... Kudos to you for your excellent response..

[RFurey 0]

I didn't see anyone mention TCP connections. I had a somewhat similar problem a year ago. After much anguish, I discovered that there was a relatively low number of TCP connections that my controller could handle and I exceeded it when I added an RSView32 station to the network. Removing the RSView32 station cleared the problem. Adding it back to the network caused the communication problems to resurface. Live and learn. The title to the thread is Overloaded Ethernet IP network (?) Maybe it'll help. This post from that thread sums up my ordeal. ........................ Update: (In case anyone was following this thread) After a bunch of digging, I've been informed that my processor (1769-L35E) does not have enough TCP connections for the components I've added to the system. It is my understanding that the number of TCP connections is a function of the number and type of components that are accessing the processor, and not necessarily dependent of the amount of communication activity involved. According to the literature the L35E has 32 connections. According to Tech Connect, my component/connection breakdown is as follows; Panelview Plus 600 – up to 5 connections (may be less but it is not possible to limit how many) Panelview Plus 600 - up to 5 connections Panelview Plus 600 - up to 5 connections Ethernet Adapter for remote I/O (1734-AENT) with two output module 1734-OW4, three Analog module 1734-IE2V, three TC module 1734-IT2I. – 7 connections Ethernet Adapter for remote I/O (1734-AENT) with two output module 1734-OW4, three Analog module 1734-IE2V, three TC module 1734-IT2I. – 7 connections Ethernet Adapter for remote I/O (1734-AENT) with two output module 1734-OW4, three Analog module 1734-IE2V, three TC module 1734-IT2I. – 7 connections Desktop PC running an RSView32 project – 2 connections Laptop running an RSView32 and RSLogox5000 project – 3 connections Worst case (during troubleshooting) – 32 connections available, 41 connections required. Normal operating – 39 connections required. I think this is might be a good example of the word, screwed…. I am unaware of any painless fix to this problem. Post-ordeal observations a) After reviewing this thread I notice that the term connection was used a couple of times. Unfortunately I was not savvy enough to pick up on the significance of the term. b) My conversations with Tech Connect did not zero in on the number of connections until late in the game. c) The connection limitation of the L35E processor never came up during the initial planning stages with my local supplier, though it seems like a pretty apparent bottleneck now. d) I had to make repeated calls to Tech Connect to tally up an accurate (I hope) count of the connections I need for the remote IO, Panelview Plus, RSView32 and RSLogix5000. Apparently there was no one person with all the answers and I frequently got different answers to the same question. (I usually stop when I get the same answer from two different sources) I imagine that part of the usefulness of this forum is to learn from others mistakes. Here is my contribution.

[TimWilborne 108]

Counting connections is a must on any network application. It is no different than making sure that an input module can handle the signal frequency that is coming to it, assuring you have enough memory, etc. I didn't see then or now how you can point that one at Tech Connect. It's kind of like my internet service, they have a Value, Preferred, and Premier packages which have speeds of 1.5, 5, and 12 mbps download speeds. If I have the value package can I really blame my provider because I purchased the Value package and need 3.5 mbps? Or someone trying to take a PLC with a HSC and turn it into a servo controller, that's another one that seems to get blamed on the manufacturer somehow... This doesn't sound like an issue with connections but the SLCs overall connection limitations are 1747-L551 - 32 1747-L552 - 48 1747-L553 - 64 http://literature.rockwellautomation.com/i...um011_-en-p.pdf

[Nathan 7]

Paul - somehow missed your response until today. I thought I type too much. You, my friend, take the cake. Very good points as usual - I learned something about the backoff algorithm. I do think you're still missing some of the benefit of VLANs versus separate (non-managed) switches in terms of manageability and uplink switches. With Cisco gear there's often very little difference between physical and logical setups. It really comes down to a matter of preference, and, since we both understand the main points on each side, it comes down to initial simplicity versus price and manageability - depends on your setup. I especially like your point about polling rates. It's common for Engineers to get carried away here. Guys are used to fast PLC response rates for safety, which matters. PLC writes from HMI systems nearly always happen as fast as the system can handle. Really fast polling rates for updating the screen just slam the system unnecessarily. I've found that 1/2 second is faster than you need for many applications. Expect your actual time to average half that. The same holds true of logged data. You get users who think they need to log their slow changing data every 15 seconds or faster, but then they view graphs and look at averages over a period of a week or longer. That's 40k sets of data points. A huge graph might be 1920 pixels wide. Humm...in that case every 15 minutes may do just fine. Do you think your averages and min/maxes are going to change much? I'm obviously not speaking to the case of a fast process that you may want to record like injection molding. RFurey's point about TCP connections is a good one. I thought I may have alluded to PLCs inability to deal with simulations connections, but he put it much better. Lots of separate RSanythings talking to the PLC will cause problems here. I think that you can limit the connections in RSLinx settings, though, as it should work with an aggregate pool.

[TimWilborne 108]

While I know you know the difference Nathan, for those who don't, the issue was actually a limitation of CIP connections and not of TCP connections. These terms are NOT interchangeable

[paulengr 44]

You may want to look at this: http://timepedia.org/chronoscope/ It basically takes a block of data and turns it into a binary tree. The tree averages the data at each "level" in the tree. It dynamically scales the data up and down the tree as needed to zoom in, zoom out, scroll, etc. It is very, very fast compared to most trend charts like what you are talking about, especially considering that the whole program is implemented in an interpreted language. I was thinking that an even better version would be to implement the "tree" algorithm with global temporary tables (so that they can be shared) directly within the SQL server...saving bandwidth as well.

[Nathan 7]

TW - Fair enough. I meant to take a huge step back to consider that, in my experience, PLCs suck at concurrent connections. I wasn't trying to distinguish between sessions, sockets, or application layer connections. I don't know what's going on behind the scenes with CIP and what the various PLC models can support (or PCCC). I have some research to do. Paul - Interesting concept - was especially captivating at first. That online example was pretty cool. What particularly impressed me was how slick it felt with style sheets, AJAX, whatever in the browser. I don't think it's any particular breakthrough with respect to the algorithm. The first online example had 700 data "points" and another had 18k. That's a pretty small dataSet compared to what modern HMIs should be able to show. In any event, they are probably O(n log n) with similar properties to your balanced binary tree example. I'm not really sure - the trend implementation usually deals with this. For example, FactoryPMI uses JFreeChart, which is very capable in terms of large dataSets. It also deals with scrolling/zooming very well. I doubt any major vendor wrote their own code there from scratch - I think that's a large part of what made 90s HMI graphing plugins suck so bad. The global temporary table implantation is basically a cache, right? You're trading common network memory for repeating a huge SELECT query. This could be anywhere between hugely beneficial to negligible depending on your users and setup. The real trick would be updating the calculated data set without re-running your query. If everyone's looking for the exact same info, it's easy - MySQL supports automatic cached results for the exact same query; I bet other databases do the same. But most HMI users expect a frequently updating sliding value range, say, the last 2 hours. Your cached data set will need to be able to "splice in" earlier or later data, recalculating averages. The bigger difficulty comes with the quantity of "historical data" versus your cache size. "All data last quarter" is probably too much data to be meaningful here. Large historical date range selections tend to be rarely common. Most users have many "pens" (data sets) that they expect to be able to quickly change. If you wrote a custom application to deal with this, and you knew what kind of data needed to be presented, it could be lightning fast. Could this be done well in an off the shelf, generalized manner? Maybe - it sounds intriguing. And complicated! At some point it pulls a huge circle and gets to be what a database does. As an aside, my oversimplified (crappy) example of plotting every xth point from a sorted data set is just as fast, if not faster, than yours. It has the additional benefit of being able to add data easily (without re-balancing the tree). I like your idea a lot better because it paints a more accurate picture. The main difference between our ideas here and many "slow" actual implementations is that they tend to take the easy way out by re-running the query whenever it changes (no cache). Beneficial caching could occur on the client side (not to say that server side caching couldn't be useful either...) Am I missing anything fundamental with your global temporary tables idea? Edited 23 Dec 2007 by Nathan