jzoller

There's nothing new here, but maybe someone will be kind enough to lend me their experience. I need to store data taken at a series of points (>100k) over time (minutes to months). At each point in time, many (1000+) parameters will be tested, and each test will generate a single result (string, bool, dbl, etc). All of this data will be associated with some identifying and meta data (time stamp, serial number, etc). Not every parameter will be tested at every point in time, leading to some sparse result sets. In the past, I've taken all of the data for a time point as a "row", with each test result as a single column. That row was written as a string to a tab-delimited file as every time point test completes. I'll skip all the problems this can lead to, but too much of my life was spent on fixing them. So, I have a chance to start from scratch. I've been looking at a few options, and there seem to be issues on all of them: 1. NI's TDMS: doesn't seem to understand the concept of a row, only a column. Not expandable in ways that I'd like. 2. Database: ideal, but I've many non-networked computers... anyone know a good sneakernet DB? 3. Markup (XML and ini, for instance): poor scalability & poor LV tool support... anyone have an LV native SAX parser? Of course the database folks want future compatibility for import into the grand plan, the engineers want to use Excel (sigh...), the web folks want to be able to pull reports in .NET/PHP, and I want to be able to add or remove tests any time I'd like. What have you all done with your data systems? Thanks, Joe Z. (My budget for this is currently whatever I have in my cubicle, pockets, and/or can sneak out of IS after midnight, so large scale proprietary systems are out. Please, feel free to respond here if you have a commercial system, but no emails.)

QUOTE (Aristos Queue @ Mar 31 2008, 12:11 PM) Aristos, can you spare any comments on how LV deals with http://en.wikipedia.org/wiki/Memory_fragmentation' rel='nofollow' target="_blank">memory fragmentation ? Joe Z. (Edit: specifically, is external fragmentation an issue... you covered internal above)

QUOTE (jdunham @ Mar 11 2008, 11:17 AM) I agree, the hotspots seem a little too sensitive at times, especially as screen resolution climbs (and my eyes age!). I also find the lack of wiring hotspots on large block diagram structures a bit of a hinderance with the autotool. Joe Z.

QUOTE(Gary Rubin @ Mar 6 2008, 10:12 AM) ... I admit it, I did this last time I was in an Apple store . The saleswoman actually ran away. JZ

In a more useful vein, here's Adam Kemp's reply on the same subject from Info-LabVIEW: "It may be annoying, but it's not unheard of either. Most programs need some kind of runtime code to work. Native applications need runtime DLLs that happen to come with the OS, so you don't notice. Programs written in other languages like VB need runtime DLLs that happen to be very small. Then you have programs like Java and .Net which need bigger runtime components. Java comes in around 10-20MB and .Net is about 20-30MB. Depending on your OS version you may or may not already have these components installed, but that didn't use to be the case. If you think about file size compared to the size of drives and typical bandwidth for network connections, then LabVIEW is doing fine compared to what Java users had to go through in the 1990s. The JRE took about 30 minutes (or more) to download back then, and it seemed like every program written in Java came with a different version of the JRE. Fortunately Java has settled down a bit since then, and people can mostly get away with relying on the JRE pre-installed on most systems. LabVIEW can't get away with that (it's probably not going to be pre-installed on systems any time soon). It's unfortunate, but realistically it's not very different from the other software tools. We just don't have the benefit of coming pre-installed like they do. Maybe some day. :)"

QUOTE(brent99 @ Feb 22 2008, 12:21 PM) Dead horse: yes. Still true (and annoying): yes. JZ

QUOTE(rolfk @ Feb 22 2008, 12:21 AM) For a more explicit description of this: If your SSP expired after 8.20 but before 8.2.1, you were not entitled to receive the 8.2.1 bugfixes. I have no idea if this will remain the same for 8.5.1. This situation convinced my boss to stay at 7.1 rather than making the planned 8.2.1 upgrade, and now that we're off the upgrade trail, frankly, we're unlikely to upgrade again until things become so broken that we have no other choice. :headbang: Joe Z.

QUOTE(cmay @ Feb 19 2008, 01:41 PM) You're running a little tight on memory. If the GUI is large, it might be causing some page file swaps that are slowing down disk access on the loads. Just a thought. Hard to do more without concrete details. Joe Z.

Wow, that takes me back. It looks like my early programs... it's just lacking a 12 level stacked sequence structure! Step away from the LabVIEW for a little bit. Go read http://zone.ni.com/devzone/cda/tut/p/id/4434 and it's associated links. (Yes, it is god-awfully, mind-bogglingly boring. And necessary.) Then, go to the LAVA code repository and OpenG and look at some code. Don't worry about what the code there is doing. Just look at how it's written. Now, go back to your VI and re-arrange it so that it looks as much like the above as possible. The migraine will recede soon thereafter. Honest. Joe Z.

How about serializing the state of the controls on each tab to a scratch file? You could use something similar to the OpenG Read Panel from INI/Write Panel to INI (or XML or whatever) when you detect a tab change. If you don't store the state of the panel between program modifications, there shouldn't be any problems with mis-naming or bad ordering. Make sure you don't run into file write permissions problems if you do it this way! JZ

I have to agree with PatNN on this one. That would be a feature I would use. On a more general note, workarounds != native support, and should never be used as an excuse for not providing native support for good features. Joe Z.

QUOTE(Aristos Queue @ Jan 29 2008, 10:29 AM) Unless you're http://en.wikipedia.org/wiki/Python_%28programming_language%29#Syntax_and_semantics' target="_blank">Guido van Rossum JZ

QUOTE(Big Hank @ Jan 24 2008, 12:57 PM) Gah, zombie thread! Besides, I'd be rather surprised if the .NET runtime was cleaning up my LV references... JZ

Jim, I couldn't agree more. From my experience, the internet toolkit XML vi's are hugely inconsistent in their interface and the native LV schema is brittle to the point of being useless. I ended up writing a private API based around the MSXML support for XPATH queries, but even it suffers from versioning problems on different machines. I will admit, I didn't investigate the LabXML interface too closely because of its dependency on yet another library (Gnome's libXML2). Joe Z.

QUOTE(TG @ Jan 15 2008, 11:23 AM) Buy a Wii? http://www.cs.cmu.edu/~johnny/projects/wii/ , scroll to the bottom project. (Apologies if you've seen it before...) Joe Z.

Be sure to match to your specs. If you're going to have 5 barcodes on the label, you will probably want to read them all and differentiate, rather than trying to line up the correct one. I've used Symbol wedge scanners as well. They work well for single code scan. In the past, I've looked at a low end machine vision system from Cognex (this one) that would allow me to scan anything sales manages to come up with... The system seemed solid, though it was more expensive and only ActiveX interface. Joe Z.

I'd chime in, but I stopped using it so long ago that I don't remember most of the problems anymore. Joe Z.

Competition. That is all. Joe Z.

QUOTE(Justin Goeres @ Oct 5 2007, 01:51 PM) I haven't yet used it, but Beyond Compare has a plugin for binary files: http://scootersoftware.com/moreinfo.php?c=hexview Given the quality of the rest of their software, I suspect it's pretty good. Joe Z.

Thanks Jim, re-rereading it, the sense of it penetrated. JZ

QUOTE(Aristos Queue @ Oct 3 2007, 09:11 AM) Obviously, the universe needs operator overloading. Jim, I'm sorry, on the line that says "You should organize your classes on disk", should it say, "You should not organize your classes on disk"? Thank you, Joe Z.

The license server should (it used to anyway) be able to create "detached" licenses for just this purpose.

Another simple method is to grab a timestamp whenever the value change event fires, and toss it into a shift register. The next time the event fires, if the new timestamp isn't greater than the old timestamp plus some delay (100ms, for instance), just route the event into an empty case statement and do nothing else. It will waste some cycles, but, for user interfaces, that is usually invisible. For reference, if you tinker with the slider event change, you can measure how fast LV grabs events. On my computer in a simple test VI, LV can grab a timestamp every 15-16ms, but it fires value change events about twice that fast. Joe Z.

Thanks for the replies all, keep them coming! I went ahead and picked up a copy of Godel, Escher, Bach, and ran across an old friend: Code Complete by McConnell. One of the best practical books on software construction available. Joe Z.