Jump to content

Matt W

  • Posts

  • Joined

  • Last visited

  • Days Won


Everything posted by Matt W

  1. I found some more info http://zone.ni.com/reference/en-XX/help/371361H-01/lvexcodeconcepts/array_and_string_options/ In the section Determining When to Resize Array and String Handles in Shared Libraries It sounds like you're not supposed to use NumericArrayResize for string data. I'm guessing the problem is alignment since it looks like in LV64 on windows the string size would be padded with NumericArrayResize, when it shouldn't be padded. On LV32 on windows there is no padding so I guess it wouldn't make a difference in that case (or in any case where alignment is less than or equal to 4 bytes).
  2. I think I found the answer to my second question http://zone.ni.com/reference/en-XX/help/371361H-01/lvconcepts/how_labview_stores_data_in_memory/ Sounds like with strings it is either the handle or pointer that can be null, but with arrays it is just the handle.
  3. Thanks, that helps clarify some things. I haven't seen any nulls during my basic testing, so I'll add some code to check for that. Is there a way to encourage LabVIEW to use nulls, I would like to double check that the null handling is correct. Which part of the handle can null? Only the handle, only the pointer or either? If I pass a Handle by value into a CLN can I assume that the passed in handle will not be null (otherwise I can't resize that handle, and I would have no way to return a new handle).
  4. I have a method for handling syncing with DVR classes, but it's a bit messy and I haven't used it behind checking that it seems to work. You basically wrap the class in a temporary dvr for the synced operations (It can implemented much cleaner with SEQ Classes). Just throwing it out there in case it gives someone an idea.
  5. If you make sure the indicators are updated after the benchmark (run their wires through all the remaining sequence frames) then array will win, with your current bench mark the winner will depend on which test is run first (I suspect the difference is due to pressure on the memory allocator). Personally I use also use an always copy node to copy any input data before entering it into the first sequence frame to make sure I'm not measuring a data copy (just wiring into the first frame maybe sufficient but I like being sure). That wont matter in this case since you're not modifying the U8 array. A couple side comments. A string to U8 array can be done in place so that it doesn't take any time (a u8 array and string have identical structures in memory so LabVIEW just has to reinterpret the data) You're greatly underestimating the size of the string array. Each string takes 6 bytes 4 for size 2 for data. An array of strings is an array of handles. 4 bytes (array size)+5.5 million * (4 bytes [string handle]+4bytes[string pointer]+6 bytes [the string itself] ) so a total of ~ 73.4 mebibytes. In 64bit the handles and pointers would double in size, I'm unsure if the sizes would double or not. If you avoid the string array (and it's 5.5 million allocations) things go much faster On the first run with 5.5 million elements I get For Loop: 4725 Array: 2758 Mine: 134 On the second run (where LabVIEW can reuse the large arrays allocated in the first run). For Loop: 2262 Array: 2279 Mine: 127
  6. Using DETT and a test VI LabVIEW did deallocate strings that I allocated and put into a LabVIEW allocated array. The deallocation happened before the vi terminated, since all DS* allocated data would have been cleaned up by then anyway. I'm forcing deallocation by reshaping the array. So it looks like LabVIEW will deallocate handles placed properly into aggregate data types. I'm not sure how uncontained handles are dealt with (my guess is that those would need to be removed by hand), but that isn't relevant for my issue. I don't suppose anyone knows where this stuff is documented (I assume the people who used to write CIN's would have experience with this). Testing LabVIEW this way would be good for double checking the documentation does what I think it says, but seems error prone to figuring out what's exactly going without documentation. I'm going to assume my third approach will work, unless someone knows of some issues with it.
  7. I'm currently adding a scripting support via LuaJIT to a program I'm working on. The script is working with potentially large arrays (1 million+ elements) of doubles and strings. I pass both types via Adapt to Type Handles by value and I am able to read both of them without issue. I can also modify values within the double array. Now my problem is I want to output arrays of doubles and strings from my script, but their size is unknown until I run the script (I do know a maximum size, but that will be far larger than needed in the majority of cases). I can think of a few approaches but since I can't find a good description of the semantics of the LabVIEW memory manager and especially it's interaction with DS* functions I'm unsure if what I do will produce leaks or deallocate possibly in use data. I'm just going to list some of the approaches I have come up with with string arrays since double's should be a much easier problem. The safe method: Run the script have it store the data separately then read the string array one element at a time into the LabVIEW array. While this is certainly safe it's also horrendously slow. The I think might be safe method: Have LabVIEW pass a handle to a string array that's been preallocated with empty strings, the length of the array is the maximum amount of strings I could possibly need. Now use DSSetHandleSize on all the string handles I'm writing to to make space and write the string data and string size. Next use DSDisposeHandle on all of the unused strings (I assume I'm responsible for deallocating them since I'm throwing them out them), then DSSetHandleSize on the array handle and set the array size. With this method I'm assuming LabVIEW is still responsible for cleaning up all the resized handles. The problem with this that the array is probably several orders of magnitude too large, so I'm putting extra pressure on the memory manager. The method I want to use: Have LabVIEW pass a handle to an empty string array. Use DSSetHandleSize on the array handle to make space for the strings I'll need and set the array size. Then use DSNewHandle to create new strings to fill up and set their sizes. Now typically the thing responsible for allocating is also responsible for deallocating, is this still true when I'm putting the string handles in the array that LabVIEW is already handling (in my previous approach I deallocted handles that LabVIEW made, would LabVIEW do the same when it's done with the ones I allocated). If I need to deallocate the strings myself do I need to be extra careful that LabVIEW isn't currently using that data already (IE use the always copy node on anything that comes out of the string array, before I pass the string array back in to the script to close it). There's a decent change I might end up using arrays of single element clusters of arrays of strings. If there's some reason the concepts from the above wouldn't apply to that. Matt W
  8. If you use a reference to the subpanel with a register for events node, you can switch the state when the mouse enters and leaves the subpanel. But when I tried it with the busy state on the leave event was only triggered when I left the vi and not the subpanel. Which I think is a bug in LabVIEW, but I'm not certain how the busy state is supposed to affect events.
  9. With memory it's cheaper to have one array. Assuming the difference between times is constant then it's even cheaper to use waveforms which encode times as a starttime and the difference between times. Speed wise it depends on what you're doing to the data but it's probably going to be best as waveforms, then a single arrays then multiple arrays.
  10. I was mistaken you can use recursion, instead of messing with the VI server. I made an example in the attached zip (note poorly tested, and poorly documented since I couldn't spend too much time on it). In the example URLS of "slow" take 500 ms, "timeout" takes what ever the timeout limit is, and anything else takes 100ms. The simplest and fastest way (although it uses the maximum amount of connections), is to set C to 1 and P, as well as the generated number of loops, to the maximum amount of connections you can have at once. Multiread Recur Example.zip
  11. If you configure the parallelism in the for loop you can have up to 64 parallel instances. I meant this (note it's untested, so could have bugs). The max session limit is a setting on the router, looking up that message on google turns up Zywall routers as having a rather low limit. Some info on fixing that type of router (assuming that's what you have). http://www.broadbandreports.com/forum/r9235388-How-many-is-the-max.-number-of-session-per-host- http://www.broadbandreports.com/forum/remark,9094710~mode=flat?hilite=sessions http://www.broadbandreports.com/forum/r17252762-ZyWall-2-max.-number-of-session-per-host- I'm not sure if the datasocket caches it's connections, if it does explicitly opening and closing the datasocket might help.
  12. The simple solution is to adjust the parallel for loops settings like so. The problems with this are your attempting P network connections at once, which might cause problems if P is really high, but if the amount of troublesome links is larger than P then It'll slow down some, also if a slow page finishes then that loop instance gets another slow page it might be too slow. Since your time out is smaller than the loop time, P doesn't really need to be large than the number of troublesome pages. I think ceil(numberofpages / floor(looptime/timeouttime)) for the P value (with the generated loops set to value larger than P will ever be) should work. You could also set C to floor(looptime/timeouttime) for some performance, but I don't think you'd notice the difference. The advanced solution is dynamically launch multiple copies of a VI that handles a request and dynamically launch another when a request seems to be slow.
  13. I haven't tried it but there might be another way of going about it with the import and export string commands. There's an app reference invoke node that can batch all the different VIs together into one text file. http://zone.ni.com/devzone/cda/tut/p/id/3603
  14. I meant move the column count out of the while loop. You might want to put a note in the documentation on which execution system you're using. I've just been using the same as callers in mine. I haven't really experimented with tuning priorities in LabVIEW. I didn't find it that to be much of a problem (you would just need to add it to your SQLite_Error VI, and maybe call it a bit more often). It helps a lot with debugging, of course I'm working with raw SQL strings far more often than you are.
  15. Mine is heavily dependent on inlining for speed, you're probably much faster in LV2009 due to that. If I needed LV2009 support I'd probably switch to subroutines. VI nesting wise we're pretty similar (if we ignore the higher level abstractions on yours), I have only three VIs where I have more than one DLL call (one to copy a query string to a pointer, the other is handle strings with /00, the last is to read a column based on it's type). But I avoid the speed lose due to inlining. I added a repeated query interface to mine (I use it to run stuff like mass inserts) and In the benchmark I gain about 10% on the insert due to inlining (even though it's only called once). LabVIEW cuts of all the dead call for storing the results of the repeated query (since I don't wiring the result data on the insert). If I wire the results to something I loose my 10% gain. The dead code elimination and inlining makes writing a performant interface much easier since I don't always to write multiple version of the code for different requirements (which also simplifies the interface). If I modify your fetch all on LV2010 64 on win7 I gain about 13 ms on a select with 100000 records. A few more suggestions on your version. Drop the high priority execution setting, typically disk IO is considered low priority (starving data collection would be bad). You should support for sqlite3_errmsg in your error handler, since that'll give a much explanation of the error (stuff like table names or which command is misspelled). You'll probably want update to use binding, since currently you can't update to a blob value.
  16. Seems that I do better on my Win7 machine at home. On Win7 LV2010 32bit 10000 Records Yours Insert 40.95 Dump 37.14 Mine Insert 35.57 Dump 36.87 100000 Records Yours Insert 405.19 Dump 371.07 Mine Insert 351.46 Dump 376.54 On Win7 LV2010 64bit 10000 Records Yours Insert 39.11 Dump 41.59 Mine Insert 29.88 Dump 33.94 100000 Records Yours Insert 377.77 Dump 471.24 Mine Insert 289.93 Dump 397.06
  17. Finalizing should destroy all the bindings (as well as the statement). And since you have to finalize before closing they shouldn't be persistent. Oh I think you missed an error wire in the win32 part of fetch record. Speed wise using the current version of yours and the current version of mine (which I've made a a couple speed improvements to since the last time I compared) are very close. Your winning in inserts and If you modify fetch all so you only check the column count once per statement instead of per row you should win dumps as well. For me to to be equal speed wise, as far as I can tell, I'd need to break features that I use. On XP LV2010 Both with sync off 10000 Records Yours Insert 57.36 Dump 59.39 Mine Insert 59.21 Dump 57.1 100000 Records Yours Insert 587.89 Dump 583.53 Mine Insert 609.81 Dump 582.05
  18. Just replace the terminals with shift registers (generally a good rule with for loops anyway). But I was talking about the byte array from the string that gets passed to the bind string (it seems that empty arrays are passed as pointers to 0 in CLNs). Is there a reason you're clearing bindings before finalizing the statement. It's my understanding the clear just sets all parameters to null, and since you're closing it doesn't matter what the bindings are.
  19. I have a variant interface for differentiating nulls and empty strings when needed (which I need in my case). But with a pure string interface, making an empty string null is more useful. [edit] this seems to be wrong If you want an empty string to be written as NULL then you need to make sure the value passed doesn't point to null (just replace the 0 in my fix with any other number). [/edit] It seems an empty arrays are pointer to 0 which is why SQLite makes it null, not that it doesn't point to a /00 character.So you don't need to fix your current version to keep empty string as null.
  20. Exactly, without the fix the empty string is bound as a NULL (unless the pointer passed to the dll happens to point to a \00), which fails the constraint. With the fix it get's bound as an empty string and passes the constraint. I figured out the 2x size 32 bit example slowed down. I had left the speed test front panel open, without the front panel being open it doesn't slow down.
  21. Found a bug while optimizing mine that affects yours as well. If you insert a blank string it'll probably be bound as a NULL. It can be fixed if the empty string passed to sqlite3_bind_text is null terminated (I'm not sure if this is a bug with sqlite or a undocumented feature to allow writing nulls with string binding). Inserting as a string instead of a byte array also fixes it since LabVIEW will null terminate all strings passed to the CLN.
  22. The speed difference between 0 and -1 it should be more apparent when inserting large data elements. There's limits on how much of the hardware details one can abstract away before you start losing performance. NI could double the chunk size in 64bit, but then you get a difference when working with arrays not containing pointers. Using exponential size increases is better theoretically but can lead to out of memory issues (This wouldn't be much of a problem on 64 bit). I would be probably lean toward using exponential allocation on 64 bit and leaving 32 bit with the chunk allocator, the code would perform differently (extremely in some cases) between the two but 64 would get a nice speed boost when allocating large arrays from while loops.
  23. My understanding of the problem is that the bound data isn't read until you step the statement, and labview may unallocate the data passed to the dll by that point (in this case it's reusing the data pointer from null terminated copy for both calls), by passing -1 sqlite makes it's own copy of the data before returning. The conversion should be free (it's just a static type cast). Passing the array avoids LabVIEW making a null terminated copy, which should speed things up. But you need to be certain LabVIEW doesn't deallocate the memory the string data until after you've stepped the statement. I think the IPE can do that (that's how I interpret it's help entry), but I'm not absolutely certain what the IPE structure does at a low level. Without the IPE (assuming it does make that guarantee), your risking the compiler deallocating it. Different platforms, future versions of LabVIEW, and different optimization cases may break it. I'm using -1 just to be safe. I would suggest you at least use an IPE to be sure the relevant string data is in memory. If you can find documentation that the IPE will always delay deallocation, let me know then I'll switch over mine as well . I had the requirement on mine that any data I put in comes out exactly the same (which as far as I can tell is true with my variant interface, assuming a columns type affinity doesn't mess with things). And since strings in LabVIEW can contain /00, reading them is a requirement in my case. The problem is LabVIEW is allocating a array (arrays need a continuous chunk of memory) of string pointers in the while loop autoindex. When that array fills LabVIEW makes a new array larger array and copies the old one into it. The copy is the main cause of the slow down. Now LabVIEW seems to increasing the size of the new array in chunks (at least when the array is large). And since in 64 bit the pointers are twice the size the array needs to be resized and copied twice as often. Since the copies cost depends on how much data is getting copied, this leads problem getting exponentially worse with increasing array size. If I'm correct the size of data elements should not affect when the exponetional error starts showing up, and the 32 bit should look like the 64 bit when reading the twice the number of rows as the 64 bit. Which is the case. This can be avoided by growing the array exponentially (for example doubling it's size when filled), but you'd have to code that by hand (and it's slower since I used to do that before I saw the autoindex in yours). You could check if the returned array has reached a size, this say number of columns * number of rows read = 500000 (and half that limit in 64bit), then switch to doing exponentional growth by hand.
  24. You're correct there is a bug in SQLite_Bind.vi change the 0 on the bottom to -1 and it'll work. I must of ran my 32 bit test in 64 bit since the problem no longer shows in 32bit. I'm saying that internally LabVIEW string ARRAYS are an array of pointers to pascal strings. The CLN interface has nothing to do with it. On a side note. I would suggest just binding everything as a string, then you can speed up the binding by not checking for \00 and just inserting the data as if it was there (string length is constant time in LabVIEW but searching for /00 requires checking each byte of the string). Then you just need to add support for reading strings containing \00.
  25. The string array is an array of pointers to the individual strings (I wasn't thinking right with my original explanation ) so extending the inserted data shouldn't change the performance of the while loop index. The reason the strings need to stored as pointers in an array is because they have unknown lengths. If they where stored in one huge block then to index the array you would have to read all the earlier strings. As an array of pointers you can calculate exactly where the offset is with a single multiplication and addition. A little more info here on what I'm talking about http://en.wikipedia....ide_of_an_array A lot of LabVIEWs array operations just adjust stride lengths and array pointers which avoids data copies. Off the the top of my head this should include any form of indexing, subset, and reshape array (assuming the new size fits) I'll try to figure why mine shows the problem in 32bit, since last I checked our implementations of select should be exactly the same on the index memory requirements.
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.