ShaunR

What's the issue with putty? I can connect to my VPS and execute commands with LabVIEW without any problems. (Maybe they are hesitant that you don't see anything in the DOS prompt as it is redirected to the Shell Execute) An important point, however...... The examples tend to send the username and password in plain text. Do not do this! Instead you need to create a private and public key key pair (not very easy for point and clicky people, best to get the linux bods to do it for you and give you the private key-make sure they give you one that doesn't require a password) and tell the SSH server on the remote machine (again, linux bods) to only accept that key (or a couple of them if you have multiple users). Then putty will do a secure authenticated login using the keys. This is the command line you will need to execute (note the ppk key file). The -P parameter is the port number which should be non-default for good measure, but you can leave it out while you are getting it to work on the default port. Note: all my files including putty are in a c:tempputty directory

"Set From JSON String.vi", "Name.vi" and "First Char.vi" are not set to re-entrant.Just downloaded the latest version and "Set From JSON String.vi" now seems to be reentrant which cures the blocking. The others are still not re-entrant though.

DLLs cannot be run standalone. You always need a host application. Why do you need a parameter interface between "new and legacy" code? What I was suggesting is you just pass in a filename to process, a filename for output (cmd line parms) and hey presto you get the db that your real application can use. You can put it on your website for people to download if they need it (perhaps bundle it with the main app to begin with) and once their files have been converted, they will no longer require it at all. It never needs clutter your application code-base, rather, a separate project as a bridge to deprecation of the old file format..

A common way of solving this with the minimum effort is to create a conversion tool. It only needs to be created once, then you can forget about it and eventually you will no longer need it. Some even offer the feature as a menu option in the main application which just invokes the converter. If it's easy to spit out your custom format as a delimited text file or a number of files, you can easily import them with the API File tools (2 steps rather than one, but may be a lot easier).

Perhaps I didn't make it clear. I was not suggesting that other languages don't need the run-time. Just that they only need the 1 run-time as opposed to, say, a labview 2011 exe with a 2009 dll which needs two, I believe.

Why not just wait until you come back when the others will have been addressed too?

Well. A DLL is a self contained list of compiled executable functions (they are not equivalent to programs). If you call a function using the CLFN I don't think it has anything to do with any libraries you have placed on the block diagram (scope-wise). As long as the function exists in the DLL and the there is only one DLL with that name, that is all that is required (parameter lists of course need to be considered). Unless you are dynamically loading an external lvlib from the DLL (which is a bit silly), I don't really understand the question. DLLs are meant to make programs modular just as lvlibs are. It tends to be one or the other with lvlibs being native to labview. If you have compiled a lvlib into a DLL, then your exe will use whatever version of the lvlib you compiled the DLL with (your program only knows of the function name and parameters to pass). Replace the V1 DLL with the V2 DLL and your program will not know much is different unless the parameter lists have changed for the function calls. That's the whole point of them-so you can update/modify parts of the code without affecting/recompiling everything else. That said...... There are a couple of caveats that are peculiar to LabVIEW DLLs specifically. Rolf has outlined a bit of it in that LabVIEW DLLs have a huge dependency on the run-time and it's not easy to know what those dependencies are. So you can find yourself a while down the road backed into a corner and installing every version of labview run-time known to man to keep the various bits of your "modular" code that you have developed over several LV versions working and wondering why, this time around, it runs like a slug. You also lose your cross-platform capabilities too! (Cannot create .so or dylibs/frameworks with LabVIEW) My advice is don't use Labview DLLs unless it's for use in another programming language and your options to provide an interface for them are limited. Other languages executables don't have the same dependencies as LabVIEW executables so are less likely to run into version problems between the DLL and the program itself.

It seems to be blocking with arrays. Here's an example to demonstrate. The text file is made up of giant arrays so it makes it much more obvious (takes about .4 mins to execute) I'll take a closer look on Sunday. A bit strapped for time at the moment. Sweet. I'll take a gander later.

Indeed. There is zero error checking in the prototyping beyond what prevented me from achieving the feasability testing(thats one of the reasons why I call it prototyping rather than alpha or beta). Now I have piqued interest, we can start productionising and locking down the robustness. We also need to do the standard checks like ensuring that allocation actually succeeds etc. I'll add some of your changes to the next release. There is no point in adding a first call flag to the pointer block. That would require shift registers in the users code. Although you have added it, you haven't actually used it have you? I think I will pretty much be echoing what others are saying when I say DSCheclPtr isn't even worth being a function. The only thing it seems to check is if it is not-null (pass in "1" and it will pass). Not surprising really, it's the same problem in C.Any pointer checking is probably not going to yield a robust method. The method I described earlier works really well. It doesn't rely on the readers or writers trying to detect pointers. It relies on not being able to deallocate until everything has finished using them. This raises a slightly different issue though. It works great for asynchronous deallocation as long as the readers and writers are able to propagate their state. In a while loop this is fine as an extra iteration is possible so that they can read the registration bit and un-set their active bit. Not so good for the fixed for-loops though as the extra iteration cannot happen if you wait until all have completed their full number of iterations (works ok before then though).

+1. There was no such thing as a run-time error until that came in

Ooooh. I missed this bit. I'm not sure if I am now answering my modified readers list suggestion with the managers bit AND the readers bit or not. So I'l plough ahead with the assumption that this is in response to that scenario (apologies if that's not the case) No. The Read 1 and DeInit are not operating on the same memory location. Therefore there is no requirement for CAS and as only one writer can write to any bit, there is no race condition. Each writes to it's own bit associated with each reader (the regman writes to several locations in the list but it is still the only writer to those locations. The readers each handle their own "active" bit so they each are still the only writer for their bit in the list). The writer only reads both bits for all readers to determine. a) Is this reader to be included in the lowest check (only the registration managers bit is important for this) b) Are there any readers at all (all the regmans bits are false AND all the readers' bits are false)->exit and set "Finished". In this scenario, it is the writer that exits signals everything is OK to kill the pointers (it is monitoring all the bits), not deinit. Deinit unregisters all the readers then waits for the "finished" bit to go high then it deallocates. The"active" bit in each reader is a proxy for the regmans bit in that the regmans bit is read and then the value written to the "active" bit for that reader on exit (no other reads/writes happen after) When the "finished" bit goes high all readers and the writer have already exited and will not be reading/writing to any pointers. I'm hoping to have a knock-up of the registration over the weekend to test the idea and see what I run into (depends how drunk I get Saturday night-hangovers don't get worse as you get older, they just get longer ).

The "ball-of-mud" is the one pattern that rules them all Just don't use it on any of my projects

As it stands. Yes. The pointer cluster is "on-the-wire" and when we deinit we just leave the cluster alone. But it doesn't have to be (just makes it easier to debug). If I were to start on a "Classic LabVIEW" API, I would probably also shove the pointer cluster into memory and you wouldn't need a "reference" wire at all, hell, even a global variable to contain them (no "not allocated" issues then). With the former, I might use the "CheckPointer" call if it wasn't too expensive and it does what I think it does as belt and braces with a check for null pointer on the cluster (if the cluster doesn't exist then neither do the others and vice versa). But if you are looking at classes I thought you would want to manage all that in the class.....somehow. If I put everything into memory and handle all the scenarios, there isn't much point in a class at all apart from making people feel warm and fuzzy about OOP. I think the issue you are probably running into is that you are finding the best means to achieve what you need in a class is a DVR. But you can't use them because of the locking overhead. If you find you are looking to a DVR,, then the structure needs to be via the MM functions as they are basically just a method to provide the same function, but without the DVRs locking overhead. Anything else can "probably" be in the class. I would be "happy for now". I don't think the unallocated pointers are an insurmountable issue and at worst we just need some state flags. I would come back to it on another iteration to see what needs to change and what the performance impact is of a million flags all over the place. I don't think we do (need a test 'n set). The premise of the pattern is mutual exclusion through memory barriers only. As long as you only have one writer to any location then no test and set is necessary. As we have solved the issue of accessing individual elements in arrays without affecting others or locking the rest of the array, all we need to ensure is that write responsibility is well defined (only one writer to a single location or block). The only time a test 'n set would be required is if we couldn't guarantee atomic reads and writes of the individual bits (PPC?). As an aside. Anecdotally, it seems writing all contents to a cluster is an atomic operation in labview and incurs a marginal overhead as opposed to accessing the memory locations independently. The overhead is miniscule in comparison to the extra library calls required to achieve the latter. If this can be confirmed, then it is the fastest way to add atomicity to entire blocks of locations when needed. This is one reason why I use a cluster for the elements in the index array..

One thing that I haven't gotten round to yet. If you re operating on a large JSON stream, you cannot process any other JSON streams as it seems to beblocking. I think it just needs setting some of the subVIs to re-entrant, but like I said. I haven't gotten round to looking as yet.

As you know. Any mutexes and we will be back to square one. I'm still not quite getting it. Why do we need an "op count"?. All pointers are considered valid until we unregister all readers and writers. Unregistering a single reader amongst multiple readers doesn't mean we need to free any pointers as there are only three (the buffer, the reader index array and the Cursor). The only time we deallocate anything is when there is no longer any readers or writers at which point we deallocate all pointers. Now. We already have a list of readers (the booleans set to true in the index array) and we know how many (reader count). The adding and removing of readers, i.e. the manipulation of the booleans and the count, is "locked" by the registration manager (non-reentrant VI boundary). So I see the "issue" as how do we know that any asynchronous readers on the block diagram have read whatever flags and exited and therefore the writer can now exit and pointers can be deallocated (the writer only exits when there are no readers (race condition on startup? Will have to play......). The writer won't read any reader booleans since by this time the registration manager has set the reader count to zero (this order will change with my proposal below since it will exit before it is zero). So it goes into a NOP state and exits. It can set a boolen that says "I have no readers and have exited". The registration manager already knows there are no readers, so it just needs to know the writer has exited before deallocating pointers. The readers only need to watch their flag to see if the registration manager has changed it and go into a NOP state and exit. At this point I can see there might be a scenario whereby the reader has read it's flag (which is still OK) and by the time it gets to reading buffers and writing indexes the writer has said "I have no readers and have exited". Well. Lets put another flag in the reader index array that says "I'm active" which is just a copy of the registration managers boolean but only written once all memory reads have completed and causes an exit . Now we have a situation where deallocation can only occur if the booleans controlled by the registration manager are all false (or true depending on which sense is safer) AND the "I am active" booleans controlled by the individual readers are all false (ditto sense) AND the writer has said "I have no readers and have exited". This decomposes into just the writer saying "I have no readers and have exited" as the writer can read both fields whilst it is doing it's thing (it has to read the entire block anyway), AND them together and exit when they are all false (sense again) and set the "I have no readers and have exited". So in the end-game. The registration manager unsets all the registered booleans, waits for the "I have no readers and have exited" boolean then deallocates the pointers. Does this seem reasonable? Being colour-blind (colour confused is a better term). I really have no opinion on this Well. Get rid of the "Check For Errors" page then (I've never had an error out via this route since about LV 7.1)

Ah. Yes. IC. Here's what I think we could do....... The registration manager knows how many and which ones are allocated. It will pass back an available ID index to the reader as part of the registration process (the ID is just an offset from the base pointer in # blocks). When there is a registration request, it scans the booleans in the reader index array (no locks required) and passes the first F it comes across. If all are in use, it would increase the size of the array (setting all new values to F and indexes to the current cursor position) then give the next ID to the reader. The reader then uses that ID and starts its counter at the set cursor position. At this point the manager sets the boolean field for the newly registered reader to T and sets the reader count to +1. The next scan by the write will now iterate through the newly increased size and pick up that the new readers bool is T. Why can't the pointer cluster actually be the private data of a class? I don't see any reason for refnums if it's going in a class. Instantiating the class creates all the pointers (well. there is no constructor in LV, so until we get my atomic reads and write, they will probably call an init). If the block of pointers you are talking about is to do with the ID. Then we don't need any since the ID number is sufficient and the user will have no idea which IDs are being used as that's internal (as described previously) Well. I'm not part of the "Crashing is an acceptable behaviour" club. They are idiots (although I think you have a couple over there at NI when it comes to CLFNs )At worst, we could unregister all the readers (set all the booleans to false) then wait 2 weeks to make sure everything has had a chance to read the booleans and exit before we finally crowbar the memory. But I'm sure we can come up with something better than that You're thinking a bit ahead of me at the moment. I tend to think in chunks with a bit of prototyping for feasability (iterative development). I'm only just starting to formulate details about the registration let alone about the API itself. What do you suggest?

I couldn't look at your snipit because Lavag seems to be stripping the meta data again and I can't import it as code. But based on your quoted comment........ The varpointers.vi I attached in the first post uses MoveBlock and it works once but dies when deallocating the pointer the second time around because LabVIEW kills the variant at some point or, more probably, I've killed it when I shouldn't have.

Hmmm. OK. The Elapsed Time.vi is set to subroutine. Does setting it to "Normal" help? (I don't have access to a dual core ATM)

I don't understand what you are getting at here. Refnums? The disruptor pattern can cope with multiple writers (ours currently can't). They use a two stage process where a writer "claims" a slot before writing to it. What's the problem with having multiple readers? (isn't that the point!) We effectively have a reference counter (number of readers) and we have a method of skipping in the writer (the bool field of the pointer indexes which the readers could also use so they don't go ahead and read data-the manager only manipulates that field) so we would only deallocate the pointer when the last is unregistered. The deinit would effectively iterate through unregistering all the readers (it becomes an "unregister all readers" then kill pointer rather than just a pointer killer). I think we just need a NOP for the readers (they need to not read until the registration manager has given them an index slot and they know the current cursor position) Don't know. Just thinking on my feet at the moment but "seems" doable and I think it might give us the same sort of feature as the queues when the handle is destroyed (error out).. Not really. I have visions of fixing the i64 cursor wrap-around problem mathematically by using the fact that it goes negative but still increases (gets less negative, minus + minus = plus, sort of thing). As I haven't fixed it yet, you could use a negative number as you will hit the same problem at the same point and everything starts at zero. Well. I think we need to look at the Disruptor code again to see how they handle it (they probably have the same problem). If we can't think of a way to be able to reorganise, we can at least dynamically increase as that doesn't overwrite existing indexes; just adds new ones and disables old ones (haven't seen a realloc but have seen functions to resize handles......somewhere). So we could create, say 5 at a time and increase in blocks. We are back to the MJE kind of approach which I first outlined with the flags then. (I did say IF we were smart ) I think this is where we need to revisit the disruptor pattern details. They talk about consumer barriers, claiming slots and getting the "highest" index rather than just seeing if it is greater (as we currently do). I think we have the circular buffer nailed. It's the management and interfaces we are beginning to brainstorm now and they must have solved these already.

In the writer there is a disable structure. If you disable the currently enabled (and enable the other) it will use the previous index reading method. Does this affect the result you see?

VxWorks paths are case sensitive (windows aren't by default),

Yes. That's the tricky bit though since you cannot serialise LabVIEW objects so it's not just a case of "(un)Flatten To String" to get them across a network.. This thread on "Lapdog over the network" will highlight most of the issues (and solutions)..

You can't use a negative number with this because the indexes can be negative (strange, I know, but it is to do with wrap-around). I'm thinking that we could have a boolean with the R and Cnt (a block is converted to a cluster for processing-see the CB GetIndexes3.vi) which tells the index iterator (in the write) whether that index should be considered in the "lowest" check. This would have very little impact on performance (read a few more bytes in the block at a time and 2 AND operators.). Then you would need a "registration manager". That would be responsible for finding out how many are currently registered, updating the registered count and choosing a slot which has a F flag to give to the reader that wants to register (not disimilar to what you are currently doing). The additional difference would be that it is also capable of resizing the array if there are not enough index slots It could even shrink the array and reorganise if we wanted to be really smart so that the index iterator doesn't process any inactive slots and do away with the choosing of an inactive slot altogether. This only ever needs to happen when something registers/unregisters so even if this is locking, it will not have much of an impact during the general running (now we are starting to get to the Disruptor pattern )

Yeah. Don't want to go to a DLL. Been there, done that, torn holes in all my T-shirts. Is there no way we can utilise the callbacks (I thought that was what the InstanceDataPointer was for). That wouldn't work well for my use cases since I don't know how many I need up front (could be 10, could be 30 and may change during the program lifetime as modules are loaded and unloaded->plugins). The rest (about first call etc) is valid. It really needs to be in the next layer up but as that doesn't exist yet, its in the current read and writes. What we (or more specifically, I) don't want is for them to be reliant on shift registers in the users program to maintain the state info. How that is handled will depend whether the next layer is class based or classic labview based which is why I havn't broached it yet. At that layer, I don't envisage a user selectable "Init". More that the init is invoked on a read or write depending on what gets there first (self initialising). I ultimately want to get to a point where you just slap a write VI down and slap read VIs wherever you need them (even in different diagrams) without having to connect wires (see my queue wrapper vi for how this works, although not sure how it will in this case, ATM, since the queue wrapper uses names for multiple instances).

Indeed.I know the issues and have some ideas (easiest of which is to have a boolean in the index array cluster alongside the R and Cnt). Do you have a suggestion/method/example ?