drjdpowell

This schema seems to work (I did a 10-element buffer instead of 1000): CREATE TABLE Buffer (Value); CREATE TABLE BufferIndex (I); CREATE TRIGGER BufferIncr BEFORE INSERT ON Buffer FOR EACH ROW BEGIN UPDATE BufferIndex SET I = ((SELECT I from BufferIndex)+1)%10; END; INSERT INTO BufferIndex (I) VALUES (-1) -- Initial index is -1 Then insert data (the current time in this example) into the buffer with: INSERT OR REPLACE INTO Buffer (rowID, Value) Values ((SELECT I FROM BufferIndex),time('now')) The Trigger serves to increment the index before the INSERT. To get the ordered elements one can use a View: CREATE VIEW OrderedBuffer AS SELECT Value FROM Buffer,BufferIndex WHERE Buffer.rowID>I UNION ALL SELECT Value FROM Buffer,BufferIndex WHERE Buffer.rowID<=I;

The point of a WITHOUT ROWID table is when you want a primary key that is non-integer or composite, so the alternative is a regular rowID table with an index. Re the “order†stuff I was being sloppy; rather, I mean the primary data B-tree uses Time,WN as it’s index (rather than rowID), so one can directly search on Time,WN without needing an separate index.

It is, but it’s a big one in this case. Compare it to a regular table which is: rowID, Time, WN, Y Along with an index on (Time,WN) which is a second table, equal number of rows, of Time, WN, rowID. The index table is a significant fraction of the primary table in size. The Without RowID table eliminates the need for this index.

Remember that indexes take up memory, and memory/disk-access can be bottlenecks. If your table rows are just is X,Y,Z,Float then you may be losing out by indexing everything. How many points can you store per GB? I actually am working on a similar SQLite application at the moment. Here is my “big†table, which stores spectrometry spectral points: CREATE TABLE IF NOT EXISTS SpecPoints ( -- Individual points in the spectra Time INT, -- Time in ms, Relative to Time_Offset Attribute (allows 4-byte Integers to store +-24days) WN INT, -- Wavenumber, in units of 0.005 cm^-1, offset 40,000 cm^-1 (this makes it three bytes up to 125nm) Y INT, -- y= 0.5-Transmission, units of 0.0000001 (allows -0.34 to 1.34 Transmission range to fit in 3 bytes) CONSTRAINT pk PRIMARY KEY (Time,WN) ON CONFLICT REPLACE ) WITHOUT ROWID; I’m using a “WITHOUT ROWID†table to physically store the rows in order of Time and WN and I’m not creating any extra indexes. I’m also using scaled integers to pack things tighter than 8-byte Floats (SQLite store integers in the smallest size of I8, I16, I24, I32, I48, or I64). I’m doing averaging over Time bins and Wavenumber regions-of-interest, rather than decimation, so this may not apply to you, but note that a smaller database file makes everything faster.

Here is an illustration of what Thoric is discussing. Calling code starts two actors: an “Instrument" actor that is publishing a “Readingâ€, and a “Furnace†actor that expects to receive “Furnace Temperature†messages. If the two messages both use the same data format (a numeric in Kelvin, say) then we use the code on the left to connect the two actors directly. We configure the messages to be relabelled with the “Translate†polymorphic VI. Looking inside that API call we see that we are providing a “Translator†child class to actually do the relabelling. However, if Instrument publishes its “Reading†as Kelvin, but the Furnace expects “Furnace Temperature†to be in Fahrenheit (horror!) then we can supply a custom child Translator class. This class only needs one method, an override of “Translate.vi†(shown), which adapts the messages to the form required by Furnace. We provide this as a second translator as shown in the code at right. [Opps: just noticed I misnamed the Translator child class as degF to Kelvin when it is really Kelvin to degF]

That’s not a “poolâ€. That is three independent clones. A pool is when you don’t care which instance you are talking to.

Hi Thoric, As a quick aside to other readers let me note that it is a goal of Messenger Library that one NOT have to be a whiz at LVOOP to use it. The standard API is meant to be used by itself to make inter-loop communication easier, without the need to create new classes. But because it is based on LVOOP internally, one can extend it powerfully using inheritance. So with the proviso that these are “optional bonus power ups†that are rarely used, then yes, the “Translation†polymorphic VI on the “Observer†pallet is specifically intended for one to alter messages. The standard translations that I use commonly are “Relabelâ€, “Add Prefixâ€, and “Outer Envelopeâ€, but there is also “Add Translatorâ€, which allows one to provide a custom child class of “Translator.lvclassâ€. The only method that must be overriden is “Translate.viâ€, though you may want a creation method if your Translator need internal info. See the “LabelPrefix.lvclass†for an example. The “Translation†method is quite straight forward (see an example below). Even more generally, the “Send†parent class, usually thought of as a generic “addressâ€, can also be considered a “Callbackâ€. The “Reply†and “Notify†API methods are really callback hooks, where calling code can “hookup†functionality to an actor. The only important restriction is that the callbacks should not interfere with the operation of the actor (such as by blocking or taking an unreasonable time to execute). So one could do things like average data before it is sent, or log debugging information, for example. BUT, don’t get too fascinated by the fancy tools in the toolbox. Most of my messages contain only one piece of data. — James PS, if you want to explore another powerful LVOOP class that can be extended, look at “Async Action.lvclassâ€, which is intended to make it easy to make actions that can be run in parallel to any loop (with action results sent back in a message).

It come up in JSON to Variant, where the User of the library provides a target cluster containing their custom child classes. I need to provide a Variant that can be converted using Variant-to-Data into their cluster. My workarounds are: 1) require the User to override a “Put in Variant†dynamic dispatch method. Very simple, just puts the object in a Variant with the right wiretype, but a burden on the User. 2) get as LVObj; flatten to string; unflatten using the type descriptor from the supplied child object. This allows one to ‘transfer’ the wiretype into the Variant. But the flattening/unflattening is heavy overhead.

Stop using the 0x40 pool option. You don’t want a reference to pool of clones. Try just 0x8.

Not sure how 0x8 and 0x40 options interact, but with 0x40 you are creating three shared clone pools, so the answer would be three values between 1 and 3, depending on how LabVIEW assigns idle clones in the pool. With just 0x8, you’re making three clones, and the answer is [3,3,3].

Darn. I actually had a DEV template modification that demonstrates some of this, but I removed it from the published package because I thought it was too niche. See a screenshot below. I also need to use a subVI to name the event registration. You can also see a “Private Messenger†in the JKI Actor Template. Note, BTW, that I don’t use EventMessengers in the way you are: a different Messenger for each message. Instead I have multiple messages coming in through one event case, with a case structure on the message label. The only reason I sometimes have a second EventMessenger is to separate “Public†(or “Externalâ€) messages from calling code, from “Private†(or “Internalâ€) messages coming from my subActors. So I can prevent calling code from being able to send me private messages, and I can limit what subActors can do (like not tell me to shutdown, for example).

The second option. PS> I’m a bit stuck on this at the moment because of a problem with “Variant to Data†being too “dumb†when it comes to child classes. If one has a Variant containing a Parent-class datatype holding a child-class object, and you attempt to use "Variant to Data" to cast this to a Child-class wire, it throws a type mismatch error, even though such a conversion is easily possible. This is a problem when the library-user want to use clusters containing their own child classes. There are a couple of work around but both are ugly.

The JSON SQL functions should be working in this beta (I think it includes the 3.9.0 dll with those functions). I’ve only played around with them a little, but they do work.

Oh, I’m not even talking about a framework. Just a hook that allows someone to connect such a thing to the JSON-Variant tools without modifying JSON.lvlib. Then CharlesB could provide his object serializer as an addon library.

I suggest you just decimate your data after the filter.

Note: I’m thinking of putting in an abstract parent class with “To JSON†and “From JSON†methods, but I’m not going to specify how exactly objects will be serialized. So Charles can override with his implementation (without modifying JSON.lvlib) without restricting others from creating classes than can be generated from different JSON schema. I could imagine use cases where one is generating objects from JSON that is created by a non-LabVIEW program that one cannot modify, for example. Or one just wants a different format; here’s a list of objects from one of my projects: [ ["SwitchRouteGroup",{"Connect":"COMMS1"}], ["DI Test",{"IO":"IO_ID_COMMS_1_PRESENT","ON=0":false,"Value":true}], ["Set DO",{"IO":"IO_ID_COMMS_1_EN","ON=0":true,"Readable?":true,"Value":false}], ["Wait Step",{"ms":2000}], ["DMM Read Voltage",{"High":2,"Low":-2}], ["Set DO",{"IO":"IO_ID_COMMS_1_EN","ON=0":true,"Readable?":true,"Value":true}], ["Wait Step",{"ms":500}], ["DMM Read Voltage",{"High":13,"Low":11}], ["AI Test",{"High":3600,"IO":"IO_ID_COMMS_1_V_SENSE","Low":3400}], ["SwitchRouteGroup",{"Disconnect":"COMMS1"}] ]

I could easily make a “JSON serializable†class with abstract “Data to JSON†and “JSON to Data†methods, and have the JSON-Variant tools use them. Then you could inherit off this class and not have to modify JSON.lvlib.

I would Google “CRC16 LabVIEW†first, if I were you, and see if you can find it already done for you.

Problems show up in the unflattening part. Flattening, the part you are trying, is strait forward, because you are going from strongly-typed LabVIEW to weakly-typed JSON. The other way is harder. You don’t need to make your class a child of JSON Object; just have your class have “To JSON†and “From JSONâ€. You won’t be able to use these objects in the JSON-Variant tools, but you will be able to handle things with the other methods. This involves more coding, but is faster. And it allows one to get away from the monster config cluster and towards a distributed config, where different code modules supply different parts of the config.

The current JSON-Variant tools will store objects as flattened strings. Not human-readable, but flattened objects have a mutation history so you can freely rearrange things inside the class private data (but not on any condition re-namespace the class). I find that mixed-mode quite useful and easy. Alternately, to serialize objects as JSON I usually just have methods in the class to convert to/from JSON and I use them as I build up the JSON explicitly (i.e. without using the shortcut of the JSON-Variant tools on a cluster). There is a possibility that one could make one’s LVOOP classes children of “JSON Valueâ€, and then override “flatten†and “unflattenâ€. Then your objects would serialize to JSON even inside a cluster using the JSON-Variant tools (those tools recognize “JSON Value†objects and flatten/unflatten them). But there is a technical issue that I have to look into to make that work. [Never mind, this won’t work.] Adding or removing is fine, but don’t rename or move things, unless you want to have a “schema version†attribute and mutation code than coverts from older schema.

Ah, I see. We need something like Error 402864, SQLITE_BUSY(5), occurred at redacted.lvlib:redacted.vi on "INSERT OR IGNORE INTO redacted ( redacted, redacted, redacted) VALUES (?, ?, ?);" Possible reason(s): database is locked. See https://www.sqlite.org/rescode.html#busy. Edit> can’t do the above, but how about: Error 402864 occurred at redacted.lvlib:redacted.vi on "INSERT OR IGNORE INTO redacted ( redacted, redacted, redacted) VALUES (?, ?, ?);" Possible reason(s): SQLITE_BUSY(5) database is locked (see https://www.sqlite.org/rescode.html)

I add 402859 to the SQLite error code (apologies for that number, but those are the range of codes assigned to me by NI). I’m currently calling sqlite2_errmsg() to get an error description from the SQLite dll itself. And the message should have also contained either the database file, or SQL statement on which the error occurred. Was this information not in the error?

If you look in the latest beta version you’ll find it (though it’s not in the palettes, it is in the Connection class). PS> Some notes: — one can also use the last_row_ID() SQL function — Be careful that you aren’t executing multiple INSERT statements in parallel on the same Connection, as you might mixup the rowIDs. — note the existence of WITHOUT RowID tables, which avoids the need to determine the auto assigned rowID.

Here is a beta version with changes aimed at performance, particularly for large arrays of data. I managed to eliminate an O(N^2) performance issue with parsing large JSON strings. I also improved the Variant tools’ speed with one- and two-dimensional arrays. See the new example “Example of an Array of XY arrays.viâ€. lava_lib_json_api-1.4.1.33.vip Please give this a test and let me know of any issues before I place it in the CR. I’m also thinking of submitting to the LabVIEW Tools Network. — James

Sorry, forgot to push the changes. It’s there now.