Debug symbols for lvrt.dll

[Mefistotelis]

Labview exists for decades. It had thousands of releases.

When was the last time NI left debug symbols by mistake?

 

How to recognize a release with debug symbols left:

- Nowadays Visual Studio keeps the symbols in a PDB file - so probably lvrt.pdb, but it might also have different name - depends on whether the DLL is renamed after build.

- Previously, up to around year 2000, debug symbols were stored in the EXE/DLL - so lvrt.dll file size would be extraordinarily large, circa 2x-3x the normal size

- Most compilers can also generate MAP file - it doesn't store as much as full debug symbols do, but has all global variables and functions named - which is still way more than nothing

 

[Neil Pate]

Not exactly LabVIEW related, but an interesting related read anyway. How Diablo was reverse engineered.

[Rolf Kalbermatter]

On 11/21/2019 at 2:28 AM, Mefistotelis said:

Labview exists for decades. It had thousands of releases.

Thousends of releases? I kind of doubt it. Leaving away LabVIEW prior to the multiplatform version (2.2.x and earlier which only were Macintosh) there have been 2.5, 3.0, 3.1, 4.0, 5.0, 5.1, 6.0, 7.0, 7.1, 8.0, 8.2, 8.5, 8.6, 2009, 2010, 2011, 20012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 releases so far. Of each of them there was usually 1 and rarely two maintenance releases, and of each maintenance release between 1 to 8 bug fix releases. This does probably only amount to about 100 releases in total and maybe another 50 for beta releases of these versions (a beta has usually 2 to 4 intermediate releases although that tends to be no more than 2 in the last 10 years or so).

Quote

When was the last time NI left debug symbols by mistake?

How to recognize a release with debug symbols left:

- Nowadays Visual Studio keeps the symbols in a PDB file - so probably lvrt.pdb, but it might also have different name - depends on whether the DLL is renamed after build.

- Previously, up to around year 2000, debug symbols were stored in the EXE/DLL - so lvrt.dll file size would be extraordinarily large, circa 2x-3x the normal size

- Most compilers can also generate MAP file - it doesn't store as much as full debug symbols do, but has all global variables and functions named - which is still way more than nothing

I'm not aware of any LabVIEW release that had the debug symbols exposed. PDPs were used even in Microsoft Visual C 6.0, the first version that was probably used by NI to create a released LabVIEW version  (NI only switched to use Microsoft C for the standard 32-bit builds for Windows NT, the Windows 3.1 versions of LabVIEW were created using Watcom C 10.x which was the only compiler able to create full 32-bit executables that could run on 16-bit Windows 3.1 through the built-in DOS extender runtime).

Microsoft makes this anyhow pretty hard to happen by accident as such DLL/EXE files would normally link to the debug version of the Microsoft C Runtime library and you can't install that legally on a computer without installing the entire Microsoft C Compiler contained in the Visual Studio software. There is definitely no downloadable installer for the debug version of the C runtime engine.

The only early leak I'm aware of was that the original LabVIEW 2.5 prerelease contained a huge extcode.h file in the cintools directory that showed much more than the officially documented LabVIEW manager functions. About half of it was still pretty much unusable as you needed other functions that were not exposed in there to make use of some of the features, and a lot of those functions were removed from the exported functions around LabVIEW 4.0 and 5.0 as they were considered obsolete or undesirable to pollute the exported symbols list, but it did contain a few interesting functions that are still exported from the LabVIEW kernel but not declared in the current extcode.h file. They fixed that extcode.h bug before the official release of LabVIEW 3.0, which was the first non-beta version of LabVIEW running on other computers than Macintosh. (2.5 was basically a beta release called prerelease version to have something to show for NI Week 1992 that runs on Windows 3.1, and there was a 2.5.1 and I believe 2.5.2 bug fix release of it in 1993).

Also lvrt.dll is a development that only got introduced around LabVIEW 6.0. As this was released in 2000 it most likely used at least Microsoft Visual Studio C++ 6.0;

Before that the application builder was concatenating the generated runtime LLB to a stub executable that contained the entire LabVIEW runtime engine. That was a pretty neat feature as it created a single file executable, but as LabVIEW was extended and more and more functionality implemented in external files and DLLs that get linked dynamically, this was pretty much unmaintainable in the long run and the entire runtime engine was externalized. 

Edited November 22, 2019 by Rolf Kalbermatter

[Mefistotelis]

Thanks for all the info!

 

On 11/22/2019 at 12:53 PM, Rolf Kalbermatter said:

Thousends of releases? I kind of doubt it.

Searching for various NI things on the Internet, I found quite a lot of packages which included the LV RT. Various toolsets, drivers, and labview-based solutions. Though most of these use standard release of the RT installer.

Unfortunately I have no pre-release versions. And this is where I expect higher chance of having some debug info left.

I've seen indications that VI Logger 1.1 had a pre-release RT in it, but I could only find the fixed 1.1.1.

 

As of now, I couldn't find the LV 2.5 as well. I found LV 6.0 on one of my old CDs, that's the oldest version I have.

[Rolf Kalbermatter]

LabVIEW 2.5 was distributed on 3.5" floppy disks.

About 4 or 5 or so of these in 1.44MB high density formatted. I ditched them several years ago as it is pretty hard to find a drive nowadays to read them.

 Later releases were substantially more floppy disk. I don't think CD-ROM was an option before LabVIEW 6.0.

Edited November 23, 2019 by Rolf Kalbermatter

[Mefistotelis]

I got a 3.5 inch floppy drive with USB interface - they were very cheap at some pint.

Anyway, I browsed through various releases available in obscure places of the web. There's A LOT of things created by NI which either use the standard LV runtime environment, or use parts of the LV code.

I downloaded > 100GB of various files when I decided it's time to stop. There was probably way more to find, but I have to limit my time.

Findings:

- Files from Windows platform are the worst port to reverse engineer, mostly because Windows libraries (.DLL) define exported routines explicitly.

- UNIX base releases (solaris,linux) are the best to debug - compiled for popular CPU architectures, and shared libs (.SO) have all routines exported - meaning their names are retained

- For some obscure platforms, I found static libraries (.a/.lib) files included in the release. These have the most info on source code retained, but not the whole code was in that form. Not the part I need anyway.

- The best version to focus on seem to be LV6.0 - the code is a lot less bloated than in newer releases, and the core of the tool didn't seem to have changed much since that release

(though I didn't found much LV5.0 releases - I can tell 4.0 is too old and does not have many blocks which exist in modern VI files, for 5.0 - I'm not sure)

 

Here's the loading of LVSR block in LabView 6.0. Unfortunately, I still don't have struct definitions. But at least I know how the structs are named, thanks to C++ name mangling.

void RToVI(SAVERECORD **sr, INSTRUMENT **instr)
{
  char fld10a;
  char fld10b;
  INSTRUMENT *dInstr;
  int state;

  (*instr)->sr_field4 = (((*sr)->field_4 & 0xFF0000) >> 8) | ((unsigned int)(*sr)->field_4 >> 24) | (((*sr)->field_4 & 0xFF00) << 8) | ((*sr)->field_4 << 24);
  (*instr)->sr_field8 = (((*sr)->field_8 & 0xFF0000) >> 8) | ((unsigned int)(*sr)->field_8 >> 24) | (((*sr)->field_8 & 0xFF00) << 8) | ((*sr)->field_8 << 24);
  (*instr)->sr_fieldC = (((*sr)->field_C & 0xFF0000) >> 8) | ((unsigned int)(*sr)->field_C >> 24) | (((*sr)->field_C & 0xFF00) << 8) | ((*sr)->field_C << 24);
  (*instr)->sr_field12 = _byteswap_ushort((*sr)->field_12);
  (*instr)->instrState = (((*sr)->instrState & 0xFF0000) >> 8) | ((unsigned int)(*sr)->instrState >> 24) | (((*sr)->instrState & 0xFF00) << 8) | ((*sr)->instrState << 24);
  (*instr)->sr_field14 = _byteswap_ushort((*sr)->field_14);
  (*instr)->sr_field16 = _byteswap_ushort((*sr)->field_16);
  (*instr)->sr_field20 = _byteswap_ushort((*sr)->field_20);
  (*instr)->sr_field22 = _byteswap_ushort((*sr)->field_22);
  (*instr)->version = (((*sr)->version & 0xFF0000) >> 8) | ((unsigned int)(*sr)->version >> 24) | (((*sr)->version & 0xFF00) << 8) | ((*sr)->version << 24);
  (*instr)->sr_field24 = (((*sr)->field_24 & 0xFF0000) >> 8) | ((unsigned int)(*sr)->field_24 >> 24) | (((*sr)->field_24 & 0xFF00) << 8) | ((*sr)->field_24 << 24);
  (*instr)->sr_field1C = (((*sr)->field_1C & 0xFF0000) >> 8) | ((unsigned int)(*sr)->field_1C >> 24) | (((*sr)->field_1C & 0xFF00) << 8) | ((*sr)->field_1C << 24);
  qmemcpy(&(*instr)->sr_field34_md5, (*sr)->field34_md6, 0x10u);
  qmemcpy(&(*instr)->sr_field28, &(*sr)->field_28, 0xCu);
  RevBL(&(*instr)->sr_field28);
  RevBL(&(*instr)->field_2C);
  RevBL(&(*instr)->field_30);
  fld10a = (*sr)->field_10;
  fld10b = HIBYTE((*sr)->field_10);
  (*instr)->is_sr_field10_flag0200 = HIBYTE((*sr)->field_10) & 2;
  (*instr)->is_sr_field10_flag0100 = fld10b & 1;
  (*instr)->is_sr_field10_flag0400 = fld10b & 4;
  dInstr = *instr;
  if ( (*instr)->sr_field4 & 0x20000 )
  {
    state = dInstr->instrState;
    if ( state & 0x200 )
    {
      dInstr->instrState = state & 0xFFFFFDFF;
      DBPrintf("Fixing instrState, remove viDebugCapable from subroutine VI");
    }
  }
}

 

Edited November 27, 2019 by Mefistotelis

[Rolf Kalbermatter]

13 hours ago, Mefistotelis said:

Here's the loading of LVSR block in LabView 6.0. Unfortunately, I still don't have struct definitions. But at least I know how the structs are named, thanks to C++ name mangling.

void RToVI(SAVERECORD **sr, INSTRUMENT **instr)
{
  char fld10a;
  char fld10b;
  INSTRUMENT *dInstr;
  int state;

  (*instr)->sr_field4 = (((*sr)->field_4 & 0xFF0000) >> 8) | ((unsigned int)(*sr)->field_4 >> 24) | (((*sr)->field_4 & 0xFF00) << 8) | ((*sr)->field_4 << 24);
  ......
}

 

(((*sr)->field_4 & 0xFF0000) >> 8 | ((unsigned int)(*sr)->field_4 >> 24) | (((*sr)->field_4 & 0xFF00) << 8 | ((*sr)->field_4 << 24);

 

This is simply the BigEndian to LittleEndian swapping. All numeric values in the VI data structures are stored as Big Endian, since they need to be independent of

the CPU architecture. On loading into memory on LittleEndian platforms (currently all but the VxWorks platform for PowerPC Real-time platforms, but in the past

MacOS Classic (68k and PPC), Mac OS X for PPC, Sun Sparc and I believe HP PA RISC were all BigEndian versions of LabVIEW) they have to be swapped.

 

Basically this is a pretty unexciting routine. It just copies the data structure from the memory stream on disk into a real data structure in memory and accounts

for endian swapping on relevant platforms and massages some flags into a different format.

The in memory version of the INSTRUMENT data structure basically changed with every single LabVIEW version considerably, even with minor changes between minor

LabVIEW versions, so trying to access the fields at runtime in that structure is a very version dependent adventure.

Edited November 27, 2019 by Rolf Kalbermatter

[Cloedu72]

If you are interested in older versions, I can provide you a link to LV3.1.1:
ftp://ftp.ni.com/support/labview/updates/windows/win3x/31-to-311

=> Have fun! 😉

 

[Mefistotelis]

5 hours ago, Rolf Kalbermatter said:

Basically this is a pretty unexciting routine. It just copies the data structure from the memory stream on disk into a real data structure in memory and accounts

for endian swapping on relevant platforms and massages some flags into a different format.

You are completely right with the assessment of what this function does. But I can't agree it is unexciting:

- It provides us with full information on how to divide the LVSR block into single values, and it even names one of the values.

- It gives us offsets within INSTRUMENT where each value is stored, so that we can cross-reference it with other functions and check what they do

(as you see, I already named the items within INSTRUMENT to "sr_field*" so that I can easily search ehere each of these is used further.

So actually, almost everything about LVSR which is in my code, comes from this single function:

https://github.com/mefistotelis/pylabview/blob/master/LVblock.py#L100

4 hours ago, Cloedu72 said:

If you are interested in older versions, I can provide you a link to LV3.1.1:
ftp://ftp.ni.com/support/labview/updates/windows/win3x/31-to-311

Thanks! I actually downloaded that whole FTP already; and yes, this is where I got LV3.1.1 from. There are 2 binaries - Win16 and WinNT; the WinNT one is better for analysis. But, as I previously stated - before LV6.0, the blocks within VI files were a bit different - so I prefer to work on more recent code.

As an example, in LV5.0 there was LVIN block instead of the LVSR discussed above.

 

Edited November 27, 2019 by Mefistotelis

[Mefistotelis]

3 hours ago, Mefistotelis said:

it gives us offsets within INSTRUMENT where each value is stored, so that we can cross-reference it

A good example I just found:

void InitButtonsHidden(INSTRUMENT **a1)
{
  (*a1)->sr_field14 = gButtonsHidden[(*a1)->sr_field22];
}

Now we can rename one field to 'buttonsHidden', and can assume another field contains enum value representing either state or type.

 

Edited November 27, 2019 by Mefistotelis

[Rolf Kalbermatter]

On 11/28/2019 at 12:36 AM, Mefistotelis said:

A good example I just found:

void InitButtonsHidden(INSTRUMENT **a1)
{
  (*a1)->sr_field14 = gButtonsHidden[(*a1)->sr_field22];
}

Now we can rename one field to 'buttonsHidden', and can assume another field contains enum value representing either state or type.

The word hidden in there makes it look very exciting but the reality is that it is simply the toolbar buttons that are made invisible in the VI Properties. Apparently there is an enum that describes all possible permutations with individual enum labels and that enum gets also saved to disk but for performance reasons they decided to rather have a bitmask field to use at runtime. So this "routine" converts from the enum to the bitfield. There should most likely be another routine that does the reverse, most likely by running through the gButtonsHidden global array and comparing sr_field14 with its values and storing the found index in sr_field22, or more likely that routine is directly called in the SaveVI routine to generate the right enum value to store in the SAVERECORD version.

Why so complicated you may ask? Well the backsaving feature makes all kinds of complications necessary in order to be able to store VI files in older formats. 

[Mefistotelis]

2 hours ago, Rolf Kalbermatter said:

Apparently there is an enum that describes all possible permutations with individual enum labels and that enum gets also saved to disk but for performance reasons they decided to rather have a bitmask field to use at runtime.

Nope. that's not what I found.

You're right with ButtonsHidden field being an integer which bits correspond to something (I assume visibility of some buttons), but sr_field22 turned out to be something else - the type of VI. This type is actually part of LV documentation:

https://zone.ni.com/reference/en-XX/help/371361R-01/lvprop/vi_vi_typ/

I already did proper renaming in my code.

 

There seem to be many properties which I could find in code based on the documentation (click "VI Properties" in the link above to see them). Below, they are listed with "property ID" of each - will check whether this can provide me with some more field names:

https://labviewwiki.org/wiki/VI_class

 

[dadreamer]

Just to clarify, gButtonsHidden field is not only those settings, that you could see in VI Properties -> Window Appearance -> Customize dialog and could manipulate with scripting properties like Tool Bar:Show Run Button etc. There are some additional VI flags also to show or hide the toolbar buttons, they're called HiddenButtonsFlags in Heap Peek (buttHid for LV2013 and earlier versions). You may investigate them on this sample VI. As you can see, the VI has Pause button hidden completely and you cannot show it with VI Properties entries, even if you set "Allow debugging" checkbox. Heap Peek displays, that HiddenButtonsFlags equals to 0x348.

Honestly I don't know, why NI made that VI to have Pause button hidden, but I know, that LabVIEW stores the buttons' display state in HiddenButtonsFlags e.g. when switching to Subroutine Priority or vice versa. Sadly I have not found a way to alter all these flags from the diagram (using scripting or some other tools), so if one wants to modify the flags, (s)he has to manipulate the memory. In Heap Peeek's VI Fields there's the VI pointer in form vi=[some value]. Let's go there with any debugger of our choice.

Now go to the address at the "blue" offset (I'm not saying concrete numbers, because they could change between platforms, bitnesses, LV versions, weather or who knows, what else; I was testing on LV2019 32-bit on Windows). And here we should have our HiddenButtonsFlags equal to 0x348.

(The mem block is actually 2-bytes long and I'm too lazy to remake a screenshot).

Now if we set that memory block to zeros, we'd have all the toolbar buttons shown (the debugging should be set earlier in the VI Properties).

Such a task could even be done easier with some utility, which is able to search integers in memory and filter them in stages (ArtMoney or similar).

Edited June 19, 2020 by dadreamer
a little rephrasing + small addition

[Mefistotelis]

@dadreamer if you're often looking at the binary data, you might have some use of "print-map" function of my readRSRC script:

./readRSRC.py --print-map=RSRC -x -i 'test_out/lv10/vi.lib/Utility/Convert RTD Reading (waveform).vi'
test_out/lv10/vi.lib/Utility/Convert RTD Reading (waveform).vi: Warning: Block b'PRT ' section 0 size is 152 and does not match parsed size 128
test_out/lv10/vi.lib/Utility/Convert RTD Reading (waveform).vi: Warning: Block b'VICD' section 0 XML export exception: The block is only partially exported as XML.
00000000: RSRCHeader[0] (size:32)
00000020: BlockData (size:13559)
00000020:   BlockSectionData[LVSR,0] (size:140)
000000AC:   BlockSectionData[RTSG,0] (size:20)
000000C0:   BlockSectionData[OBSG,0] (size:20)
000000D4:   BlockSectionData[CCSG,0] (size:20)
000000E8:   BlockSectionData[LIvi,0] (size:52)
000000EC:     Block[LIvi,0].LinkObject[0].NextLinkInfo (size:2)
000000EE:     Block[LIvi,0].LinkObject[0].Ident (size:4)
000000F2:     Block[LIvi,0].LinkObject[0].Unk1 (size:34)
00000114:     Block[LIvi,0].LinkObject[0].Unk2 (size:2)
0000011A:     Block[LIvi,0].LinkObject[1].NextLinkInfo (size:2)
0000011C:   BlockSectionData[CONP,0] (size:6)
00000124:   BlockSectionData[TM80,0] (size:64)
00000164:   BlockSectionData[DFDS,0] (size:388)
000002E8:   BlockSectionData[LIds,0] (size:52)
[...]
00003A44:   BlockSectionStart[VCTP,0] (size:20)
00003A58:   BlockSectionStart[FTAB,128] (size:20)
00003A6C: NameStrings (size:34)
00003A6C:   NameOfSection[LVSR,0] (size:34)

When you use "--print-map=RSRC" it prints what is stored at which offset of the RSRC file. Obviously it can't print compressed data this way, so for compressed sections you can map the specific section, ie. "--print-map=DFDS". And to get BIN file which matches this mapping, you can extract the VI with "-d" instead of "-x" - then you will get uncompressed BIN files for all sections, instead of some being converted to XML.

[dadreamer]

On 6/17/2020 at 4:08 PM, Mefistotelis said:

"print-map" function

Thanks, I will take a look, when will have fun studying any VI internals again. Could I request one more feature, if possible? It would be very nice to have a support for in-place sections modification (e.g., type, id or binary contents), without unpacking into .xml and packing back (like in flarn's utility). I assume, the checksums should be recalculated, if dependent sections are altered, as it's already done for the password option. That would save time on simple binary operations.

Meanwhile I was going to make a VI to show/hide those toolbar buttons, just for fun. Reality shows that the offsets in VI's memory are varying vastly between different LV versions, bitness and IDE/RTE mode. So likely I won't be posting that. I just put this little picture, so you could name ViBhBit3 and ViBhBit4 bits in ButtonsHidden field. I suppose the rest of the bits is for reserved purpose and does nothing, but I'll recheck on older LV versions and update this posting.

Now closer to your code:

class VI_BTN_HIDE_FLAGS(enum.Enum):
    """ VI Tool Bar Buttons Hidding flags
    """
    RunButton = 1 << 0	# Indicates whether to display the Run button on the toolbar while the VI runs and when it's in edit mode as well. In LV14: Customize Window Apearence -> Show Run button. When off also hides Run Continuously button.
    SetBPButton = 1 << 1	# Set Breakpoint button (LV 3.x and earlier).
    StepIOButton = 1 << 2	# Step Into/Over button (LV 3.x and earlier). When on the button is shown, but disabled (inactive).
    PauseButton = 1 << 3	# Indicates whether to display the Pause button on the toolbar while the VI runs and when it's in edit mode as well. Not implemented in LV as a separate GUI setting.
    DebuggingButton = 1 << 4	# Indicates whether to display the Highlight Execution, Start Single Stepping (Step Into, Step Over) and Step Out buttons on the toolbar while the VI runs and when it's in edit mode as well. Not implemented in LV as a separate GUI setting.
    FreeRunButton = 1 << 5	# Indicates whether to display the Run Continuously button on the toolbar while the VI runs and when it's in edit mode as well. In LV14: Customize Window Apearence -> Show Run Continuously button. The button isn't shown, if Run button is hidden.
    LogAtCompButton = 1 << 6	# Log at Completion button (LV 3.x and earlier).
    AbortButton = 1 << 7	# Indicates whether to display the Abort Execution button on the toolbar while the VI runs and when it's in edit mode as well. In LV14: Customize Window Apearence -> Show Abort button.
    ViBhBit8 = 1 << 8	# unknown
    PrintAtCompButton = 1 << 9	# Print at Completion button (LV 3.x and earlier).
    EditRunModeButton = 1 << 10	# Change to Edit/Run Mode button (LV 3.x and earlier).
    ViBhBit11 = 1 << 11	# unknown
    ViBhBit12 = 1 << 12	# unknown
    ViBhBit13 = 1 << 13	# unknown
    ViBhBit14 = 1 << 14	# unknown
    ViBhBit15 = 1 << 15	# unknown

---

Here's my "dirty"/hacky tool to show and hide all the toolbar buttons: But_Show_Hide.vi Tested that on LV 2009 to 2020 (both 32- and 64-bits). Maybe it will work on anything newer than LV 2020 in the future, nobody knows yet. It won't work on versions older than LV 2009, because the difference between LV 2009 and LV 8.x is way too large. And I will not be maintaining this tool at all, as it's (almost) useless for anyone including me. Use it as is or forget it.

Small update:

In LV 3.1.1 and earlier ones there was no separate Tools palette, all the tool buttons were on the toolbar. Here's how it looked for stopped VIs:

This was for running VIs:

Button 2 ("Pencil with Run arrow") was for switching between Edit and Run modes, button 5 ("...") was to set a breakpoint on the VI, button 6 ("___") was to pause the running VI, buttons 8 and 9 ("Run arrow with file/floppy") were to enable logging or printing respectively at the VI completion. Sure you know the rest. When pressed button 5 was turning into "!" button (breakpoint was set). When pressed button 6 was turning into "Square wave" button (to unpause the VI) and was showing "Single square step" button (to step into/over the VI nodes). Here's how those buttons looked pressed:

I've updated the corresponding bits in my tool to reflect those early buttons too, even though the tool won't work on anything lower than LV 2009. The source of pylabview might be updated accordingly.

Edited July 24, 2020 by dadreamer
referring to pylabview + attaching buttons tool + tool update

[Mefistotelis]

3 hours ago, dadreamer said:

Could I request one more feature, if possible? It would be very nice to have a support for in-place sections modification (e.g., type, id or binary contents), without unpacking into .xml and packing back (like in flarn's utility).

Well, that's really against what I want to achieve.. my idea is to extract everything to XML and allow users to do all the modifications in XML form. Then you can easily re-create the VI. Note that the re-created VI will, in most cases, be identical to the original (there are few exceptions, as LV uses multiple threads to save the file and is generating names section in pseudo-random order as a result, and in older versions of LV even the section data is ordered randomly).

The password option is the only exception to that rule, and I'm thinking about removing it completely. It's untested anyway, so I'm not even sure if it works.

For the in-place modifications, you'd have to modify  the tool, and make the modifications you want through Python code.

3 hours ago, dadreamer said:

I assume, the checksums should be recalculated, if dependent sections are altered, as it's already done for the password option. That would save time on simple binary operations.

All the known checksums are re-calculated when importing from XML. These checksums are not even put into XML, as they're just derivative values which can be re-created.

The XML import of a single file takes seconds. Currently I'm testing everything on _all_ RSRC files from 4 versions of LV (6.0, 7.0 10.0 and 14.0), that's over 41 000 of files. Exporting and then importing all of these files takes less than 24 hours. That means export+import of one file takes on average 2 seconds.

  

3 hours ago, dadreamer said:

Meanwhile I was going to make a VI to show/hide those toolbar buttons, just for fun. Reality shows that the offsets in VI's memory are varying vastly between different LV versions, bitness and IDE/RTE mode.

Did you tried 'Cheat Engine'? It's a tool for cheating in games, which basically means - a tool to edit memory of application. Exactly what you want.

And it has a great ability to "find pointer chain" - it allows locating pointers to dynamically allocated memory. If the issue you have is dynamic allocations, and not variable shift within allocated blocks, that might be useful.

 

 

 

Edited June 17, 2020 by Mefistotelis

[dadreamer]

1 hour ago, Mefistotelis said:

Well, that's really against what I want to achieve..

Okay then. Your software - your rules. 🙂 I'm just worried about some cases with it, so I'm going to ask.

1. As I'm mostly work on modern LabVIEWs (2018, 2019, 2020) and your tools weren't tested on anything higher than LV 2014... What is the worst thing that can happen, when I try to unpack/pack such VIs? Could those be not fully unpacked or packed? Or something got corrupted?
2. Is it safe to ignore the frequent warnings on VI (un)packing? There are always few of them, e.g. Block b'VICD' section 0 XML export exception: The block is only partially exported as XML. , Block b'VICD' section 0 binary prepare exception: Re-creating binary is not implemented. , Block b'VICD' section 0 left in original raw form, without re-building.
3. Sometimes I saw the message "No matching salt found by Interface scan; doing brute-force scan", when packing back VI w/ some sections slightly modified. It then leaves me waiting to get the process finished (honestly, my biggest record is 10 minutes 😄, I always interrupted it). How/what could I do to escape that? Is there some option to force the recalculation, like in flarn's utility (fixing checksums)?

1 hour ago, Mefistotelis said:

The password option is the only exception to that rule, and I'm thinking about removing it completely. It's untested anyway, so I'm not even sure if it works.

It works fine even on the recent LabVIEW 2020.

1 hour ago, Mefistotelis said:

For the in-place modifications, you'd have to modify  the tool, and make the modifications you want through Python code.

Do you mean rewriting pylabview for my needs? Even if I, say, want to rename some section? I imagined that as few additional parameters, e.g. "section" and "new section", and that's all to do the job. And no need to repack everything. Ok, how could I easily rename some section, having all the VI's files already extracted?

1 hour ago, Mefistotelis said:

The XML import of a single file takes seconds.

I don't worry about the times at all now, as it's not for my work tasks. I experiment with that mostly for my own purposes.

1 hour ago, Mefistotelis said:

Did you tried 'Cheat Engine'?

I'm aware of tools like that and use some. Thx for the advice anyway. This is not a question of finding some proper addresses in memory for me, it's all about universality. 😉 In the past I already made some plug-in, that was relying on the internal memory offsets. It was a total pain in the ass to find and code correct offsets for each and every LV version. And that was even much more pain with each new LV version, because there were absolutely new offsets. So it was requiring a large amount of time to provide versatility. Finally I gave that up years ago. Since then I hate hard-coding variables based on unreliable internal knowledge, such as memory locations etc. That may change and changes rapidly and all the code goes to trash. I prefer not to write such programs at all.

[Mefistotelis]

1 hour ago, dadreamer said:

As I'm mostly work on modern LabVIEWs (2018, 2019, 2020) and your tools weren't tested on anything higher than LV 2014... What is the worst thing that can happen, when I try to unpack/pack such VIs? Could those be not fully unpacked or packed? Or something got corrupted?

The worst, but really really rare case, would be for the tool to create damaged binary. But I'm doing a lot of checks to avoid that. And you already stumbled upon some of the checks:

1 hour ago, dadreamer said:

Is it safe to ignore the frequent warnings on VI (un)packing? There are always few of them, e.g. Block b'VICD' section 0 XML export exception: The block is only partially exported as XML. , Block b'VICD' section 0 binary prepare exception: Re-creating binary is not implemented. , Block b'VICD' section 0 left in original raw form, without re-building.

The tool does a lot of checks and raises exceptions if anything looks out of the ordinary. The exceptions are then captured and the block which raised them is exported as a binary file, without trying to make it XML. In case of VICD you'll actually always see this, as I didn't published parsing of the data. This means the VICD will just always stay as binary.

If you want to be sure the data is identical to source, just go with:

./readRSRC.py -vv -x -i ./lv10/vi.lib/dex/DexPropertyNode.vi
./readRSRC.py -vv -c -m ./DexPropertyNode.xml
cmp -l ./lv10/vi.lib/dex/DexPropertyNode.vi ./DexPropertyNode.vi

In other words - export the file, then re-create the binary without changes, and compare both binaries.

1 hour ago, dadreamer said:

Sometimes I saw the message "No matching salt found by Interface scan; doing brute-force scan", when packing back VI w/ some sections slightly modified. It then leaves me waiting to get the process finished (honestly, my biggest record is 10 minutes 😄, I always interrupted it). How/what could I do to escape that? Is there some option to force the recalculation, like in flarn's utility (fixing checksums)?

The tool checks whether it can re-create the checksum from your file - after all, it always tries to re-create identical data after export and import. Brute-force scan will happen only on damaged files - where the checksum doesn't match using the usual algorithm. The tool then assumes the issue is in salt. If the tool can't figure out how to re-create the salt used for password, it will export the salt into XML, and won't re-compute it when re-creating binary (unless you modify the XML to re-enable auto-computation).

The tool will re-create everything correctly, if only you will modify it in exported form.

I could add an option which would make it assume the input file is damaged, and skip that scan.

1 hour ago, dadreamer said:

Do you mean rewriting pylabview for my needs? Even if I, say, want to rename some section? I imagined that as few additional parameters, e.g. "section" and "new section", and that's all to do the job. And no need to repack everything. Ok, how could I easily rename some section, having all the VI's files already extracted?

Well, you could use the same code which handles "change password", only replace password work with your fun.

But the way to rename blocks using my work model is:

1. Export VI to XML

2. Change the tag name in XML, for example replace "<CLIv>" and "</CLIv>" with "<LIvi>" and "</LIvi>".

3. Re-build the VI from XML

Yes, block idents are just the tag names; they are only a bit modified to meet XML standard, ie. "#" are replaced by "sh" and "\0" codes are just skipped; but the tool will re-create the 4-char ident from tag name during import.

  

1 hour ago, dadreamer said:

It was a total pain in the ass to find and code correct offsets for each and every LV version.

I see. Yeah, there's no way around that.

 

Edited June 17, 2020 by Mefistotelis