A) Is this in LV 2010?
B) Does it involve a library that contains another library? (The inner library can be any type of library -- lvlib, lvclass, xctl, lvsc.)
If the answer to both of those questions is "yes," then it's known and should be fixed in LV 2010 SP1. If not, please post a reproducing example on ni.com so it can be evaluated and hopefully fixed.
Good luck indeed. INCA is a very feature complete ECU monitoring/calibration/firmware changing application that does just about every thing dealing with many kinds of vehicle control modules. The UI leaves alot to be desired with menus that are unintuitive and functions that take alot of training to fully understand.
That being said I believe there are ActiveX calls into INCA, but good luck understanding them. With the software it self being as hard to use as it is I can only imagine how difficult it will be to be able to get the functions you need.
Generally people only want to use a very small subset of the functions of INCA. If this is the case with you I would suggest maybe looking into if you can do the functions INCA does nativily in LabVIEW. There does exist a ECU Measurement and Calibration add on for LabVIEW that allows you to read XCP and CCP (possibly KWP2000) messages over the standard CAN interface. This means that with a $200 (estimate) USB CAN device from NI, and this tool kit you can measure and write to XCP and CCP messages.
So I'd be curious to know what functions of INCA you need. Most likely you will not be able to communicate with INCA to get the functions you need and either will reinvent the wheel for LabVIEW, or just use INCA as it was intended.
Also is there a reason you aren't using the latest INCA which I think is in the 6.x version? The newer the version the more likely that the ActiveX functions you need exist.