LVSpeak - tell LV what to do

[Norm Kirchner]

I have taken a few moments to re-factor LVSpeak and get it ready for the masses.

For those of you who haven't heard me rant on about talking to LV I'll take a moment to give you a backstory.

Back when LV scripting was unveiled in ~LV 7.0, I realized that we could merge the Microsoft Speech to Text API and LV scripting to aid us all in the process of coding.

The idea sat idle until QuickDrop came out in LV8.6, and I couldn't not do it.

Now, w/ the aid of this package, some rusty nails and the Microsoft Speech SDK, you can command LV with your voice w/ things like...

[DISCLAIMER] This is not foolproof code. BUT I use it on a daily basis (actually I refuse to program w/out it turned on)

Directions:

Download the Microsoft Speech SDK 5.1 and install it

Download and install the attached package to this post.

Restart LV and now in your 'Tools' menu, you should see Enable LVSpeak.

After you click it you will see two windows now floating whenever LV is active (untill you restart).

The commands that it accepts currently are shown in the image above.

[NOTE] They do not all work on both FP and BD. ex Alignment commands only work on the BD.

[NOTE] This does not include integration w/ Quick Drop (although I would expect it to be added soon)

I would recommend starting out w/ Saying things like

'V I Properties' ----------(both FP and BD)

'Align Left'----------------(on the BD)

'Graph Pallette'----------(on a FP graph)

To stop the programs, simply say "LabVIEW Speak Stop"

'LVSpeak Engine' simply sets up the link between LV and the MS Speech API

'Quick Edit' is the engine that does the editing (This is where the fun happens)

[Norm Kirchner]

Tutorial video

http://screencast.com/t/0HetuUrAGu

[TobyD]

QUOTE (Norm Kirchner @ Mar 29 2009, 08:30 PM)

Tutorial video

http://screencast.com/t/0HetuUrAGu

This looks really cool Norm. I don't have time to test it now, but I'm looking forward to playing with it.

-Toby

[Neville D]

Very cool Norm!

dunno what my co-workers in our cubicle-farm (all C guys) will think with me muttering commands all day..!

Neville.

[Norm Kirchner]

QUOTE (Neville D @ Mar 30 2009, 02:20 PM)

Very cool Norm!

dunno what my co-workers in our cubicle-farm (all C guys) will think with me muttering commands all day..!

Neville.

I'll tell you what they will think. They will wonder what the F is going on, and then you show them that your using voice commands to make your programming go faster and they'll all go sweeeeeeet.

To be honest.... just show them the get pizza command in action and that should suffice.

[JustinThomas]

Hey Norm that actually looks pretty cool. Can't wait to get my hands on it. When r u going to get it uploaded?

[Norm Kirchner]

Use in good health WITH NO WARRANTY OF USAGE.

This uses unsupported and possibly buggy features of LabVIEW that may or may not corrupt your code. (but I use it every day and so far haven't encountered any corruption issues)

Some commands operate on the premise of the selected objects, (alignment stuff)

Other commands operate on what you are hovering over (Create Control)

Other commands are context insensitive (show project , get pizza)

The ability to do "Create Constant, Create Control and Create Indicator is now enabled!!!! Yup, it's that damn cool

THIS IS NOT SUPPORTED BY NI

let me repeat

THIS IS NOT SUPPORTED BY NI

This is something that I was working on well before I got into the walls. This is in the hands of the community and we'll support it and tear it down/apart as needed

Most commands are for the block diagram at current but are as follows

BD:

Label Visible / Invisible

Align (left, right, top, bottom, horizontal center, vertical center

VI Properties

Create Control / Constant / Indicator

Show Project

Get Pizza

FP:

Label Visible / Invisible

VI Properties

Show Project

Graph Palette

Replace

Get Pizza

[gleichman]

QUOTE (Neville D @ Mar 30 2009, 02:20 PM)

Very cool Norm!

dunno what my co-workers in our cubicle-farm (all C guys) will think with me muttering commands all day..!

Neville.

Maybe Norm could update it to use mind control like the wheelchair demo at NIWEEK 2007. :thumbup:

[Norm Kirchner]

QUOTE (gleichman @ Apr 1 2009, 01:37 PM)

Maybe Norm could update it to use mind control like the wheelchair demo at NIWEEK 2007. :thumbup:

I already contacted them and I'm waiting to hear back.

[LAVA 1.0 Content]

QUOTE (Norm Kirchner @ Apr 1 2009, 02:09 PM)

I already contacted them and I'm waiting to hear back.

be careful, perhaps they will contact you using there mind.

[Norm Kirchner]

ADVANCED AND BRAVE USERS ONLY

In other words, if this messes something up.... Don't call NI complaining.

Really the only thing that has the chance to break is the fact that this package replaces the installed Quick Drop VI (but it does make a backup)(or at least that's the way the package is setup)

Addon :: Voice Integration with Quick Drop

Download File:post-208-1238696190.ogp

CORE DEPENDENCY!!! (NEW VERSION!!! Must Install)

Download File:post-208-1238696196.ogp

[crelf]

The "get pizza" command is freaking hillarious!

[Norm Kirchner]

Finding out that XP is not liking the loading of the QuickDrop Grammar (the list of things that you can say)

So don't be surprised if it's not working in XP. (Stay tuned though)

[TobyD]

I'm in love Norm. This is great! I just have a crappy desktop microphone and it picks up every command. I was talking to a coworker and happened to say "replace" during my normal conversation and up pops the dialog box to choose a new control.

Very cool! :thumbup:

[Norm Kirchner]

QUOTE (TobyD @ Apr 3 2009, 12:20 PM)

I'm in love Norm. This is great! I just have a crappy desktop microphone and it picks up every command. I was talking to a coworker and happened to say "replace" during my normal conversation and up pops the dialog box to choose a new control.

Very cool! :thumbup:

Actually, that's how Get Pizza came to life. I was getting heckled by my co-workers and they kept running by screaming 'Get Pizza'.

So I just made it actually do something. Now if I could only get their credit card #s it would really make them stop the harassment.

But I've found that a headset w/ a mic mute switch has helped. Actually I'll probably be making a foot switch if the ambient noise or hecklers gets too annoying.

To be fully honest, the previous version of my code, on vista, would enable the full vista speech system that had the capacity to close windows, hit keyboard keys, or do other system functions. Which is cool, but if you forget to shut the microphone off and turn around and have a conversation, you will find that your pc has been doing stuff since you walked away (like closing or opening windows you don't want closed or open)

The current version bypasses the Vista Environment Speech integration.

Actually if you do leave the voice system in Vista open and leave your laptop running next to a TV running a Bollywood movie, you'll find that it starts to outsource your coding (although the code is not documented and sloppy)

[mje]

Having it only process commands when a configurable key is pressed would be sweet. Hold down ctrl to speak, for example.

[Norm Kirchner]

Give it a whirl by getting into the Callback VI in the LVSpeak engine.

I would say use the User Input VIs and either fire or don't fire the user event if the key is pressed.

Try it out, let me know if you like it, I suppose we could make it a configurable option.

[Wulfi]

Not Found

The requested URL /old_files/post-208-1238696196.ogp was not found on this server.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

Apache/1.3.41 Server at lavag.org Port 80

Can somebody upload it again?

Thanks

[Michael Aivaliotis]

Not Found

The requested URL ...post-208-1238696196.ogp was not found on this server.....

Can somebody upload it again?

Thanks

I fixed the link, click on it again.

[Daklu]

Very cool Norm. Thanks for sharing that with us! Couple points/questions:

• Do you have an online source code repository or should we just make changes and post them here for review? (I think this tool would benefit from a plug-in architecture.)
• TRef Traverse for CursorContext.vi and TRef LocatorCallback.vi are both password protected. Just pointing that out in case it was unintentional.
• The Quick Drop package doesn't have the core package listed as a dependency.
• Why did you distribute it as an llb? I've noticed lots of tools continue to be distributed that way even though NI has depreciated them.

[Wulfi]

Thanks for relinking.

I've got one question. In my LV (8.62 Dev. and Scripting VI) I can't find the "GetActiveViRef".

I can copy it from your VI and It works fine, but why can't i find it in my PullDown Menu when I klick on it?

[Norm Kirchner]

Very cool Norm. Thanks for sharing that with us! Couple points/questions:

• Do you have an online source code repository or should we just make changes and post them here for review? (I think this tool would benefit from a plug-in architecture.)
• TRef Traverse for CursorContext.vi and TRef LocatorCallback.vi are both password protected. Just pointing that out in case it was unintentional.
• The Quick Drop package doesn't have the core package listed as a dependency.
• Why did you distribute it as an llb? I've noticed lots of tools continue to be distributed that way even though NI has depreciated them.

Code Repository: Should be arriving in the LAVA CR soon (less than a week)

Password protection: intentionally done at a time before scripting was not black market material. Uses some scripting stuff none of us had done before (thanks dnatt) and thought it prudent

Dependency: Because I'm using the OpenG Package Builder I have never been able to get the dependency enforcement working properly. Simply giving it the expected package name and version never did squat for me. (probably operator error)

LLB distribution: I have never made a "tools Menu" app before and this looked like the most straight forward way to get the kind of behavior I wanted. And wrt LLB's, I believe (not because I know) that they are not going to be disallowed any time soon so it was still a concise way of keeping a group of files together for the tools menu and simplified the packaging process which I'm still relatively new at.

I'll let DNatt make a nugget one of these days on the traverse code (or take it on if he deems me able to do so)

Thanks for relinking.

I've got one question. In my LV (8.62 Dev. and Scripting VI) I can't find the "GetActiveViRef".

I can copy it from your VI and It works fine, but why can't i find it in my PullDown Menu when I klick on it?

This is a 'Private' event and must still be hidden under the "RUSTY NAIL" clause of these things. I have never seen any specific documentation on it's pitfalls, perils and usages but I know for darn sure that all of my scripting tools rely on it's operation from what I have been able to understand of it. Let's hope it doesn't change functionality or go away because any tool that needs to know what was the last LV thing touched would then need to revert to a TopMostVI checking style routine which is a bit of a pain and still might not work for this kind of implementation.

If you want to know about the Private methods/functionalities, head to the rusty nails section to get the proper ini key.

Enjoy!

Having it only process commands when a configurable key is pressed would be sweet. Hold down ctrl to speak, for example.

I'm not sure I have explicitly state it yet, but I quickly found this also a requirement.

EVERYONE: FYI

In order to have a command detected you must hold down the control key and not just while you are speaking but for a fraction of a second after as well. The Microsoft API is constantly translating and only LV does the check "is Ctl pressed" after it gets the signal from the MS SAPI that something had been detected.

[EstarTroy]

Norm,

Loved the demo you gave when you were on site at one of LV User Group meetings. Been wanting to do this and have just now had some time to try this.

Being a little slow on the up take here, but which files do I need to download (now that it appears updates have been made since original post) and what other packages (rusty nails I believe you called them) do I need if any.

This is my first attempt to add non-NI supported product to my development platform.

[Norm Kirchner]

Being a little slow on the up take here, but which files do I need to download (now that it appears updates have been made since original post) and what other packages (rusty nails I believe you called them) do I need if any.

Ok, was just about to detail it all out, but head to

http://eyesonvis.blogspot.com/2009/07/lvspeak-automating-vi-development.html

and the newest info/details are on that there.

and once that's done come back here w/ questions

Thanks,

Norm

[Yair]

I have to say that I like the idea ever since I first saw Norm's demo of it last year, but I haven't used it all due to not wanting to install the speech recog utility or speaking all the time in the office.

Another thing which I think is a major hinderance is the list of options. I think that what QE needs to really take off is a smart parser. This is certainly a tricky bit, but also important, due to the following improvements it could allow:

1. Writing less code for the framework. To demonstrate this, think of the command "Label visible". Today, it has its own specific VI.
   In the ideal version, the parser would recognize the structure, use the word "label" to get a reference to the label of the object you selected and then pass that reference into the "visible" VI (which was only implemented once, for the GObject class). That would mean that you only have to write the code for each action once. Now that you've written it, "visible" would work all objects, their labels, their captions, etc.
2. The parser could allow parameters. For example, "caption align left" would obtain the reference to the caption, then call the "align" action and pass "left" as a parameter. Immediately you have only one action instead of several.
3. Synonyms - imagine if you could say "label visible" or "visible label" or "show label" or "label show", etc. That would make it a lot more "natural", although I admit it's a secondary objective.

As I said, writing such a parser would be tricky. It's possible that maybe an initial version could use some explicit vocal separators to break the different parts of the text.