Requirements Hatchway

[jcarmody]

I replied to a post here where I said

I pitched Requirements Gateway to my manager and he ignored me. As a result, I've written a small program that searches a VI for Free Labels like "[Covers ....] and brings you to that portion of the code. It gets a list of requirements from (for now a csv file) and calculates coverage based on the finding a label for each one.

Here's a screenshot of it in action. Clicking on a requirement in the list brings you to the label in the Top Level VI that demonstrates coverage. I'll work on a simple demo that I can post here this weekend.

I'm thinking about implementing reading requirements directly from MS Excel and I expect to optimize the search and extend it to include sub VIs (or, perhaps, set it loose on a directory).

Can you think of anything that would be good to include in this?

Would you ever use it?

[crelf]

Would you ever use it?

I'm not trying to rain on anyone's parade here (I think it's a good tool and you've obviously put a lot of work into it), but since you asked... Did you develop it to a documented ISO audited and approved process? I need something that I know the vendor used an approved quality process, so I can be confident that the artifacts that it produces are complete and correct, and my customers can too. That's why I need to stick with Requirements Gateway for the time being - my customers can defend it because I can defend it because NI can defend it Also, we do a lot of TestStand work and RG can dive into that too...

[jcarmody]

you've obviously put a lot of work into it

Not so much, really. (It's built upon the JKI State Machine, which makes cranking out solid a proof-of-concept very easy. ) It began as a fun attempt to use the LabVIEW API and grew.

but since you asked... Did you develop it to a documented ISO audited and approved process?

No.

I need something that I know the vendor used an approved quality process, so I can be confident that the artifacts that it produces are complete and correct, and my customers can too. That's why I need to stick with Requirements Gateway for the time being - my customers can defend it because I can defend it because NI can defend it

It would be great if, by demonstrating this tool, my managers would see the need for Requirements Gateway. I hope to learn more about this whole aspect of Software/Test Engineering because we don't do this here. I'm going to use this to give myself a warm/fuzzy feeling that I'm ready for SQA.

Also, we do a lot of TestStand work and RG can dive into that too...

That might be a problem, but maybe not. I think your first objective will be the show-stopper for most people.

And another thing!!! My understanding (based on watching one of NI's videos) is that Requirements gateway looks for the same type of tag. But, finding one doesn't prove that the requirement is covered, it just brings you to the place where someone said it is. It still remains to be demonstrated that the purported requirement-covering code actually does what it's claiming to do, and that takes an auditor that understands the test language. No?

Anyway, thanks for the feedback. I didn't sound too defensive, did I?

Jim

Edited July 31, 2009 by jcarmody

[crelf]

It would be great if, by demonstrating this tool, my managers would see the need for Requirements Gateway. I hope to learn more about this whole aspect of Software/Test Engineering because we don't do this here. I'm going to use this to give myself a warm/fuzzy feeling that I'm ready for SQA.

Sounds like a good reason to be doing it. There are some good resources on the general software development models on NI's site, and Jeff Plotzke will be leading a presentation at NI-Week about this very topic that you might find interesting.

And another thing!!! My understanding (based on watching one of NI's videos) is that Requirements gateway looks for the same type of tag. But, finding one doesn't prove that the requirement is covered, it just brings you to the place where someone said it is. It still remains to be demonstrated that the purported requirement-covering code actually does what it's claiming to do, and that takes an auditor that understands the test language. No?

Oh totally, but what I was getting at was that RG was developed using an audited quality process, so NI can stand behind it in saying that it won't miss any of those tags (or create any new ones that don't exist). It's not about the tagging within the custom software that I write (that's up to me), it's about that the tool finds and traces those tags. The requirements I deal with aren't necessarily covered in one area, and one area can often cover more than one requirement, so I don't think it's practical to have anything otherwise.

Anyway, thanks for the feedback. I didn't sound too defensive, did I?

That depends - were you being defensive? No, you didn't sound defensive at all.

[jcarmody]

That depends - were you being defensive?

No, I'm just excited.

[Black Pearl]

Hey Jim,

that is brilliant.

If you would publish it for the community and in a version I could use, I would start using it and doing my part of improving it. Well, as soon as I find time...

Felix

[Michael Aivaliotis]

I've moved this thread to the "Code In-Development" forum. Sounds like another code repository candidate.

[Aristos Queue]

And another thing!!! My understanding (based on watching one of NI's videos) is that Requirements gateway looks for the same type of tag. But, finding one doesn't prove that the requirement is covered, it just brings you to the place where someone said it is. It still remains to be demonstrated that the purported requirement-covering code actually does what it's claiming to do, and that takes an auditor that understands the test language. No?

The NI Requirements Gateway interacts with the NI Unit Test Framework. It matches the tags in the code against the tags put on a given test against a given test requirement from the original spec document to show that a) the requirement was implemented in code b) there was a test written for that specific requirement and c) that test is passing.

[jcarmody]

The NI Requirements Gateway interacts with the NI Unit Test Framework.

I was thinking about that when I replied to Chris' post. One of the examples in the video gave an example of a requirement that a specified value had an indicator on the Front Panel. How can the Unit Test Framework validate that? A common requirement I'd have would specifiy a valid range for a measurement and the coverage would be code that tests for that. How would the UTF test that?

I'm going to have to dig into a trial of RG.

[jcarmody]

Here's something you can try. I just got it working this afternoon and haven't tested it enough since drastically changing the approach.

1. unzip the attachment into <LabVIEW>\project
   
2. restart LabVIEW
   
3. put a "requirements.txt" file in the folder with your LabVIEW project
   
   • the first line must be the name of the top-level VI (as a relative path from the project's folder)
     
   • the second line must be blank
     
   • the third and subsequent must be the requirement ID followed by a tab then the requirement description
     
   • put free labels in your VIs in the form of [Covers <id>]
     

• run Requirements Hatchway from the Tools menu
  
• profit
  

Tell me how it goes.

By the way - this is only in LabVIEW 2012 because I can't manage to save it to a previous version. All I get is a folder with its dependencies, but no main VI. I'll try from home in case this is an issue with my installation.

RequirementsHatchway.zip

[jcarmody]

It's not finding [Covers <id>] tags in dynamically dispatched VIs saved in an llb.

[Daklu]

"Can you think of anything that would be good to include in this?"

Well if you don't mind me addressing the question you asked 3+ years ago, I don't have a need for anything like Requirements Gateway, but I have often wished for a tool I could use to review all the developer notes I've dropped on block diagrams. I'll typically use braces around a keyword, like {ToDo}, {Refactor}, {Magic Number}, or {Fugly Hack}, then fill in the rest of the note with an explanation. The problem is I never remember what keywords I used and hitting Ctl-G until I find the note I'm looking for is a pain.

I realize it's not quite within the scope of what you're doing... Would you object to me poking around and possibly forking your code?

[ShaunR]

"Can you think of anything that would be good to include in this?" Well if you don't mind me addressing the question you asked 3+ years ago, I don't have a need for anything like Requirements Gateway, but I have often wished for a tool I could use to review all the developer notes I've dropped on block diagrams. I'll typically use braces around a keyword, like {ToDo}, {Refactor}, {Magic Number}, or {Fugly Hack}, then fill in the rest of the note with an explanation. The problem is I never remember what keywords I used and hitting Ctl-G until I find the note I'm looking for is a pain. I realize it's not quite within the scope of what you're doing... Would you object to me poking around and possibly forking your code?

You want one of these then

[jcarmody]

I realize it's not quite within the scope of what you're doing... Would you object to me poking around and possibly forking your code?

Nope. I don't mind.

You want one of these then

So do I.

[0_o]

Great work.

Thanks jcarmody!

I'm working on a version of mine (on my free time which is not a lot) and the JKI State Machine you implemented might make it a bit more elegant compering to the LVOOP I used.

I promise to post here when done.

I have a few suggestions for now:

1. Instead of writing [Covers xxxx] you could have a vi that reads through the requirements.txt and gives you out a ComboBox filled with the requirements and the structure of the message. All you'll need to do is concatenate the strings and thus avoiding spelling mistakes like writing covers (as you did in your post) instead of Covers as your pattern matching requires.

2. Your tool scans only the main vi while there should be a tree of requirements that are implemented inside any vi in the project on even in a parallel project of a plugin for example. Some requirements will be green without ever writing a label about them simply since all their sub tasks are done.

3. If you go and scan in this way via the vi server all the files in your projects you'll get into a very big problem: the vi server opens each vi in order to do a traverse! It will take you forever to scan everything. The proper way to do it is by adding a text message into the binary code of the vi since then you could simply search the binary code of each vi. The way I currently thought about implementing it is by creating an empty vi (stub - https://decibel.ni.c.../docs/DOC-22847) for each requirement in a separate project and adding this vi next to the label and the code that covers the requirement. Any vi appears as is inside the binary code of a vi. If there is something else that is added as a text into the binary code of a vi I would love it hear about it or if someone knows how to get label texts from the binary code of the vis by using zlib or what ever way it is encrypted by please do tell me. P.S. - this way instead of a requirements id we can use a name.

4. A requirement might be implemented in multiple locations. You might want to mark a list of vis and enforce them to implement a requirement and thus each vi will have a list/table of tasks and task status + time which is very important since you might want to compare between the time of update and the time of status change.

5. In theory, though I'm light years from it, the CI (continues Integration - automatic builds that are being done once the repository is changed) should run a requirements scan and give a report if any change is made or if a test shows that even though a requirement is marked as done the test failed or even run only tests for new finished requirements.

Beside 5 the rest are feasible I think. I hope to see a more in depth demonstration and some free automation code to integrate with Jenkins and then I'll be happy to the roof.

Edited September 12, 2012 by 0_o

[jcarmody]

It's not finding [Covers <id>] tags in dynamically dispatched VIs saved in an llb.

It's not finding [Covers <id>] tags in any SubVIs. I can't remember if this ever worked (it seems to me that it did) or if TRef Traverse for References.vi has changed in LV2012.

Anyway, this attachment finds the missed tags (in a most inelegant manner).

Dror, this addresses some of the issues you raise in item 2 of your post. Also, there aren't any mistakes in I fixed the capitalization in my other post.

RequirementsHatchway.zip

[Daklu]

You want one of these then

Yup, and I can get that where?

[ShaunR]

Yup, and I can get that where?

Right click. Save As

[jcarmody]

Here's a single VI (so you don't need the extra folder in <LabVIEW>\projects) that's been cleaned a bit.

ReqHatch.vi

Right click. Save As

Boo.

[hooovahh]

Right click. Save As

Put me down as another member who would love to see this tool.

You are welcome to do as you please, but why even post an image of this tool, if you are unable, or unwilling to share it?

[Tim_S]

You are welcome to do as you please, but why even post an image of this tool, if you are unable, or unwilling to share it?

"Barely imperceptible sadistic tendencies."

[GregSands]

"Barely imperceptible sadistic tendencies."

Don't you mean "Beautifully Admirable Sadistic Tendencies And Radical Diatribes"?

[ShaunR]

Put me down as another member who would love to see this tool.

You are welcome to do as you please, but why even post an image of this tool, if you are unable, or unwilling to share it?

Because, whilst it is OK for me to use as I know how to get around its foibles, it needs quite a bit of work to make it "prime-time". But the main part (the scan engine) is not dissimilar to JCarmodys. To get something similar, you could just spend a day or so writing it yourself since JCarmody has done most of the work for you .

Edited September 14, 2012 by ShaunR

[Daklu]

Because, whilst it is OK for me to use as I know how to get around its foibles...

Or you could post it to this forum and let others tinker around with it too.

I'm just sayin'...

[ShaunR]

Or you could post it to this forum and let others tinker around with it too.

I'm just sayin'...

It'd end up full of classes then-no thanks...lol.

I only release stuff that works and looks clean. I also don't believe in alpha or beta testing (it's just an excuse lazy people/companies use for free resource so they don't have to test it themselves!).

I had the intention of doing something with it (I've been using it for over a year now), It was a weekends "playing" that did the job at the time.However, now I'm engrossed with with the SQLite API (only one platform left.....woohooo!). Too many homers what with that, websockets and dispatcher. They are all far more mature than this and consume a considerable amount of my spare time. I probably won't get much time to look at it this side of Xmas (but you never know).

But come on Daklu. You could knock up something very similar in a day or two yourself with JCarmodys code if you really wanted it that badly All you need to do is to add a snazzy UI and a database (incidentally, it's not a total coincidence it happens to look very much like a database browser that you may have seen before ). If you want something now, then JCarmodys is your best bet. Why not tinker with that and help him productionize it? With you and JCarmody on the case; I'm sure it will knock the socks off anything I can produce. Otherwise we'll just have to see what Santa brings.

Meanwhile. Back at the ranch............