Recursive Directory Listing VI

[sam]

This VI give the directory listing of files and directories, it is also able to do a recursive listing. I was using the OpenG VI but that could not handle very large directories efficiently.

Any comments are welcome.

-Sam

Download File:post-10-1077208058.vi

[Jim Kring]

This VI give the directory listing of files and directories, it is also able to do a recursive listing.  I was using the OpenG VI but that could not handle very large directories efficiently.

Any comments are welcome.

-Sam

Hi Sam,

I was wondering if you are interested in helping to make the OpenG version more efficient. Perhaps you would want to contribute some of your improvements.

Cheers,

-Jim Kring

[Michael Aivaliotis]

Any comments are welcome.

It took me a while to grasp why your VI would be more efficient than other implementations. I have my own version that is more efficient than the OpenG version as well. I compared my approach to yours and noticed no performance benefit on several thousand files and directories.

I noticed that you are initializing an array of filepaths to 500 items. You resize and build an array of another 500 items at a time when you require it. This seems like an excellent approach however as i said, this didn't show up in the performance comparisons I did between my version and yours. When is the benefit evident?

I also noticed that Windows does some caching of directory access. This seems apparent when you run a recursive search for the first time regardless of what implementation you choose.

I've attached my version (LV7). Let's compare notes...

Download File:post-10-1077255628.vi

[sam]

Hi Jim,

Sure, I have benefited from OpenG and I am more than happy to give back.

-Sam

[sam]

hi Michael,

I really like your approach; for most of my algorithms (solutions) that I like, they are best implemented in a text language, that way recursive algorithms really shows their power. I am certain we take something

[Dean Mills]

Hi Mike, Sam, Jim

In my recursive directory listing I ignore Windows shortcuts. We had a Coop student accidentally create a a shortcut to the same directory the shortcut was in. Caused a good infinite loop and few hours to figure out why the software was filling all the memory. You can steal a prototype from the Tree example (called Directory Hierarchy in Tree Control.vi) in LabVIEW 7. The shortcut check is called Validate.vi.

Dean Mills

Meikle Automation

[Rolf Kalbermatter]

Hi Michael

I just took a look at your List Directory and thought there would be something to improve.

Look at the attached version. On my computer it is almost two times faster in enumerating large directory trees even if it is called first in the timing test to not give it advantage through Windows directory list caching.

Download File:post-10-1081496463.vi

[Michael Aivaliotis]

In my recursive directory listing I ignore Windows shortcuts.  We had a Coop student accidentally create a a shortcut to the same directory the shortcut was in.  Caused a good infinite loop and few hours to figure out why the software was filling all the memory.  You can steal a prototype from the Tree example (called Directory Hierarchy in Tree Control.vi) in LabVIEW 7.  The shortcut check is called Validate.vi.

Yes Dean this is a great idea. I took a look at the VI you were writing about. I have an issue however with the label that NI gave to the indicator. It should not be labeled "Shortcut or not a path?". That implies that the boolean will be true if it's a shortcut or not a path, which it is not. It would be better to call it "Valid Path?". If you look at the icon it says "Valid?" right on it. I know this is minor, but still unacceptable in my opinion. Either you apply standards to your coding or you don't.

In any case...

It seems simple enough to integrate this code into the Recursive Directory Listing VI. It would make sense to add it into Rolf's version since it appears to be the fastest. I guess a switch to turn link checking on or off would be nice too. I will take all the suggestions and enhancements in this topic and create a VI which can then be submitted to the OpenG library. Whoever has submitted versions or suggestions to this topic will be given credit. Personally I use the OpenG library quite a bit however I don't use certain VI's in it because they are not optimized. This is our chance to help out and create a VI that we can all use.

Since most of you who have posted here are already willing to share your code, I will assume that you agree with the OpenG licensing. However if you are not familier with it, please read it over here. You must agree to this license if you want to participate.

So, on that note, can anyone suggest any other improvements to the VI's listed in this topic? You can take a previously submitted VI and re-write it (like Rolf did) or create another approach. I will work on creating a test routine to provide performance checking. If you have suggestions on a test VI please let me know or just submit it here.

[Jim Kring]

So, on that note, can anyone suggest any other improvements to the VI's listed in this topic?

I will check to see how symbolic links (unix shortcuts) are handled in linux.

[sam]

One thing Michaels' code allows to do is to be able to add the capability to define file patterns.

If you pass anything but *.* or leave unwired you will not get back any Folder names.

I myself use file patterns.

I have changed mine and removed the FOR loop for the file names and brought the whole thing inside the WHILE loop and the same steps for folder is done on files.

I would post it again but your approach is much easier on the eyes.

-Sam

[Rolf Kalbermatter]

One thing Michaels' code allows to do is to be able to add the capability to define file patterns.

If you pass anything but *.* or leave unwired you will not get back any Folder names.

Well that is not entirely true. It's only true because you usually don't have folders with endings but if you name your folders for instance Folder.dir a search pattern of *.dir will return those folders. Of course I agree with you that looking for particular file patterns is a nice feature and I thought the OpenG version does that also. The problem with this is the requirement of the two List Directory Nodes for each hierarchy level which will almost take double the time than one node does.

I think with preallocating the arrays and such one can win another few percent but the real time spent is in the List Directory node.

[Michael Aivaliotis]

All this talk on Info-LabVIEW lately regarding recursive algorithms has prompted me to post a version of the Recursive directory listing VI that uses true recursion in LabVIEW. By this I mean using a VI that lists the directory structure of one directory and then have that VI call itself for any subdirectories it may find along the way.

I realize this setup creates a huge performance hit however I think it should be included in this posting for completeness so it can be compared to other implementations.

Enjoy.

Download File:post-10-1086381407.zip

[brian175]

I realize this setup creates a huge performance hit

Here's a modified version of Michael's VI that doesn't have a "huge" performance hit. The trick is to minimize the number of times that the VI calls itself (due to the Open VI reference call). On my computer the listing of the LabVIEW directory took the following times:

Michael"s VI 2399 (ms)

Rolf K's VI 478 (ms)

Michael's VI modified 552 (ms)

OpenG 6717 (ms)

The test VI as well as Michael's and Rolf's original VIs are included.

Brian

Download File:post-987-1100028178.zip

[Michael Aivaliotis]

Since the new OpenG Forums are up and running, I decided to close this thread and hand-off any further discussions on this topic to the new OpenG Forums. This topic thread is now closed. Please continue all further discussions on this topic here:

http://forums.openg.org/viewtopic.php?t=32