Slow FTP transfer to LV RT

[flintstone]

Hello,

since this is my first post here I start with a welcome to everybody!

Now for my problem: In our project for LV RT we use a LV framework that requires me to build the project as source distribution and save it to the PXI controller where it is loaded on next boot-up. So essentially I have now a folder containing ~2500 VIs totalling to ~50 MByte and I want to load it to a 8133 controller via FTP. The problem is that the transfer starts at acceptable speed (>150 KByte/s) but becomes incredibly slow after the first ~1000 files are transferred (< 5KByte/s). I have tried up to now with three different FTP clients (Filezilla, WinSCP and the FTP client in LV MAX) and from two different machines. It seems to be connected to the number of files in the target directory. I managed to upload the source distribution by manually distributing it to four target directories but now again moving the contents of the four directories to one on the server with WinSCP takes ages. Even if this would work doing this manually is really no solution as it is error-prone and way too slow.

Am I missing something here or could the number of files in the target directory really be such a performance problem. Any recommendations?

Best regards

flintstone

[ned]

Am I missing something here or could the number of files in the target directory really be such a performance problem. Any recommendations?

That's likely the problem. Filesystems in general do not deal well with huge numbers of small files. Also, there will be a substantial amount of overhead transferring that number of files over ftp since each file is a separate transaction.

[JamesMc86]

Hi Flinstone,

I have seen real time (although on VxWorks, not PXI) struggle with large directories before, if you can break it down this is probably best.

The only other thing I could suggest is maybe trying the file transfer through the web interface. If it was protocol related then this uses a different technique (something over HTTP) but if it is the number of files you will probably see the same thing again.

Cheers,

James

On trying to find a recommmended number I found this: http://digital.ni.com/public.nsf/allkb/C9B0A1443BF1C3398625760A004DB976?OpenDocument seems 'significant' means a few hundred files

[flintstone]

Thank you James for this link, this seems to be exactly the problem I am facing. During FTP upload it looks like the list command starts to time out at some point. So I guess I'll have to start thinking about reducing the number of files, e.g. by using libraries.

Regards

Flintstone

[Jordan Kuehn]

You could also look into diff'ing the versions and only transferring the modified files. You would have a better grasp of whether or not that would be worth your effort though.

[flintstone]

I thought about this but I guess that could slow down the performance of my application and of course I do not want that.

Cheers

Flintstone