How to transfer large files between RT and Windows

[JustinThomas]

My application is a remote cRIO based data logger. I use the 9014 controller with 2GB memory. My total sampling rate is around 3MBps. I use a WLAN module to communicate between the cRIO and my PC. I have tried sending the data over TCP/IP during acquisition but I am not able to support the 3 MBps data rate between RT and Windows. So I have decided to go with local logging on the RT. As my test duration is small in the order of 20 to 25 minutes the memory on the controller is sufficient. I get file sizes of around 1.5 GB after each test. I then transfer the files using FTP to my Windows PC. This process is painfully slow. It requires more than 45 min for me to copy the files from RT to Windows using the Windows File Explorer. Using the internet toolkit FTP vis have not helped much. The time taken to tranfer the files in this case is also around 45 min.

The problem is for a 20 min test I need to wait for double to time after the test to analyse my results. I was looking for a more efficient way to perform this operation.

My thinking right now is to implement a TCP/IP based file copy module which would compress file sections on the RT before transmitting and decompress on the Windows PC. I was wondering if members can point me to some useful information on this method. Also any other suggestions regarding the implementation are most welcome

Regards,

Justin Thomas

[Chris Davis]

You don't specify which WLAN module you are using, so maybe it has the bottleneck. Have you tested the throughput at the location in question via another method? Have you tried a different FTP client (i.e. Filezilla )

[LAVA 1.0 Content]

I have also seen that the RT-FTP engine can be slow (even on PXI based, ethernet connections).

So setting up your own data-service might be the best idea.

Ton

[David Wisti]

1.5 GB / 45 min = 0.0333 GB/min or 555 KB/sec. I have never seen a file transfer with RT FTP of 555 KB/sec wired or wireless with my cRIO. I only get 450 KB/sec over a wired network.

[Neville D]

QUOTE (JustinThomas @ Aug 25 2008, 05:57 AM)

I get file sizes of around 1.5 GB after each test. I then transfer the files using FTP to my Windows PC. This process is painfully slow. It requires more than 45 min for me to copy the files from RT to Windows using the Windows File Explorer. Using the internet toolkit FTP vis have not helped much. The time taken to tranfer the files in this case is also around 45 min.

The problem is for a 20 min test I need to wait for double to time after the test to analyse my results. I was looking for a more efficient way to perform this operation.

If it takes 45 min with Windows ftp, it will probably take that long using any other method. Can you use Gig-E with your controller?

Have you tried chunking the files into smaller sizes? I'm not sure if this will make a difference, but worth a shot. Just copy over a few large files and see if that changes things.

Are you using binary files? (They might be smaller in size).

N.

[crelf]

QUOTE (Neville D @ Aug 25 2008, 03:48 PM)

Have you tried chunking the files into smaller sizes? I'm not sure if this will make a difference, but worth a shot. Just copy over a few large files and see if that changes things.

Good point - and it can work in the converse too - lots and lots of little files may take longer than one big file...

[JustinThomas]

QUOTE (Neville D @ Aug 26 2008, 01:18 AM)

Have you tried chunking the files into smaller sizes? I'm not sure if this will make a difference, but worth a shot. Just copy over a few large files and see if that changes things.

N.

My file sizes are around 50 MB. Earlier I tried with a single file but the transfer was not possible always. I get a timeout error on the Windows Explorer. I have not tried the same with LabVIEW. I tried chunking the files into smaller sizes around 5 MB each but did not get any significant improvement.

QUOTE (Neville D @ Aug 26 2008, 01:18 AM)

Are you using binary files? (They might be smaller in size).

N.

Yes I am using Binary files, basically logging raw data acquired from the cRIO modules

QUOTE (Neville D @ Aug 26 2008, 01:18 AM)

Can you use Gig-E with your controller?

N.

I don't think the cRIO controller has a GigE port its a 10/100 port. Anyways I need to use wireless so cannot think of GigE as an option

[Scott Carlson]

QUOTE (JustinThomas @ Aug 25 2008, 08:57 AM)

My application is a remote cRIO based data logger. I use the 9014 controller with 2GB memory. My total sampling rate is around 3MBps. I use a WLAN module to communicate between the cRIO and my PC. I have tried sending the data over TCP/IP during acquisition but I am not able to support the 3 MBps data rate between RT and Windows. So I have decided to go with local logging on the RT. As my test duration is small in the order of 20 to 25 minutes the memory on the controller is sufficient. I get file sizes of around 1.5 GB after each test. I then transfer the files using FTP to my Windows PC. This process is painfully slow. It requires more than 45 min for me to copy the files from RT to Windows using the Windows File Explorer. Using the internet toolkit FTP vis have not helped much. The time taken to tranfer the files in this case is also around 45 min.

The problem is for a 20 min test I need to wait for double to time after the test to analyse my results. I was looking for a more efficient way to perform this operation.

My thinking right now is to implement a TCP/IP based file copy module which would compress file sections on the RT before transmitting and decompress on the Windows PC. I was wondering if members can point me to some useful information on this method. Also any other suggestions regarding the implementation are most welcome

Regards,

Justin Thomas

Hey Justin --

Create small files.. Transfer the finished ones while you are measuring, that way, at the end of your 20-25 minute test, you have your data as well.

-Scott

[LAVA 1.0 Content]

Given that your hardware path is fixed, the best advice I can offer is to experiment with TCP/IP transfers and try different sized packets.

Using PC's and old 100 Mb ethernet I have read that people have attained 5Mb/second transfer rates when they found the right size packet (try 65,000 to start).

Ben