Jump to content
Sign in to follow this  
JustinThomas

How to transfer large files between RT and Windows

Recommended Posts

My application is a remote cRIO based data logger. I use the 9014 controller with 2GB memory. My total sampling rate is around 3MBps. I use a WLAN module to communicate between the cRIO and my PC. I have tried sending the data over TCP/IP during acquisition but I am not able to support the 3 MBps data rate between RT and Windows. So I have decided to go with local logging on the RT. As my test duration is small in the order of 20 to 25 minutes the memory on the controller is sufficient. I get file sizes of around 1.5 GB after each test. I then transfer the files using FTP to my Windows PC. This process is painfully slow. It requires more than 45 min for me to copy the files from RT to Windows using the Windows File Explorer. Using the internet toolkit FTP vis have not helped much. The time taken to tranfer the files in this case is also around 45 min.

The problem is for a 20 min test I need to wait for double to time after the test to analyse my results. I was looking for a more efficient way to perform this operation.

My thinking right now is to implement a TCP/IP based file copy module which would compress file sections on the RT before transmitting and decompress on the Windows PC. I was wondering if members can point me to some useful information on this method. Also any other suggestions regarding the implementation are most welcome

Regards,

Justin Thomas

Share this post


Link to post
Share on other sites

You don't specify which WLAN module you are using, so maybe it has the bottleneck. Have you tested the throughput at the location in question via another method? Have you tried a different FTP client (i.e. Filezilla )

Share this post


Link to post
Share on other sites

1.5 GB / 45 min = 0.0333 GB/min or 555 KB/sec. I have never seen a file transfer with RT FTP of 555 KB/sec wired or wireless with my cRIO. I only get 450 KB/sec over a wired network.

Share this post


Link to post
Share on other sites

QUOTE (JustinThomas @ Aug 25 2008, 05:57 AM)

I get file sizes of around 1.5 GB after each test. I then transfer the files using FTP to my Windows PC. This process is painfully slow. It requires more than 45 min for me to copy the files from RT to Windows using the Windows File Explorer. Using the internet toolkit FTP vis have not helped much. The time taken to tranfer the files in this case is also around 45 min.

The problem is for a 20 min test I need to wait for double to time after the test to analyse my results. I was looking for a more efficient way to perform this operation.

If it takes 45 min with Windows ftp, it will probably take that long using any other method. Can you use Gig-E with your controller?

Have you tried chunking the files into smaller sizes? I'm not sure if this will make a difference, but worth a shot. Just copy over a few large files and see if that changes things.

Are you using binary files? (They might be smaller in size).

N.

Share this post


Link to post
Share on other sites

QUOTE (Neville D @ Aug 25 2008, 03:48 PM)

Have you tried chunking the files into smaller sizes? I'm not sure if this will make a difference, but worth a shot. Just copy over a few large files and see if that changes things.

Good point - and it can work in the converse too - lots and lots of little files may take longer than one big file...

Share this post


Link to post
Share on other sites

QUOTE (Neville D @ Aug 26 2008, 01:18 AM)

My file sizes are around 50 MB. Earlier I tried with a single file but the transfer was not possible always. I get a timeout error on the Windows Explorer. I have not tried the same with LabVIEW. I tried chunking the files into smaller sizes around 5 MB each but did not get any significant improvement.

QUOTE (Neville D @ Aug 26 2008, 01:18 AM)

Are you using binary files? (They might be smaller in size).

N.

Yes I am using Binary files, basically logging raw data acquired from the cRIO modules

QUOTE (Neville D @ Aug 26 2008, 01:18 AM)

Can you use Gig-E with your controller?

N.

I don't think the cRIO controller has a GigE port its a 10/100 port. Anyways I need to use wireless so cannot think of GigE as an option

Share this post


Link to post
Share on other sites

QUOTE (JustinThomas @ Aug 25 2008, 08:57 AM)

My application is a remote cRIO based data logger. I use the 9014 controller with 2GB memory. My total sampling rate is around 3MBps. I use a WLAN module to communicate between the cRIO and my PC. I have tried sending the data over TCP/IP during acquisition but I am not able to support the 3 MBps data rate between RT and Windows. So I have decided to go with local logging on the RT. As my test duration is small in the order of 20 to 25 minutes the memory on the controller is sufficient. I get file sizes of around 1.5 GB after each test. I then transfer the files using FTP to my Windows PC. This process is painfully slow. It requires more than 45 min for me to copy the files from RT to Windows using the Windows File Explorer. Using the internet toolkit FTP vis have not helped much. The time taken to tranfer the files in this case is also around 45 min.

The problem is for a 20 min test I need to wait for double to time after the test to analyse my results. I was looking for a more efficient way to perform this operation.

My thinking right now is to implement a TCP/IP based file copy module which would compress file sections on the RT before transmitting and decompress on the Windows PC. I was wondering if members can point me to some useful information on this method. Also any other suggestions regarding the implementation are most welcome

Regards,

Justin Thomas

Hey Justin --

Create small files.. Transfer the finished ones while you are measuring, that way, at the end of your 20-25 minute test, you have your data as well.

-Scott

Share this post


Link to post
Share on other sites

Given that your hardware path is fixed, the best advice I can offer is to experiment with TCP/IP transfers and try different sized packets.

Using PC's and old 100 Mb ethernet I have read that people have attained 5Mb/second transfer rates when they found the right size packet (try 65,000 to start).

Ben

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.