Slow transfer of large files with USB 2.0: solution

Yesterday my core facility, BGI, sent me two 500 GB harddrives (HDs) full of compressed Fastq datafiles. I wanted to make a back up of these before I started to work on them. I have two 2 TB HDs that I use for working and storing this data. So I wanted to use a computer that I had both plugged into to manage the transfer from the 500 GB drive over to one of the 2 TB drives.

Transfer time of a set of files 250 GB in size  = 130 hours .... and this kept getting larger and was up to 200 hours before I cancelled the whole thing.

My first thought was that the problem was that the USB 2.0 transfer speed is very, very slow.

For a table brakedown of transfer speeds, in both Mb / sec, and MB / sec (yes, they are very different, see the Comparison with other buses table (about half way down) on wikipedia.

http://en.wikipedia.org/wiki/Serial_ATA

USB 2.0 is 60 MB/s on a good day .....  so 250,000 / 60 = 69 minutes ... but I was getting much, much slower reads and transfer rates ..... why? because the drive was NTFS (Windows format) and my computer was Mac, and these two file formats don't play well together unless you upgrade the driver on the Mac (Paragon sells something like this for $20).

Solution:
Transfer speeds plugging the drive in to a Linux were about 40 MB/sec, so this takes a little less than 2 hours to transfer.

• 250,000 / 40 = 104 minutes