FTP or NFS for large number of files?
What is the best option between FTP and NFS if I have 1 million files (~1 TB) to transfer? I want to make sure that files don't get corrupted during the transfer.
21 Answer
FTP tends to be problematic, especially with its text file features, it's not a remote filesystem implementation. But for simple copying of a large set of files I would also recommend rsync.
For moving the data over the network, rsync is the way to go. If you get disconnected or any other problem occurs, rsync will scan through the files and only copy the differences. You can even change the original data and run rsync to synchronize again. When you're done, you can simply remove the original if you don't need it any more.
FTP is sometimes trying to be too clever and differentiate between binary and text files and possibly recode the text files. You can turn that off in your client, though. NFS is being used by lots of businesses as a network filesystem for critical data which makes it more tested than FTP whose common purpose is download data from anonftp and push data to websites, etc. If you're careful, you can just use any of the protocols but rsync is careful for you ;).
