Foo - What's the most efficient way to transfer data?

Something came up at work and we're wondering if we could, on about a weekly basis, transfer about 1.0 to 1.3 GB of data from one computer to about 5, or as many as 10, others, all of which are in different locations.

Do you think this could be done in a reasonable amount of time? I guess the cheapest solution would be USB 2.0 flash drives or external hard drives?

Anyone have experience doing this kind of stuff?

Efficient for the computer or efficient for the human? Different locations in the same building or geographically separated? If geographically separated, is the data sensitive? Why not just make a bunch of copies via file sharing? Sure, if your network and/or internet connection is slow it might take a while, but you can initiate the process at the end of a workday and just let it go.

I'd try using a network. Network should be tons times faster then USB and much less processor intensive. A 1gb file can transfer quite fast over a modern network (preferably wired)

Pretty much passing around a flash drive would be the quickest/cheapest

http://www.amazon.com/SanDisk-Cruzer-SDCZ2-2048-A10-Retail-Package/dp/B0007TC38G/sr=1-5/qid=1162505391/ref=sr_1_5/002-8846984-2206450?ie=UTF8&s=electronics

Flash drives are slow. The fast ones are expensive and they have a limited # of writes.
I would never use a Flash Disk system for something like this. Unless there was no way to run network cables or run wireless as wireless should be faster then USB, is not limited by write speed and does not wear down after x writes.

When you say different locations...different cities?
Publish once to FTP/ or intranet site (if offices are linked with one)...let branch office download at their leisure. Or write to their respective sites at the same time (overnight lan admin operation). You network administrator will have your solution where they write to the locations all at once.

If it's within the same building/wired network, simply post to network volume and it's accessible.

If it's Sensitive with capitol S burn and ship DVD's

All the computers are in different geographic locations, meaning, not in the same building.

I thought about the internet, but I'd guess that the max download speed would be 1 mb/sec, so that would take forever, and since we'd like to not have the data get into anyone else's hands.

One other idea I thought about are burning it onto DVD's. Have the people who will be bringing the data to the remote machines get the data off our network, and the burn it. For something that would happen on a weekly basis, I guess it could work.

Proper encryption (which would be necessary even with the DVDs) would help protect sensitive information. So I don't see going over the Internet via some sort of VPN as being a problem. I know of companies that transfer huge amounts of extremely sensitive data that way. As for time, well, at 1 MB/s, you'd copy those files in about 20 minutes. Let's say it takes 30 minutes. You really think you could burn a DVD, drive it over, and then upload it faster than that? And the electronic transfer takes far less work. Even if you just make transfers all from the same computer (you could do it faster if you script it so that after the first transfer, that computer starts transferring, too), and it takes 30 minutes per transfer, that only takes 5 hours max. Even if it takes an hour per transfer, that's 10 hours. Surely you have a 10 hour window over the weekend? And importantly, you don't have to be there to get the job done. It could be scripted to run automatically. Or you could just start the transfers at the end of the work week and let them go.

you can get blistering fast data transfers... I have three fibre lines to our press centre at the other end of the city, but it's full all day everyday. A weekly burn and ship seems fair inexpensive in comparison

Dvd Ram

Why aren't all your buildings on a network? Sneaker-net went out of style decades ago... You can set up a private WiFi network using directional antennaes with ranges of about a mile.

jschen's got the right idea. It's simplest to use the network someone else has developed. Saves you time, effort and money. You can even set up a sftp-server and have the files posted for all them to download. Encryption will ensure security. Compression can probably crunch that gig of data down quite a bit depending upon what type of data it is.

If it's something that you do on a regular basis, you can even write a script that'll upload to the server and automatically download to each of the client stations.

I'd try using a network. Network should be tons times faster then USB and much less processor intensive. A 1gb file can transfer quite fast over a modern network (preferably wired)
+1

If its the same data, you could even automate it with a batch file or script.

you can get blistering fast data transfers... I have three fibre lines to our press centre at the other end of the city, but it's full all day everyday. A weekly burn and ship seems fair inexpensive in comparison
I wonder why printing press and likees networks are always bogged down :p

^:lol:^ NOT that our ISP runs through those channels, TG smar-ty

If all the computers are hooked up to one main network, you can setup up so that all the computers can be shared, that way files can be transfered from one computer to as many as you want. On my home network i have 3 computers hooked up, and i can transfer data really easily, up to 100mbp/s.

If all the computers are hooked up to one main network, you can setup up so that all the computers can be shared, that way files can be transfered from one computer to as many as you want. On my home network i have 3 computers hooked up, and i can transfer data really easily, up to 100mbp/s.
that aspect of the problem has already been delineated. They're not. They are in different geogrpahical locations.

you could do a point to point; some browsers, i.e. DEEPNET enable this and you can set passwords.
FTP is archaic and slower, and an open share on a network means ANYONE that can see it can snatch it.
of course, you could set passwords on that share as well but throughput would degregate depending on amount of traffic in that subnet. Not to mention if you were leaving that net you would be subjugated to the router or switches ability to provide bandwith as well. Regardless of how much bandwidth you have, you are still limited by the slowest hop along the path between two nodes.

Passwords alone don't do you much good if you're transmitting unencrypted data in the clear. If the data is at all sensitive, encryption is necessary. Whether you do this at the file level or at the network level is up to you. I'd do it at the network level to avoid having to encrypt and decrypt files over and over.

Thanks for all the input, I'm still trying to wrap my head around this and figure something out.

There are a lot of variables, so I don't know if we'll be able to afford the time of downloading over the net, but that option is on the table.

Because the person that will be going to the remote machine already has their own machine here in the office with a DVD burner, is on the network, and a batch file set up to retrieve the data from the main machine, I'm leaning towards having each person burn the data onto the DVD.

This might all be done sorta on the fly, so weekend downloads probably aren't an option, and I think we're already getting flack for our bandwidth use.

How far are the buildings? Transfering data by shuffling people back and forth is truly inefficient. If these transfers are made "on the fly", how urgent is the data?

If you must shuffle people, I'd go with portable hard drives (Firewire or USB2). Having lots of DVDs sitting around with sensitive data is potentially problematic. You'd have to ensure each of those DVDs is destroyed after use. And hard drive transfers are far faster.

How far are the buildings? Transfering data by shuffling people back and forth is truly inefficient. If these transfers are made "on the fly", how urgent is the data?

If you must shuffle people, I'd go with portable hard drives (Firewire or USB2). Having lots of DVDs sitting around with sensitive data is potentially problematic. You'd have to ensure each of those DVDs is destroyed after use. And hard drive transfers are far faster.
The same issue happens with USB also. Some janitor picks up usb, takes it home . bam. Information is compromised. I'd be tempted -

if they are going from laptop to computer why not just use a wireless network.

How far are the buildings? Transfering data by shuffling people back and forth is truly inefficient. If these transfers are made "on the fly", how urgent is the data?

If you must shuffle people, I'd go with portable hard drives (Firewire or USB2). Having lots of DVDs sitting around with sensitive data is potentially problematic. You'd have to ensure each of those DVDs is destroyed after use. And hard drive transfers are far faster.

Well, I'd assume we'd destroy the data DVD once it was transfered, and the cost isn't prohibitive.

And the the other computers are too far to utilitize a wireless network.

How about a shared workspace application like Groove (http://www.groove.net/home/index.cfm)?

Install it on each machine, leave them connected to the internet and logged in to a shared Groove workspace. The files in the workspace are automatically distributed to each computer over the internet, encrypted, in the background. No human interaction required. It'll even do automatic backups of everything, and lets you use version tracking if the files are changed over time by multiple users.

that aspect of the problem has already been delineated. They're not. They are in different geogrpahical locations.In that case external USB hard drive, have one of those as well, works great.

I think everyone forgot to ask the critical question here...

What's your budget?

Also: What's the distance? Can the data be streamed and/or updated incrimentally? Is it one big file or multiple files? What's the nature of the data? Text? Binary?

I think everyone forgot to ask the critical question here...

What's your budget?

Also: What's the distance? Can the data be streamed and/or updated incrimentally? Is it one big file or multiple files? What's the nature of the data? Text? Binary?

Not sure of the budget, but seeing as we're a small department in a smallish company, it isn't very much. That's why I'm leaning to DVDs; it's only going to cost us for the dvd itself ($.40 per disc?) x 3-5 every 1-1.5 weeks, since travel will be done regardless.

Varying distances, updated incrimentally, and multiple binary files.

The easiest and cheapest way i know off is get a HD that is big enough to fit your needs. Transfer all the data on it, and instead of having the jumpers set to primary, switch them to be secondary making the HD be as extra storage, then just hook it up to the CD-ROM drive cable and it will appear as local disk D. So pretty much hook it up to every computer you need.

Varying distances, updated incrimentally, and multiple binary files.
Hmmm... seems like you have several options.

Let's start with the transport layer. One is to build a dedicated physical network on top of dedicated circuits but that can be costly and also involves a fair bit of OpEx. Another would be to build a logical network (MPLS VPN, IPSEC VPN). And a third would be to use a pre-existing infrastructure such as the public Internet with IPSEC or SSH tunnels for encryption and security.

Now about the application side... You can use some sort of distributed filesystem, such as AFS, NFS or SMB. Then it's simply a matter of one machine mounting the filesystem off another and performing a copy operation. Another method would be a user-application layer client-server file distribution system such as those employed by peer-to-peer apps. One thing you might want to investigate is rsync. Another is CVS or SVN (Subversion). These are generally used for source code control/distribution and include versioning as well as check-in/out with locking but the key feature is that they can handle updating local caches of files incrimentally based on change-detection. These applications can of course ride on top of whatever underlying network layer option you choose.

since travel will be done regardless.
This piece of information seriously alters the outlook. If travel is going to occur regardless for business reasons, and such travel coincides with the need to transfer data, just burn DVDs. Otherwise, the labor cost will become very high over time if people are physically transporting data when they should be doing other things.

This piece of information seriously alters the outlook. If travel is going to occur regardless for business reasons, and such travel coincides with the need to transfer data, just burn DVDs. Otherwise, the labor cost will become very high over time if people are physically transporting data when they should be doing other things.

Yeah, sorry, I should have stated that earlier. Haha, when I'm in a hurry, I don't explain myself very well

Do you need the data to be/remain consistent within a certain timeframe? Sneakernetting can work but this means your data synchronisation is done in a very course-grain lockstep manner. I understand you stated an update interval on a weekly basis so I assume you'd be okay with that.

you could try transmitting it over the internet with bittorrent.

In that case external USB hard drive, have one of those as well, works great.
I dont think you've tried to swap 2-3gb of files around on a usb. :p it's very inconvenient and not the fastest thing in the world :p and "usb hard drives" are a pita to lug around.

I dont think you've tried to swap 2-3gb of files around on a usb. :p it's very inconvenient and not the fastest thing in the world :p and "usb hard drives" are a pita to lug around.

Just give them an iPod and they'll be happy to lug it around. ;)

OK, I am having a hard time understanding your business process here. It looks to me like you have data that needs to be accessible to many people at different locations. If that's the case, store the data in a single location, SAN or database. Remote users would have to download the data one time, and from there simply synchronize changed data, via a WAN or Internet VPN solution. But again, not knowing the type of data and both your business process and requirements, I'm just throwing a kludge off the top of my head.