To start with, here's an example of a .7z compression command line might be:
7z a -t7z -mx1 -m0=LZMA2 -mmt8 -ssw -bd -y -pPassword -r My7zFile.7z "C:\test\*"
-t7z means use 7z file format (excellent compression at -mx9, and good speed with -mx1)
-mx1 means use very fast compression, while -mx2..8 are intermediate steps, and -mx9 is very tight compression More compression = more RAM required
-mmt8 means 8 threads. -mmt3 means 3 threads, -mmt1 means 1 thread, etc. More threads = more RAM require (at 4, 6, 8, etc., so 3/5/7 are good choices)
-ssw means also try to back up files with share locks, i.e. some types of opened/locked files. Consider carefully about this one.
-bd means no percentage indicator
-y means no prompts
-pPassword means use enryption (for -t7z files, that's AES), and use Password as the password. You could use -psWO4UXZRaykA and have sWO4UXZRaykA as your password.
-r means recursive - take if off if you're specifying files
As another post suggested, you must break each situation down.
First, you have requirements:
A) Maximum timeframe available (starting)
Ai) On the source side
Aii) On the target side
B) Maximum timeframe available (change set)
Bi) On the source side
Bii) On the target side
C) Allowable impacts (slowness, offline, shutdown, etc.)
Ci) On the source side
Cii) On the target side
D) Privacy, confidentiality, and other encryption/security requirements
Di) In some cases, this means encryption
Dii) In others, this means a licensed, bonded courier
Then, you have your situation - note that "compressed" values depend on your particular dataset - you'll have to test them.
1) Starting data size (uncompressed)
1a) Accounting for growth over N years
2) Starting data size (compressed)
2a) Accounting for growth over N years
3) Change set over time size (uncompressed) (minimum, average, maximum)
3a) Accounting for growth over N years
4) Change set over time size (compressed) (minimum, average, maximum)
4a) Accounting for growth over N years
5) Latency for each transfer method (uncompressed)
6) Latency for each transfer method (compressed)
7) Throughput for each transfer method (uncompressed)
8) Throughput for each transfer method (compressed)
9a) Impact of restartability if not required, and if required
If we look at a few possible transfer options, remembering that the physical drives can be easily set up in parallel if your machines have the ports:
I) Local gigabit
I5) 5ms latency uncompressed
I7) 110MB/s throughput uncompressed
II) 15Mbps internet link
II5) 50ms latency uncompressed
II7) 1.5MB/s throughput uncompressed
III) 4TB USB2.0 drive, local
III5) 9ms latency uncompressed
III7) 30MB/s throughput uncompressed
IV) 4TB USB3.0 or eSATA or SATA or SAS drive, local
IV5) 9ms-22hr latency uncompressed
IV7) 110MB/s throughput uncompressed
V) 4TB USB3.0 or eSATA or SATA or SAS drive, shipped, 12 hour transit time
V5) 12-34 hour latency uncompressed (0-11 hours to fill, 12 hours to ship, 0-11 hours to read)
V7) 0-32MB/s throughput uncompressed (0-11 hours to fill, 12 hours to ship, 0-11 hours to read)
VI) 10.3TB Fusion ioDrive Octal, local
VI5) negligible latency uncompressed
VI7) 3.9GB/s write, 6.7GB/s read
VII) 10.3TB Fusion ioDrive Octal, shipped, 12 hour transit time
VII5) 0-12 hr latency uncompressed
VII7) 0-215MB/s throughput uncompressed (.75 hrs to fill, 12 hours to ship, .5 hours to read)
For transfer over networks, we can use copy, robocopy, Windows GUI, etc.
For compression, I'm going to join in with the 7-zip crowd, and specify a few things:
DO use 7z format, and LZMA2 mode.
You do NOT have to compress, copy, uncompress. Instead, you can compress locally, and then uncompress from the remote machine, so only the compressed data goes over the network! Alternately, you can compress _to_ the remote machine.
Note that I ran a quick, fake data test - a 768MB files of all 0x00 values (trivially compressible)
Straight Windows copy to a machine several hundred miles away, 250KB/s.
7-Zip compression via GUI, .7z, LZMA2, 64KB dictionary size, Solid, 273 byte word size, Fastest, directly to the machine (\\remote\share\test.7z), 20MB/s BEFORE decompression.
7-Zip compression via GUI, .7z, LZMA2, 64KB dictionary size, Solid, 273 byte word size, Fastest, directly to local and then the .7z file copied via Windows GUI, 96MB/s BEFORE decompression.