big option

/big[:<chunk size>]

Examples

/big
/big:25 (uploads in 25MB chunks instead of the default 10MB)
/big:0.1 (uploads in chunks of about 100KB, useful on very slow networks)

/big splits files into 10MB chunks suffixed with .000, .001 etc. This is done without creating any temporary files on disk, providing a useful get-out if you are completely out of local disk space and wanting to shift files to S3. A custom chunk size can be specified in megabytes.

When uploading with /big, s3.exe always fetches the MD5 checksum of any existing chunk on S3 before uploading a new chunk, and skips the upload if the checksum matches with the corresponding part of the local file. So if the local file is unchanged since it was last uploaded then nothing will be uploaded, and if only parts of the file have changed then only the chunks that need to be uploaded will actually be transferred.

The maximum number of chunks is 10,000 (even though the first 1,000 chunks are uploaded with a three digit suffix) so with the default chunk size of 10MB, the maximum file size is over 90GB. It's recommended to use the default 10MB chunk size unless transferring a file larger than this. Amazon's own EC2 service uploads multi-gigabyte machine images to S3 in 10MB chunks.

When the upload is complete, s3.exe tries to delete the chunk following the last one uploaded. For example, if the final chunk is myfile.005 then it will try to delete myfile.006. This is to avoid pre-existing chunks from erroneously being downloaded when the file is later downloaded.

Last edited Dec 1, 2009 at 12:41 AM by maxc, version 4

Comments

MichaelRG May 5, 2011 at 6:49 PM 
Can s3.exe be updated to reflect the new 5TB file size limit on S3? I have files in the 10-20GB range that I'd like to try uploading whole (from EC2 to S3, so I trust the network). Alternatively, could s3.exe take advantage of the multipart upload option where the file is re-assembled by S3? Thanks! Your app has been invaluable for my db backups.

maxc Nov 29, 2009 at 1:21 PM 
The /big option has now been greatly improved and will only upload the missing or changed chunks when told to upload the file again.
http://s3.codeplex.com/Release/ProjectReleases.aspx?ReleaseId=36555

maxc Nov 25, 2009 at 10:28 PM 
The default of 10MB is the largest size that is practical, but a smaller size will be needed on a slow network (this is my opinion and also Amazon's it seems -- EC2 stores stuff to S3 in 10MB chunks). What sort of interruption do you have in mind? Network problems and even the laptop that's doing the uploading going into hibernation (been there!) won't interrupt s3.exe, as it's pretty robust and does lots of retries. But that does depend on the chunk size being small enough for the quality of the network connection, or uploads will fail so frequently that it's continually doing retries. So it doesn't try to upload only missing chunks.

P.S. It can cope with >999 chunks, so your 10GB file will be OK.

MichaelRG Nov 25, 2009 at 4:08 PM 
Is there a maximum or optimal size for chunking? I'm trying to upload a 10GB file with fairly large chunks (1GB, then 512MB) and the upload is failing. Also, if a chunked upload is interrupted, does S3.exe know to only upload the missing chunks when told to upload the file again?