Here is the code I am currently trying to use when calling getting large items from S3. This file is 6.1BG and has never been able to be pulled down completely. I implemented a function that takes advantage of the Range header on S3 but I am one using it wrong (very possible) or it's broken.
// Download file from s3 to your local computer
// this will currently put the file in the
// directory you run it from. Need to add in
// some support for failover in case where TCP
// Dial timesout. Seems to happen somewhat often.
func downloadFileFromS3(file string) {
svc := s3.New(&aws.Config{Region: "us-east-1"})
params := &s3.GetObjectInput{
Bucket: aws.String(s3Bucket),
Key: aws.String(file),
resp, err := svc.GetObject(params)
if awserr := aws.Error(err); awserr != nil {
// A service error occurred.
fmt.Println("Error:", awserr.Code, awserr.Message)
} else if err != nil {
// A non-service error occurred.
panic(err)
f, err := os.Create(file)
if err != nil {
fmt.Println(err)
defer f.Close()
fmt.Println("starting download of " + file)
w := bufio.NewWriter(f)
_, err = w.ReadFrom(resp.Body)
if err != nil {
w.Flush()
f.Close()
if reflect.TypeOf(*resp).String() == "s3.GetObjectOutput" {
retryDownload(file, resp)
fmt.Println(err)
} else {
// Pretty-print the response data.
fmt.Println(awsutil.StringValue(resp))
w.Flush()
// On large files s3 is unreliable to complete the entire
// download and only has a 25% success rate. This function
// will try to complete the download by starting at the
// range that was successfully downloaded and appending
// to the file from that point.
func retryDownload(file string, obj *s3.GetObjectOutput) {
fmt.Println("attempting to fix downloaded file...")
svc := s3.New(&aws.Config{Region: "us-east-1"})
params := &s3.GetObjectInput{
Bucket: aws.String(s3Bucket),
Key: aws.String(file),
Range: aws.String("bytes=" + strconv.FormatInt(*obj.ContentLength, 10) + "-"),
fmt.Println("making request")
resp, err := svc.GetObject(params)
if awserr := aws.Error(err); awserr != nil {
// A service error occurred.
fmt.Println("Error:", awserr.Code, awserr.Message)
} else if err != nil {
// A non-service error occurred.
panic(err)
fmt.Println("Opening file to append")
f, err := os.OpenFile(file, os.O_RDWR|os.O_APPEND, 0666)
if err != nil {
fmt.Println(err)
defer f.Close()
fmt.Println("trying to append")
w := bufio.NewWriter(f)
_, err = w.ReadFrom(resp.Body)
if err != nil {
w.Flush()
f.Close()
if reflect.TypeOf(*resp).String() == "s3.GetObjectOutput" {
retryDownload(file, resp)
fmt.Println(err)
} else {
fmt.Println("in here...")
// Pretty-print the response data.
fmt.Println(awsutil.StringValue(resp))
The errors that I get are as follows.
read tcp 54.231.9.32:443: operation timed out
AcceptRanges: "bytes",
Body: &{0xc2082cf380 {0 0} false 0xc2082ce140 <nil> 0x161290},
ContentLength: 6665384021,
ContentType: "application/x-gzip",
ETag: "\"f0cf293fe2291cc0e4d8b72901be18fc-424\"",
LastModified: 2015-03-15 20:12:50 +0000 UTC,
Metadata: {
S3cmd-Attrs: "uid:1000/gname:ubuntu/uname:ubuntu/gid:1000/mode:33204/mtime:1426407620/atime:1426407658/md5:cde9c42fbc30b5d68868de45455182db/ctime:1426449167"
When i go to try and fix the issue using the second function I get.
attempting to fix downloaded file...
making request
Error: InvalidRange The requested range is not satisfiable
Opening file to append
trying to append
attempting to fix downloaded file...
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x55e0]
goroutine 1 [running]:
main.retryDownload(0xc2081e6620, 0x1d, 0xc2080356c0)
/Users/mschuett/Go/src/github.com/michaeljs1990/scripts/cmd/rds-util/main.go:327 +0x230
main.retryDownload(0xc2081e6620, 0x1d, 0xc208034a90)
/Users/mschuett/Go/src/github.com/michaeljs1990/scripts/cmd/rds-util/main.go:355 +0xba3
main.downloadFileFromS3(0xc2081e6620, 0x1d)
/Users/mschuett/Go/src/github.com/michaeljs1990/scripts/cmd/rds-util/main.go:303 +0x7e8
main.fetchBackupsFromS3(0x0, 0x0, 0x0)
/Users/mschuett/Go/src/github.com/michaeljs1990/scripts/cmd/rds-util/main.go:221 +0xed
main.runMigration()
/Users/mschuett/Go/src/github.com/michaeljs1990/scripts/cmd/rds-util/main.go:149 +0x5b
main.main()
/Users/mschuett/Go/src/github.com/michaeljs1990/scripts/cmd/rds-util/main.go:98 +0x517
goroutine 17 [syscall, 12 minutes, locked to thread]:
runtime.goexit()
/usr/local/Cellar/go/1.4.2/libexec/src/runtime/asm_amd64.s:2232 +0x1
goroutine 15 [select]:
net/http.(*persistConn).writeLoop(0xc20825c210)
/usr/local/Cellar/go/1.4.2/libexec/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/Cellar/go/1.4.2/libexec/src/net/http/transport.go:661 +0xcbc
goroutine 14 [runnable]:
net/http.(*persistConn).readLoop(0xc20825c210)
/usr/local/Cellar/go/1.4.2/libexec/src/net/http/transport.go:928 +0x9ce
created by net/http.(*Transport).dialConn
/usr/local/Cellar/go/1.4.2/libexec/src/net/http/transport.go:660 +0xc9f
exit status 2
Sorry for bugging you so much 😦
It looks like what i would want is multipart download but I don't see it any place in the current API. I am guessing you could implement it fairly easily using goroutines and the Range header? I would be willing to do this although I am not sure if multipart download has an official spec it needs to meet?
Hi @michaeljs1990 initially i think the retryDownload is failing because
Range: aws.String("bytes=" + strconv.FormatInt(*obj.ContentLength, 10) + "-"),produces a range request header value like:
bytes= 6665384021-
This in incorrect when retryDownload is called for the first time due to downloadFileFromS3 encountering an error. The value that should be used in the number of bytes written to your file. So that the retryDownload starts off where the connection error occurred.
Also I'd suggest updating retryDownload to take the f file reader, and length of previous copy instead of s3.GetObjectOutput pointer. This will ensure that the correct bytes will be requested. This also prevents the need for the bufio and flushes if io.Copy is used instead.
