Example: Upload a File to AWS S3 with Boto

Example Code

Amazon Web Services (AWS) is a collection of extremely popular set of services for websites and apps, so knowing how to interact with the various services is important. Here, we focus on the Simple Storage Service (S3), which is essentially a file store service.

All files must be assigned to a bucket, which is assigned a name and can be addressed by http://s3.amazonaws.com/{bucket}/{key}. Each file is assigned a unique key, which can be used later on to retrieve the file.

There are plenty of other options to assign to buckets and files (encryption, ACLs, etc.), but we won’t get in to it much here. Just notice the references to ‘public-read’, which allows the file to be downloaded by anyone.

The Code

The code below shows, in Python using boto, how to upload a file to S3.

import os

import boto
from boto.s3.key import Key

def upload_to_s3(aws_access_key_id, aws_secret_access_key, file, bucket, key, callback=None, md5=None, reduced_redundancy=False, content_type=None):
    """
    Uploads the given file to the AWS S3
    bucket and key specified.
    
    callback is a function of the form:
    
    def callback(complete, total)
    
    The callback should accept two integer parameters,
    the first representing the number of bytes that
    have been successfully transmitted to S3 and the
    second representing the size of the to be transmitted
    object.
    
    Returns boolean indicating success/failure of upload.
    """
    try:
        size = os.fstat(file.fileno()).st_size
    except:
        # Not all file objects implement fileno(),
        # so we fall back on this
        file.seek(0, os.SEEK_END)
        size = file.tell()
    
    conn = boto.connect_s3(aws_access_key_id, aws_secret_access_key)
    bucket = conn.get_bucket(bucket, validate=True)
    k = Key(bucket)
    k.key = key
    if content_type:
        k.set_metadata('Content-Type', content_type)
    sent = k.set_contents_from_file(file, cb=callback, md5=md5, reduced_redundancy=reduced_redundancy, rewind=True)
    
    # Rewind for later use
    file.seek(0)
    
    if sent == size:
        return True
    return False

Using the Code

And here is how you’d use the code:

AWS_ACCESS_KEY = 'your_access_key'
AWS_ACCESS_SECRET_KEY = 'your_secret_key'

file = open('someFile.txt', 'r+')

key = file.name
bucket = 'your-bucket'

if upload_to_s3(AWS_ACCESS_KEY, AWS_ACCESS_SECRET_KEY, file, bucket, key):
    print 'It worked!'
else:
    print 'The upload failed...'

boto works with much more than just S3, you can also access EC2, SES, SQS, and just about every other AWS service. The boto docs are great, so reading them should give you a good idea as to how to use the other services. But if not, we’ll be posting more boto examples, like how to retrieve the files from S3.

Resources