02. S3 Storage Setup

Author

Sky (Kehan) Sheng

Create EC2 Credentials

In order to manage your Arbutus Object Store, you need your own storage access ID & secret key. In terminal, create credentials using:

openstack ec2 credentials create

This will output your Access Key and Secret Key. Save these credentials securely - you’ll need them for the next step.

Configure s3cmd

To configure the s3cmd tool, use the command:

s3cmd --configure

Then make the following configurations using your credentials printed out in the terminal above:

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key []: 20_DIGIT_ACCESS_KEY
Secret Key []: 40_DIGIT_SECRET_KEY
Default Region [US]:

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint []: object-arbutus.cloud.computecanada.ca

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket []: object-arbutus.cloud.computecanada.ca

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password []:
Path to GPG program [/usr/bin/gpg]: 

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol []: Yes

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:

Important prompts:

  • You will be prompted for “Test access with supplied credentials?” → Answer “y”
  • You will be prompted for “Save settings? [y/N]” → Answer “y”

Acknowledgments

This tutorial is based on the s3cmd setup guide.

Next Steps

Once s3cmd is configured, you can start working with object storage:

  • S3 Operations - Create buckets, upload files, and manage access