03. S3 Operations

Create Buckets

Buckets contain files. Bucket names must be unique across the Arbutus object storage solution.

Valid bucket names may only use:

  • Upper case characters (A-Z)
  • Lower case characters (a-z)
  • Digits (0-9)
  • Period (.)
  • Hyphen (-)

View All Buckets

To view all buckets, use the ls command:

s3cmd ls

Create a Bucket

To create a bucket, use the tool’s mb (make bucket) command:

s3cmd mb s3://BUCKET_NAME/

Check Bucket Status

To see the status of a bucket, use the info command:

s3cmd info s3://BUCKET_NAME/

The output will look something like this:

s3://BUCKET_NAME/ (bucket):
   Location:  default
   Payer:     BucketOwner
   Expiration Rule: none
   Policy:    none
   CORS:      none
   ACL:       *anon*: READ
   ACL:       USER: FULL_CONTROL
   URL:       http://object-arbutus.cloud.computecanada.ca/BUCKET_NAME/

Upload Files

To upload a file to the bucket, use the put command:

s3cmd put --guess-mime-type FILE_NAME.dat s3://BUCKET_NAME/FILE_NAME.dat

where the bucket name and the file name are specified.

Delete Files

To delete a file from the bucket, use the rm command:

s3cmd rm s3://BUCKET_NAME/FILE_NAME.dat

Access Control Lists (ACLs) and Policies

Buckets can have ACLs and policies which govern who can access what resources in the object store. These features are quite sophisticated. Here are two simple examples of using ACLs using the tool’s setacl command.

Make Bucket Public

The result of this command is that the public can access the bucket and recursively (-r) every file in the bucket:

s3cmd setacl --acl-public -r s3://BUCKET_NAME/

Files can be accessed via URLs such as:

https://object-arbutus.cloud.computecanada.ca/BUCKET_NAME/FILE_NAME.dat

For videos and media files: Once your bucket is public, you can directly access and stream videos through these URLs in web browsers or media players.

Make Bucket Private

The second ACL example limits access to the bucket to only the owner:

s3cmd setacl --acl-private s3://BUCKET_NAME/

View Bucket Configuration

The current configuration of a bucket can be viewed via the command:

s3cmd info s3://BUCKET_NAME/

Acknowledgments

This tutorial is based on the s3cmd setup guide.

Additional Resources