Quickstart Guide using S5cmd with B2 Cloud Storage Quickstart Guide using S5cmd with B2 Cloud Storage

Quickstart Guide using S5cmd with B2 Cloud Storage

Permanently deleted user Permanently deleted user

Introduction

s5cmd is a free open source tool written in the Go language, that supports a wide range of object management tasks both for cloud storage services, and local filesystems. It also supports parallel tasking and multiple threads for very fast throughput.  s5cmd provides binaries for most operating systems on the releases page in the git repository.

 

s5cmd can be configured with Backblaze B2 Cloud Storage by following the steps below.

 

Guide

 

1) Create a directory that will contain your credentials file.

mkdir ~/.aws

 

2) Create your credentials file, and enter your Key ID and Application Key (App Keys) in the format shown below.

vi ~/.aws/credentials

 

Enter the following information and then save and exit.

[default]
aws_access_key_id = 9xxxxxxxxxxx00000000003
aws_secret_access_key = Kyyyyyyyyyyy00000g

 

3) You can streamline the use of s5cmd by creating an alias so that the endpoint URL is already defined.  You can add this command to your .bashrc file in Linux for example, so it is defined automatically.

alias s5cmd='s5cmd --endpoint-url https://s3.us-west-002.backblazeb2.com'

 

s5cmd is now configured to use your B2 endpoint, let's go over some simple usage patterns. 

 

Usage

List all buckets.

s5cmd -endpoint-url https://s3.us-west-002.backblazeb2.com ls

Note: The endpoint URL must be specified on each command unless you configured an alias as we did above. The next examples assume you did. 

 

List all buckets. (aliased)

s5cmd ls

 

List contents of a bucket. (aliased)

s5cmd ls s3://mydatabucket-1/

 

Create a bucket. (aliased)

s5cmd mb s3://mydatabucketus1/

 

Copy a local file into a bucket. (aliased)

s5cmd cp my-pets2020.jpg s3://mydatabucket-1/

 

 

Performance optimizations for high-speed links.

For most circumstances, the default logic for how many threads to use for copy operations works fine.  However, if you have a fast internet connection, you can manually increase the number of threads being used with the cp —-concurrency command.  

 

Example:

s5cmd --endpoint-url https://s3.us-west-002.backblazeb2.com cp --concurrency 50 /tmp/sample.txt s3://smcmdus1/test/

 

If you have used the alias command as suggested above, the command would be:

s5cmd cp --concurrency 50 /tmp/sample.txt s3://smcmdus1/test/

 

Be aware that the more threads you use, the more memory you will use, so monitor your system memory as you test for the optimal number of threads. On a system with 8 CPUs and 24 GB RAM, our testing showed that “cp --concurrency 100” achieved the fastest transfer rate.  Your own environment may require different settings.

 

 

 

The S3 Compatible API for Backblaze B2 Cloud Storage allows 1000’s of integrations to work with B2 natively. If you’re new to the S3 Compatible API, please see our Getting Started Guide. If you have any trouble using s5cmd with B2, please let us know by emailing us at b2feedback@backblaze.com.