How to use rclone with Backblaze B2 Cloud Storage

Installing Rclone

Rclone is available for just about every platform.  You can download the latest version here or you can install it from your system's default repository.  Depending on your operating system, the command may be: apt install rclone, yum install rclone, brew install rclone, choco install rclone, etc.  The latest version with the newest features will always be available directly from rclone, as repositories are often out of date, and you won’t get the latest features.

 

Configuring Rclone

To configure rclone, open a command prompt or terminal window, and enter:

rclone config

 

This will list your currently configured accounts (if any), and then list the menu of commands.

 

 

To create a new configuration, enter “n” for new, and then enter a name for the configuration.  Short descriptive names are best, as  you will have to use them in every command.  In this example we are choosing “b2”.

 

Then a list of possible cloud accounts will be shown, currently numbering over 50 types in the latest version.

 

Each cloud account type has a short name (alias) as well as a number. You can choose the number or the alias of the cloud account you want to use. In this case we will also enter “b2”.

 

Then enter your Application Key and Key ID, press Enter at the hard_delete prompt to take the default, and Enter one more time to skip the advanced configuration page. You can edit the configuration at any time to review or change options.

Review the configuration summary screen and press Enter to accept it.

 

Now when you use rclone commands, this configuration will be referred to as “b2”.

 

To test it enter the following to list all the buckets (directories) in your account.  

rclone lsd b2:

 

Using Rclone text GUI to explore your buckets

rclone has a text-based GUI that was inspired by Norton Commander, called ncdu.

 

To browse the buckets in your account, enter the following command.

 

rclone ncdu b2:

 

Tip: If the ncdu command encounters some files or directory structures it doesn’t like, the screen can display some errors.  Pressing Ctrl+L in Linux or MacOS, or resize the DOS window will clear up the screen.

.

The total bucket size is shown in the left column, and you can press enter on any bucket to browse the files in it.  To see a full list of possible commands press the question mark “?”.

 

 

 

Using the rclone web interface

Rclone also has a web interface that is still considered experimental but can be very useful and is worth exploring.

rclone rcd --rc-web-gui

 

 

You can browse, edit or add new configs.

 

You can also browse buckets in your account, first by selecting the account, which in this case is b2-002.

 

Then you can further browse the buckets in your account, similar to how you could with the ncdu command.

 

Useful commands

lsd - list buckets/directories (depending on if  you are at the top level in your account or inside a bucket)

ls - list files

mkdir - make a bucket/directory.  Buckets are automatically made “private”.

copy - copy files

sync - sync files (make a mirror copy so both sides match)

 

Examples:

List buckets or directories with the lsd command and list files with the ls command.

rclone lsd b2:  (Will list buckets in your account)
rclone ls b2:my-bucket   (will list directories in your bucket)

 

To create a new bucket, or a folder in a bucket use the mkdir command.

rclone mkdir b2:new-unique-bucket
rclone mkdir b2:my-bucket/new folder

 

To copy or sync files, use this format:

rclone copy myfiles.zip b2:mybucket
rclone sync C:\Downloads b2:mybucket/Downloads

 

Or to copy/sync from B2 to  your local file system.

rclone copy b2:mybucket/myfiles.zip /home/mfarace
rclone sync b2:mybucket/downloads /home/mfarace/downloads

 

 

Performance tuning

The default number of threads to use for file transfers is 4. You can change that with the --transfers command.  The faster your internet connection, and the more the --transfer command will help your file transfer speed.  For most people, 10-20 threads will be more than enough.

 

Example:

rclone --transfers 10 copy Ubuntu.iso b2:mybucket

 

If you have a 100 Mbps or faster Internet connection, you try 50-100 threads.  The more threads you use, the more memory it will use.  If you are transferring files over a long distance, increasing the number of threads will help compensate for the latency as well.

 

Note:  If you share your connection with others in a work or similar environment, please keep in mind you can consume all available bandwidth with these commands and become very unpopular :)

 

Using Rclone mount

Rclone’s mount option lets you mount a bucket from your account to a local directory.

 

To use rclone mount, you need to have a FUSE layer installed in your operating system.  What is FUSE?  Per the Wikipedia definition Filesystem in Userspace (FUSE) is a software interface for Unix and Unix-like computer operating systems that lets non-privileged users create their own file systems without editing kernel code”.

 

Installation notes by operating system

For Catalina (10.15) use OSXFuse 3.10.6.  Starting with OSXFuse 3.11 there are changes for OSX 11 / BigSur and Intel Silicon chip support that cause problems for OSX 10.x.

 

You can install rclone via Homebrew with “brew install rclone” or download the Mac DMG here.

 

For Linux, the FUSE file system is bundled with s3fs in linux repositories, so install s3fs-fuse, and then install rclone.

Example:

sudo apt-get install s3fs-fuse rclone
sudo apt-get install rclone

 

For Windows, install winfsp for the same functionality as FUSE does on Mac/Linux.  You can choose github or a chocolatey repository as safe sources to install from.

 

 

 

Linux Examples:

On Linux systems,  you must mount the bucket to an empty directory.  By default rclone mount runs in the foreground. On Linux systems, the --daemon option runs rclone mount as a background process.

 

Use case #1 - Mount a bucket read-only

mkdir /mnt/b2bucket
rclone mount b2:fuse-mf1 /mnt/b2bucket --daemon 

Use case #2 - Mount a bucket for writes

rclone mount --vfs-cache-mode writes b2:fuse-mf1 /mnt/b2bucket --daemon 

*applications expect to be able to write into the middle of a file, and without this option, they can’t.

 

 

Mac Example:

On Macs, you need to use the “mount2” command, instead of simply “mount”.  Other than that the commands work exactly the same as on Linux systems.

 

rclone mount2 b2:mybucket /Users/mfarace/b2bucket

 

 

Windows Examples:

On Windows systems, you don’t have the option to run rclone mount in the background, but you do have the option to give it a volume name and assign it a drive letter, which is not available on Mac or Linux.  If you want to mount the bucket to a directory, the directory must not exist, and also must not be at the top level of a drive. It must be a non-existent sub-directory. All other options work the same.

 

rclone mount b2:fuse-mf1 Z: -o volname=B2bucket

 

rclone mount b2:fuse-mf1 C:\Users\mfarace\non-existent-directory

 

Performance tuning rclone mount

 

Below is a list of some rclone mount options and their default values that you may want to change.

Vfs-cache-mode:          disabled  (read only mode)

Vfs-read-chunk-size:     128 MB    (read objects from the buckets in chunks)

Dir-cache-time:          5 mins    (limits how often a bucket is scanned)

Vfs-cache-poll-interval: 1 min     (when to check cache for file changes)

Vfs-cache-max-age:       1 hour

 

By default, “VFS Cache Mode” is disabled.  Enabling it uses memory (cache) in trade for improving file operations.

Options to try

--vfs-cache-mode=minimal   (minimal read/write buffering to disk, but not enough to do seeks in a file that is open, and if an upload fails, it won’t retry).

--vfs-cache-mode=writes   (writes are buffers but reads are not)

--vfs-cache-mode=full     (reads and writes are buffered)

--vfs-cache-mode full     (no equal sign on Windows)

 

Vfs cache mode is an important option when copying large files to and from your mounted bucket.  If  you were doing this on Windows, the suggested command would be:

rclone --vfs-cache-mode full --transfers 40 mount b2:fuse-mf1 Z:

 

Other options to explore.

--buffer-size        VFS cache size buffered in memory

--vfs-read-ahead  VFS cache size buffered to disk

--dir-cache-time 60m Use to minimize bucket reads on buckets that rarely change.

Advanced options to consider:

--fast-list  normally rclone reads one directory at a time (one directory per API call), but with the --fast list option, it will read files as quickly as possible, utilizing more memory but fewer transitions.  Example it can use about 1 GB of RAM per 1 million files.

--bw-limit           Limits the maximum bandwidth rclone can use.

--no-modtime Don't read/write the modification time (can speed things up)

--use-server-modtime   Use the server’s recorded time of the file modification

 

Have more questions? Submit a request

0 Comments

Article is closed for comments.
Powered by Zendesk