How to set Lifecycle Rules on B2

Lifecycle Rules enable the B2 service to automatically hide and/or delete older versions of files stored in B2. This capability is available for the Web GUI, CLI, and API. Lifecycle Rules are applied to some or all of the files in a B2 Bucket with each Bucket having its own Lifecycle Rules. Let’s take a look at how you can use this capability to help manage your B2 storage space.

Overview

By default, Backblaze B2 will keep all the files you upload and all of the different versions of a file you upload. While B2 will keep the different versions, it will “hide” older versions to keep the file list easier to view as shown in the B2 file list (from the Web GUI) below.

There are two indicators that we have multiple versions of the same file. First the file name has a (2) at the end. This is the file version count and in this case means there are two (2) versions of the file. The date/time stamp shown is for the most recent version of the file. The second indicator is the “missing” file information icon on the right.

When you click on the file version count (i.e. the 2), the list will expand to show all of the file versions.

The older version, V2 in the example, is hidden. All versions of a given file are available to download, delete or just keep around as desired. If the file names are different at all, they are stored as different files, so myfile-version1.doc is not the same as myfile-version2.doc, even if the content is identical.

There is no limit to the number of versions you can have of a given file. Also, by default, the length of time older versions stay in B2 is forever, unless you delete them or you stop using the B2 service.

Since all versions of a given file are kept around you can use Lifecycle Rules to help manage the older versions. Let’s see how.

Lifecycle Rules and the Web GUI

To use Lifecycle Rules you will have an active Backblaze B2 account and at least one B2 Bucket. It would help to have a few files in your B2 account as well. After logging into your B2 account and clicking “Buckets” on the B2 Cloud Storage menu, you’ll see a list of your buckets. For the “Bucket” of interest select the “Lifecycle Settings” link as shown in the example below.

The “Lifecycle Settings” pop-up screen, similar to the one below will appear.

Let’s cover the more common ways to use Lifecycle Rules.

File Versions

The file version rules apply to all the files in the bucket, unless otherwise specified. The four options for File Versions work as follows:

• Keep All Versions (Default): Keeps all versions of a given file.
• Keep Only Last Version: This keeps only the most current version of a file. The previous version of the file is “hidden” for one day and then deleted.
• Keep Prior Versions for this number of Days: Older versions of a file are deleted after the specified number of days. The number of days can be 1 or greater. The most current version of a file is always kept unless it is explicitly deleted.
• Lifecycle Rules Below – Allows you to construct a rule that applies to one or more files. The three parameters are covered in the next section.

Most common situations can be handled using one of the first three options.

Constructing a Lifecycle Rule in the Web GUI

Before you start constructing Lifecycle Rules, here are a few things to know:

1. You are not allowed to create two or more rules that apply to the same file or files.
2. Remember that the old version of a file is automatically hidden when a new version of the same name is uploaded.
3. Read the “file versions” documentation to understand how file versioning works in B2.
4. Only use this option if you know what you are doing. You are giving B2 the ability to hide and then permanently delete your files. Make sure this is something you want to do. Once deleted, files can not be restored by Backblaze. Please be careful.

There are three variables you need to enter to construct a Lifecycle Rule. All three are required for each rule added.

• fileNamePrefix: This specifies which files in the B2 Bucket the rules apply to. For example, a value of “workinprocess/” will apply to all files in the “workinprocess” directory/folder. Leaving the prefix field blank means the rule applies to all the files in the bucket. Please note that a prefix of * or  "" (empty quotes) will result in a lifecycle rule that literally applies to files that have a prefix of * or "". 
• daysFromUploadingtoHiding: Setting this value causes the specified files to be hidden automatically after the specified number of days. This applies to all copies of the file, even the most current version. For example, setting this variable to 30 will hide all of the files in the specified directory/folder 30 days after they have been uploaded – this includes the most current version of the file. Only use this option if you want to hide all versions of the files specified. Valid values are “null” or numbers 1 and greater. Null, means no files are hidden based on this rule.
• daysFromHidingtoDeleting: Setting this value causes the specified hidden files to be deleted automatically after the specified number of days. Remember that files are automatically hidden when they are replaced with a newer version. For example, if you set this variable to 10, then 10 days after a newer version of the file is uploaded, the older version will be deleted. Valid values are “null” or numbers 1 and greater. Null, means no files are deleted based on this rule.

Lifecycle Rule Examples

Let’s assume that you back up your “log” files once a day to B2 and that these files are stored in a directory/folder named log. Below are sample Lifecycle Rules you could set-up to manage the log files in B2 based on your data retention policies.

Example 1:

fileNamePrefix: log/

daysFromUploadingtoHiding: 7

daysFromHidingtoDeleting: null

Result: The files in the log/ folder will be hidden 7 days after they are uploaded to B2. This means the current version, as well as any previous versions of every file in the log/ folder, will be hidden after 7 days. The hidden files are still available for download using their specific file_id and continue to count as part of your B2 storage amount.

Example 2:

fileNamePrefix: log/

daysFromUploadingtoHiding: 7

daysFromHidingtoDeleting: 30

Result: The files in the log/ folder will be hidden 7 days after they are uploaded to B2. After the files have been hidden for 30 days, they will be permanently deleted. Until they are deleted, the files are still available for download and count as part of your B2 storage amount.

Example 3:

fileNamePrefix: log/

daysFromUploadingtoHiding: null

daysFromHidingtoDeleting: 180

Result: The files in the log/ folder will be deleted 180 days after they have been either 1) explicitly hidden, or 2) been replaced by a file of the same name (i.e. they are an older version of the file – remember older file versions are automatically hidden). Until the files are deleted, all versions of a file are available for download and count as part of your B2 storage amount.

Lifecycle Rules for the CLI and API

Both the B2 CLI and B2 API can utilize Lifecycle Rules.

For the B2 API, the following calls work with Lifecycle Rules:

• b2_create_bucket
• b2_update_bucket
• b2_list_buckets

Each API call has an optional parameter named lifecycleRules that requires the three variables noted above in the following JSON format:

{
    “daysFromHidingToDeleting”: 30,
    “daysFromUploadingToHiding”: null,
    “fileNamePrefix”: “backup/”
}

In addition, b2_delete_bucket will return the list of lifecycleRules for the bucket being deleted.

B2 Integrations and Lifecycle Rules

Application developers are encouraged to look at Lifecycle Rules to use in their applications that integrate into B2. This is especially true for applications that perform Sync or Backup operations to B2. For example, the user interface of the application could allow the user/admin to specify the number of days older versions of files should be kept in support of data retention rules in an organization.

B2 Command Line Tool

The B2 Command Line Tool provides command-line access to the B2 service. To use Lifecycle Rules with the B2 CLI, you will need to be running version 0.7.0 of the B2 Command Line Tool. To upgrade to this version you can follow the “Get Command-Line Tool” instructions on the Backblaze website. The following commands utilize Lifecycle Rules in the CLI.

b2 create-bucket [–bucketInfo <json>] [–lifecycleRules <json>] <bucketName> [allPublic | allPrivate]

b2 update-bucket [–bucketInfo <json>] [–lifecycleRules <json>] <bucketName> [allPublic | allPrivate]

b2 get-bucket <bucketName>

The “b2 get-bucket” command will return all the variables for the named bucket.

Summary

Lifecycle Rules are very handy in helping manage multiple versions of files you are storing in B2. These rules can help save storage space by automatically deleting older, unneeded versions of files and can make the B2 file list easier to navigate and manage. Since Lifecycle Rules will delete older file versions, one recommendation is to start with a large number of days for “daysFromHidingtoDeleteing” and reduce this number as you become comfortable with this feature. The easiest way to do this is to use the “Keep Prior Versions for this number of days:” option in the WebGUI.

There is more information on Lifecycle Rules in the B2 documentation section of the Backblaze website. The website also has code examples for the API calls, information in the B2 Command Line tool and more. Let us know if you find anything there we need to update or you feel is missing.