How to allow Cloudflare to fetch content from a Backblaze B2 private bucket

Storing Content in Backblaze B2 for a CDN

Many of our customers have expressed interest in hosting static data for their website (ranging from minified Javascript applications to multi-hour 8K video) because of the security, reliability, and affordability of Backblaze B2 storage. One solution to ensuring performance and availability is to route requests through a CDN (Content Delivery Network) such as Backblaze's Bandwidth Alliance partner Cloudflare, taking advantage of Cloudflare's performance and the free data transfer between Backblaze B2 and Cloudflare.

How does a CDN like Cloudflare work?

Cloudflare leverages DNS (Domain Name System) so that content requests come to Cloudflare's servers. Through caching and private high-speed links, Cloudflare ensures high availability and reliability from storage. A website's domain name is registered with Cloudflare (and transferred from its domain name registrar), so that Cloudflare becomes responsible for serving content from that domain. Behind the scenes, Cloudflare allows a website's domain to be aliased to some other domain, so that a user may see images and content from https://www.coffeemaniacs.com when those images and that content is actually being served from Backblaze B2 (https://f345.backblazeb2.com/file/coffemaniacs-storage)

Backblaze Buckets are on the Internet Securely

Although all buckets are addressable from the internet, only public buckets can be accessed by just anybody. By default, Backblaze B2 storage is private, which means that access requires authentication. Backblaze's various integration partners have incorporated this security into their tools to keep Backblaze B2 as user-friendly as possible while still maintaining security.

Website Content from Secure Buckets

Putting these elements together means that customers serving data from their website want to serve from their website; they want to store their photos and videos and all of their digital content in a private bucket, available through (and only through) their website. When hosting a website directly, adding the authentication required to pull data from Backblaze B2 is straightforward. Fronting a website through Cloudflare is slightly more complex: now Cloudflare has to access private buckets to retrieve and cache data, which means Cloudflare has to authenticate its requests to Backblaze B2.

Web Workers for the Win

Cloudflare offers web workers, small Javascript snippets that allow rewriting HTTP and HTTPS requests on the fly. These make it straightforward to add authentication headers to content requests, and authenticate the link between Cloudflare and Backblaze B2. Even better, Cloudflare's web workers can be uploaded directly into Cloudflare's servers making the automation of the process straightforward.

Setting up to enable access to your private bucket

Building this connection requires:

  • A Backblaze account
  • An ApplicationKey and ApplicationKeyID that gives read access to the private bucket
  • The name of the Backblaze B2 file server for the bucket
  • A Cloudflare account
  • A top-level internet domain (such as www.pawneeparks.org)
  • Cloudflare Web Worker access
  • The Cloudflare API key
  • The Cloudflare Zone ID
  • A Python3 program to get a refreshed authorization token and upload it to Cloudflare along with the worker code (available at Github>. (Once Cloudflare's recently announced Workers KV comes out of beta, we will update this article to store the authorization key in this distributed database.)
  • A server running cron (or something similar) to run the python3 update at regular intervals

Step by Step

Gather the information needed about your Backblaze B2 account

If you do not already have a Backblaze account, sign up for one here. Backblaze B2 includes 10 gigabytes of free storage and does not require a billing method to get started exploring the possibilities.Screenshot_2018-10-02_08.33.09.png

 

After creating (or signing into) your Backblaze B2 account, go to My Account on the top menu, and then select Buckets in the right hand menu

Screenshot_2018-10-02_08.42.41.png

This will open the bucket UI screen, and if this is a new account, it will have no buckets (yet).

Screenshot_2018-10-02_10.10.35.png 

Redirecting traffic requires having a bucket. Clicking on 'Create Bucket' brings up the bucket creation dialog.

Screenshot_2018-10-02_10.19.23.png

Choose a name (not the one in the dialog). Bucket names must be globally unique across all Backblaze B2 accounts. Choosing a name already in use will return an error; should this happen, simply choose another name. Users of the redirected content will not see this name.

Screenshot_2018-10-02_10.20.01.png

As long as we are here, get the BucketId for this bucket (this is a globally unique identifier). Next, upload a file (it does not matter what). Click on Upload/Download.Screenshot_2018-10-02_10.22.10.png

 

Click on upload, and send a text or HTML file up to Backblaze B2 (we will retrieve it as part of testing the integration) later. This file will be referred to later as uploadedFile.html.

Screenshot_2018-10-02_12.04.46.png

 Just drag and drop a file, and upload it to the Backblaze B2 bucket.

Screenshot_2018-10-02_12.12.43.png

Find the white i in a small gray circle at the far right of the file listing (circled in green), and click on that. It gives information about the file: we are looking for the fileserver for this account (all buckets from this account will utilize this particular fileserver). 

Screenshot_2018-10-02_12.14.35.png

As shown, the fileserver for this bucket is https://f001.backblazeb2.com. Make a note of the fileserver, as this is the top-level domain that is the target of the remap. Also note that the filepath is <fileserver>/file/<BucketName>/filename — this pattern is used to source content through the remapped domain (more detail on this later).

Next, get (or create) an applicationKeyId and applicationKey to generate authorization tokens. Once the file information dialog is dismissed, click on the 'Buckets' menu item in the left-hand menu to return to the main Buckets page. Near the top of the page, is a link to Show Account Key and Application Key.

Screenshot_2018-10-02_12.46.37.png

Click on this link to go to the key management.Screenshot_2018-10-02_13.02.46.png

Although it is possible to use the AccountId and AccountMasterKey, it is preferable to use a key with less access. Scrolling down this screen a bit gives:

Screenshot_2018-10-02_10.38.28.png

This will create an ApplicationKeyID and ApplicationKey (and this one is scoped to provide full read and write access to the bucket). Please note that the ApplicationKey is displayed exactly once. Although another key can be created with similar permissions, this particular key cannot be regenerated. This key gives access to your bucket, and should be kept securely. Application keys enable a great deal of flexibility in granting access to your stored content.

Screenshot_2018-10-02_10.40.38.png

Note the ApplicationKey (again: this is the only time it will be displayed) and the ApplicationKeyId for the bucket. The ApplicationKeyId, along with the KeyName, are listed (as are all keys created for an account).

This is all the information required from Backblaze.

 

Set up the Cloudflare account

If you do not already have a Cloudflare account, sign up for one at Cloudflare.

Screenshot_2018-10-01_16.01.33.png

and sign up:

Screen_Shot_2018-10-01_at_4.07.09_PM.png

After signing up, register your top-level domain with Cloudflare by going to 'add record'.  Ensure the record type is CNAME. This will map your top level domain (here static.pawneeparks.org) to the source that Cloudflare will fetch content from.

Click on 'add record', and then make sure the cloud icon directly to the right of the 'Automatic TTL' choice box and directly to the left of the 'Add Record' button is orange; if it is gray, click it once to change the setting from 'DNS only' to 'DNS and HTTP Proxy'.

The next step requires adding billing information, and a subscription to web workers. After this is accomplished, go the the workers page, and launch the editor.

Screenshot_2018-10-02_14.44.26.pngDo not modify the default script (there is no need). However, the script must be saved by clicking the 'save' button (circled in green). Clicking this button may not appear to do anything, but it is absolutely required for the next step, which is to specify the route on which the worker script is enabled.Screenshot_2018-10-02_14.53.20.png

Click on the 'routes' tab (circled in green) to show the routes.

Screenshot_2018-10-02_15.04.43.png click 'Add Route' to add a route, and it should show up.

Screenshot_2018-10-02_15.31.31.png

and the script will show as enabled for this script:

Screenshot_2018-10-02_15.21.33.png

 

The routing is set up, and the script will be taken care of automatically by python script, but we will need an API key for Cloudflare. Click on 'Dashboard' to return to the main Cloudflare interface, and then go to the Overview.Screenshot_2018-10-02_15.44.05.png

 

Several things require attention here. First, SSL should be set to Full (Strict) to ensure that Cloudflare verifies certificates from Backblaze B2 storage. Next, the Zone ID is required to upload our worker script, as is the API key. After noting the Zone ID, click on 'Get your API key'.

Screenshot_2018-10-02_15.55.21.png

and scroll down to the bottom of this page. The final section is API Keys. The required information is the global API key. Click View to display the API key, and make note of it. Cloudflare will require account verification.

Screenshot_2018-10-02_16.00.58.png

It will reveal your API key:

Screenshot_2018-10-02_16.01.58.png

Please note: under no circumstances display your API key in a public forum.

 

Setting up the authorized web worker with Cron

Since Backblaze B2 authorization tokens expire, feeding a CDN requires updating the authorization token. The simplest way to do this is to replace the entire worker script with a new one, where the authorization token is hard-coded into the Javascript. To make this easier, here is a script which, given the identities, identifiers, and keys, will get a new authorization token from Backblaze, embed it into a web worker script that will add the token as a header to incoming requests, and then upload the script to Cloudflare. By default, the script's authorization tokens are valid for one week (the maximum possible time). If this script is scheduled to run once a day, then the script can be missed for five days before the authorization token expires.

Setting up a Python script to run at regular intervals is beyond the scope of this guide, as is setting up a Python3 environment. This is the Python3 script to get a B2 authorization token and upload a web worker to authorize Cloudflare requests to the private bucket is available.

This script requires some customization, as a number of values are specific to the user.

cloudflareEmail
The email address registered as the account owner in Cloudflare
bucketSourceId
The hexadecimal bucket identifer for the source bucket in Backblaze B2
bucketFilenamePrefix
The filename prefix (if any) for which the B2 ApplicationKey is valid
cfZoneId
The Zone ID for the Cloudflare account
b2AppKey
This is the Backblaze B2 Application Key to authorize access to the Backblaze B2 bucket. This is the secret key that is displayed exactly one time and never again.
b2AppKeyId
This is the Backblaze B2 Application Key ID. It is also a long string, but it is displayed in the list of existing keys.
maxSecondsAuthValid
The number of seconds for which the authorization is valid when created. The default script value is a week, which is the maximum time-to-live of any authorization token. The valid time may be set to a smaller value. However, this should be weighed against how often the cron job will run. If the cron job runs once a day, then if it is skipped for a day or two or even five, the authorization remains active if the authorization token lasts for a week.

 

Have more questions? Submit a request

2 Comments

  • 0
    Avatar
    max.gashkov

    That's a lot of work. All your competitors allow to set up a CNAME with custom domain pointing to bucket. And this will work perfectly with Cloudflare too, but w/o workers and cron scripts updating access credentials.

  • 0
    Avatar
    Nathan Verrilli

    Hello Max

    To an extent, you're correct, and that is why I laid it out in detail -- to make it easy. We do intend to simplify both the process (mapping a CNAME to a bucket) as well as the authentication. We also wanted to make integration with Cloudflare available today.

    Today, we have a working solution. On some tomorrow, we'll have a better one!

    Thank you!

    Nathan

Article is closed for comments.
Powered by Zendesk