Preface

In my previous article (Home Data Center Series: Chevereto Image Hosting Multi-Node Scheduled Auto-Sync Tutorial), I mentioned that after building a ”same-city dual-center” and ”local cloud + public cloud hybrid cloud” structure consisting of the main site of my currently self-hosted Chevereto image hosting service (located in the home data center) + a backup site (located in the home data center) + a same-city disaster recovery site (located on a Tencent Cloud server) (while achieving real-time image synchronization + scheduled automatic database import), I also wanted to build an off-site disaster recovery site for the Chevereto image hosting service, so as to ultimately evolve into the ultimate form: ”three centers in two locations”.

After careful consideration, I decided to choose Cloudflare R2 object storage to achieve this, because of its ”10GB free storage space” and ”unlimited traffic” features (compared to Tencent Cloud COS's outbound traffic fees, you won't know until you compare, and the difference is shocking~). Plus, I was already planning to use a Hexo-based static blog deployed on Cloudflare Pages to implement an off-site disaster recovery solution for my main WordPress blog site. Cloudflare Pages + Cloudflare R2 is just a perfect match, isn't it? So, it's settled!

However, the underlying technology supporting my current Chevereto synchronization solution is Syncthing, which is not suitable for syncing my local images to Cloudflare R2, so a new solution must be introduced: rclone.

Building an Off-Site Disaster Recovery Image Hosting Service Using rclone and Cloudflare R2

Introduction to rclone

In fact, in previous articles, I mentioned rclone when comparing existing synchronization solutions on the internet (see:Docker Series: Detailed Tutorial on Implementing Multi-Point Folder Synchronization Based on Syncthing Using Docker), and also briefly introduced its advantages:

However, at that time, Syncthing was more in line with my needs in the specific scenario of ”folder data synchronization”. Now, when facing the scenario of ”syncing images from the local Chevereto image folder to Cloudflare R2”, rclone is the best choice.

---

In fact, in the migration of this kind of image hosting solution, a key issue is the compatibility of image URLs. For example: the real address of the previous image ishttps://image.tangwudi.com/images/2024/05/21/202405211354063.png, and its relative path address is:images/2024/05/21/202405211354063.png, and the storage path of images in Chevereto happens to be like this:

So Chevereto's image storage is very well-organized. Therefore, when migrating the image hosting service, if the migration destination can ultimately output the same image addresses, then the migration is considered successful. COS-type solutions can achieve exactly this, by simply copying the entire `images` directory in the Chevereto image hosting service to the R2 bucket.

---

Batch Uploading Chevereto Image Hosting Files to R2 Using rclone

Prerequisites: Creating an R2 Bucket

Since R2's AccessKey and SecretKey will be needed later when configuring rclone, you need to create an R2 bucket first (you need to enable R2 first, which requires binding a credit card, but this step will not incur any fees).

Installing rclone

First, install rclone on macOS. You can install it using brew:

brew install rclone

Alternatively, you can download the package suitable for you directly from the rclone official website. The official website link is as follows:https://rclone.org/downloads/：

Note: If you choose to download directly, just unzip it and run the rclone executable file directly.

Initializing rclone

Add the rclone path to the environment variables of the current account (this should not be necessary if installed via brew):

vim ~/.bash_profile #编辑当前账号的环境变量

Modify it according to the actual path of rclone. I put the unzipped rclone folder into the Applications folder, so I added its path to the end of PATH:

PATH=/bin:/usr/bin:/usr/local/bin:/Applications/rclone: 

After saving and exiting, use the following command to reload the environment variables:

source ~/.bash_profile

Run the following command to initialize rclone:

rclone config

Note: If you choose y, you will enter advanced options (dozens of them). Actually, only the first one needs to be set, which is to change the bucket's ACL to public-read:

For other options, just press Enter all the way to use the default values. The final configuration is as follows:


In fact, all the operations mentioned aboverclone configare aimed at generating the ”rclone.conf” configuration file, so we can actually just create a new configuration file and edit its content directly. The path of the rclone configuration file in macOS is:

/Users/your-username/.config/rclone/rclone.conf

I will post the configuration file content directly, so you can use it directly:

[images-bed]
type = s3
provider = Cloudflare
access_key_id = xxxxxxxxxxx  #改为自己access_key_id
secret_access_key = xxxxxxxxxx #改为自己的secret_access_key
region = auto
endpoint = https://xxxxx.r2.cloudflarestorage.com #改为自己的endpoint
bucket_acl = public-read
description = cloudflare R2

Batch Uploading Chevereto Image Hosting Images to R2 Using rclone

type = s3provider = Cloudflareaccess_key_id = xxxxxxxxxxx # change to your own access_key_id

rclone tree images-bed:images-bed #根据自己实际设置进行修改

region = autoendpoint = https://xxxxx.r2.cloudflarestorage.com # change to your own endpointbucket_acl = public-read

rclone copy -P -v /Volumes/data/docker/chevereto/data/images images-bed:images-bed/images

Note: In the above command, the-Pparameter displays the transfer progress in units of 500 milliseconds (default is one minute),-vfollowed by the actual path of the images directory in my Chevereto, and the finalimages-bed:images-bed/imagesrefers to the images directory on the bucket (images-bed) created on the previously defined R2 remote storage (images-bed). Through this format, we can achieve the effect of consistent output paths mentioned earlier.
Wait for the upload to complete:

Use theprovider = Cloudflare command again to confirm:

Configuring Custom Domains on Cloudflare R2

After the images are uploaded, the ”images-bed” bucket on R2 is inaccessible by default, as shown in the figure below:

At this point, you need to configure a custom domain for the ”images-bed” bucket on R2:




It has already taken effect. Let's prove it with an experiment by opening any image using its original link:

Replace the original domain name in the red box of the image above withr2image.tangwudi.comThen press Enter again to open:

Success. If you want to completely replace my Chevereto image hosting with R2, you only need to delete the originalimage.tangwudi.comresolution, and then change the custom domain of images-bed on R2 toimage.tangwudi.com.

Note: rclone has many other features, including mounting remote storage locally, etc. I won't introduce them one by one, mainly because I don't have much need for them myself. If you are interested, you can research them on your own; there are also many related tutorials online.

Afterword

rclone itself is just a simple, one-time execution program. If you want to achieve automatic execution based on set conditions, such as automatic execution at startup, scheduled automatic execution, etc., you need to rely on external operations to cooperate, such as functions provided by the operating system itself, like startup programs, scheduled tasks, etc.

Generally speaking, the most common way to use rclone is to run it at startup and run backups periodically. On macOS, you only need to use the built-in launchctl or crontab in cooperation with rclone to achieve this. Other operating systems have their own implementation methods. However, my requirement is that once there is a change in Chevereto's images folder (new images added), rclone will automatically sync to R2. Otherwise, would I have to manually run rclone every time I add a new image? That would be too low-tech. So I did some research and found that fsnotify seems to be able to do it, but that would require writing a dedicated article to study. Let's stop here for now. For the time being, I can only use a very low-tech way: manually running rclone to upload new images after they are added.

Note: I plan to try switching my image hosting to R2. I used to feel that opening images was a bit slow (mainly because cached images cannot enjoy the benefits of optimized IPs). Let's see how it works after switching to R2. After all, there is an automatic location default option when creating a bucket, though I don't know if it's useful.