Preface

In my previous article (Tutorial on how to synchronize multiple nodes of Chevereto image bed in home data center series) mentioned that after building the "dual centers in the same city" and "hybrid cloud of local cloud + public cloud" structure consisting of the main site (located in the home data center) + backup site (located in the home data center) + disaster recovery site (located in Tencent Cloud Server) of the current self-built image storage chevereto (while realizing real-time synchronization of pictures + automatic import of scheduled databases), I also want to build a remote disaster recovery site for the chevereto image storage so that it can eventually evolve into the ultimate form: "two locations and three centers".

After careful consideration, I decided to use cloudflare R2 object storage because of its "10GB free storage space" and "unlimited traffic" features (compared with the outbound traffic fee of Tencent Cloud COS, I was shocked by the comparison). In addition, I was also going to use the Hexo-based static blog deployed on cloudflare pages to implement the remote disaster recovery solution for the main WordPress blog site. Cloudflare pages + cloudflare R2, it's a perfect match, right? That's it!

However, the underlying support technology of my existing chevereto synchronization solution is syncthing, which is not suitable for synchronizing my local images to cloudflare R2, so a new solution must be introduced: rclone.

Using rclone and cloudflare R2 to build a remote disaster recovery image bed

Introduction to rclone

In fact, in previous articles, I mentioned rclone when comparing existing synchronization solutions on the Internet (see:Docker series: A detailed tutorial on how to synchronize multiple folders using Docker based on syncthing), and briefly introduced its advantages:

But at that time, syncthing was more in line with my needs in the specific scenario of "folder data synchronization". Now, when faced with the scenario of "synchronizing the pictures in the local chevereto picture folder to cloudflare R2", rclone is the best choice.

---

In fact, in the migration of this kind of image hosting solution, a very critical issue is the compatibility of the image address. For example: the real address of the previous image ishttps://image.tangwudi.com/images/2024/05/21/202405211354063.png, its relative path address is:images/2024/05/21/202405211354063.png, and the storage path of the pictures in chevereto is exactly like this:

Therefore, the storage of chevereto image storage is still very regular. Therefore, when migrating the image storage, if the migration destination can eventually output the same address of the image, then the migration is considered successful. The COS type solution can just achieve this. Just copy the entire directory of images in the chevereto image storage to the bucket of R2.

---

Use rclone to batch upload chevereto image bed files to R2

Preliminary preparation: Create an R2 bucket

Since you will need R2's Accesskey and SecretKey when configuring rclone later, you need to create an R2 bucket first (you need to activate R2 first and bind a credit card, but this step does not incur any fees).

Install rclone

First install rclone on macos, you can use brew to install it:

brew install rclone

You can also download the package that suits you directly from the rclone official website. The official website link is as follows:https://rclone.org/downloads/:

Note: Use the direct download method, decompress it and run the rclone executable file directly.

Initialize rclone

Add the rclone path to the environment variables of the current account (this should not be necessary if installed using brew):

vim ~/.bash_profile # Edit the environment variables of the current account

Modify according to the actual path of rclone. I put the folder after decompressing rclone in the application folder, so add its path to the end of PATH:

PATH=/bin:/usr/bin:/usr/local/bin:/Applications/rclone: 

After saving and exiting, use the following command to reload the environment variables:

source ~/.bash_profile

Run the following command to initialize rclone:

rclone config

Note: If you select y, you will enter the advanced options, which are dozens of options. In fact, only the first one needs to be set. Change the acl of the bucket to public-read:

Press Enter to use the default values for other options. The final configuration is as follows:


In fact, the aboverclone configThe purpose of all operations is to generate the "rclone.conf" configuration file, so we can actually create a new configuration file and edit its contents directly. The configuration file path of rclone in macos is:

/Users/your-username/.config/rclone/rclone.conf

I will paste the contents of the configuration file directly, and you can use it directly:

[images-bed] type = s3 provider = Cloudflare access_key_id = xxxxxxxxxxx #Change to your own access_key_id secret_access_key = xxxxxxxxxx #Change to your own secret_access_key region = auto endpoint = https://xxxxx.r2.cloudflarestorage.com #Change to your own endpoint bucket_acl = public-read description = cloudflare R2

Use rclone to batch upload chevereto image hosting images to R2

1. Userclone treeRun the command to verify whether the images-bed bucket on R2 can be accessed normally:

rclone tree images-bed:images-bed # Modify according to your actual settings

2. Userclone copyThe command uploads the images from the local chevereto image bed to the images-bed bucket on R2:

rclone copy -P -v /Volumes/data/docker/chevereto/data/images images-bed:images-bed/images

Note: The above command-PThe parameter is to display the transfer progress in units of 500 milliseconds (one minute by default).-vThe parameter is followed by the actual path of the images directory in my chevereto. The lastimages-bed:images-bed/imagesIt refers to the images directory on the bucket (images-bed) created on the R2 remote storage (images-bed) defined earlier. Through this format, we can achieve the effect of consistent output path mentioned earlier.
Wait for the upload to complete:

3. Use againrclone tree Command to confirm:

Configuring custom domains on Cloudflare R2

After the image is uploaded, the "images-bed" bucket on R2 is inaccessible by default, as shown below:

At this time, you need to configure a custom domain for the "images-bed" bucket on R2:




It has taken effect. Let's use an experiment to prove it. Open any picture using the original link:

Replace the original domain name in the red box above withr2image.tangwudi.comThen press Enter again to open:

Success. If I want to completely replace my chevereto image hosting with R2, I only need to delete the originalimage.tangwudi.comThen change the custom domain of images-bed on R2 toimage.tangwudi.comThat's it.

In addition: rclone has many other functions, including mounting remote storage to local storage, etc. I will not introduce them one by one, mainly because I don’t have any needs for them myself. If you are interested, you can study it on your own. There are also many related tutorials on the Internet.

Afterword

rclone itself is just a simple one-time execution program. If you want to implement automatic execution based on set conditions, such as automatic execution at startup, automatic execution at scheduled time, etc., you need to rely on external operations to cooperate, such as the functions provided by the operating system itself, such as: automatic startup at startup, scheduled tasks, etc.

Generally speaking, the most common way to use rclone is to run it at startup and run backups regularly. On the macos system, you only need to use the built-in launchctl or crontab with rclone. Other operating systems have their own implementation methods. However, my requirement is that once the chevereto image folder changes (new images are added), rclone will automatically synchronize it to R2. Otherwise, do I have to manually run rclone every time I add a new image? That's too low, so I studied it and found that fsnotify seems to be able to do it, but that requires another article to study. Let's stop here for now. For the time being, I can only use a very low method: manually run rclone to upload new images after adding new images.

In addition: I plan to change the image storage to R2. I always felt that the images were a bit slow to open before (mainly because cached images cannot enjoy the treatment of preferred IP). Let's see how the effect is after changing to R2. After all, there is a default option for automatic location when creating a bucket, and I don't know if it is useful.