1 Introduction

Since my blog adopts a special WordPress active-active architecture (the Mac mini in my home data center serves as the primary-write/secondary-read node, and the Racknerd VPS in Chicago serves as the primary-read node), once there is a content change on the Mac mini node (all blog content modifications and new post publications are completed on it), a script is required to export the WordPress database from MariaDB on the Mac mini node as a wordpress.sql file. This file is then transferred to a designated directory on the Chicago node using the scp command via an SSH connection (originally, Syncthing based on Tailscale IPs was used for synchronization, but outbound Tailscale has basically been rendered useless, so I switched to scp). Subsequently, once the inotify application on the Chicago node detects the file change, it automatically executes a local script to import the WordPress database locally, thereby achieving complete synchronization of the WordPress database content between the two nodes.

This workflow had been running very well before and never caused me any worry. However, a few days ago, I suddenly noticed that things went awry during certain periods: the status of scp kept switching between "stalled" and "ETA", and after reaching 20%, the transfer speed became slower and slower until the connection was finally closed:

Switching to rsync while limiting the average transfer speed didn't work either:

At first, I thought “a certain Wall” was up to some tricky business again, such as performing deep packet inspection on specific protocols or data patterns and resetting the connection. But after careful observation later, I found that this issue was highly correlated with the time of day: it became exceptionally bad every night after 23:00, while it was much better during the day, and many times rsync could even finish transferring successfully. If it were really the influence of “the Wall”, reasonably speaking, it wouldn't just occur frequently at night.

Digging deeper for the cause, I finally pinned the culprit down to the throttling policy of Racknerd's Chicago node itself. This is because during the most severe periods—for example, 22:00 Beijing time, which corresponds to 9:00 AM Chicago time, just when their workday peak traffic begins; and by 23:00 Beijing time, it is exactly 10:00 AM in Chicago, which perfectly aligns with the time slot when office workers use the network on a large scale. Conversely, when I tested the transfer during the day in China (such as 9:00 or 10:00 AM), it was often early morning or night in Chicago, a low-peak period for network usage, where speed limiting and packet loss were much lighter. Whether using scp or rsync, there was a fairly high chance of successfully completing the file transfer.

After doing some research, I found that this seems to be a common bandwidth protection method used by many cheap VPS providers: they usually schedule traffic, apply shaping, or enable more aggressive traffic scrubbing rules on single connections (single TCP sessions) during the data center's morning peak hours (i.e., local working hours) to guarantee the overall availability for more paying customers or the entire data center network. Because these low-cost VPSs are essentially oversold, the node bandwidth is far from enough to allow every virtual machine to run at full capacity at will. Especially for providers like Racknerd, which are famous for "low-cost annual payments," they are more likely to directly impose speed limits, packet loss, or even actively reset connections on sudden large file transfers during peak hours to suppress them.

In addition, this restriction does not only occur in SSH (such as scp, rsync -e ssh) transmission scenarios. It is actually a single connection with long-term continuous high traffic unified policy, and the following common situations are highly susceptible to it:

• Using scp or rsync to copy large files via SSH (typical single TCP stream, with continuous and stable packets)
• FTP passive or active transmission of a single large file is also easily targeted
• Running large files on overlay networks like Tailscale / ZeroTier (although they are UDP, they are often treated as abnormal traffic on outbound international lines)
• Downloading large files directly with curl or wget, which also easily triggers shaping due to continuous high bandwidth on a single connection
• Even on the Web, a single aria2c multi-threaded download can often bypass the restriction, whereas a single-threaded HTTP GET will be speed-limited or disconnected

---

For cheap VPS providers like Racknerd, this kind of single-connection speed limit or traffic scheduling barely affects most of their target customer groups (small websites, lightweight APIs, daily O&M). Only a very small number of users who need frequent large file transfers or continuous full bandwidth will find it a headache. And this is precisely the ”gray balance” they tacitly allow through price and resource control.

---

2 Solutions

After encountering this problem, I actually thought about it carefully, and there are roughly three main solutions I can think of.

First: HTTP/HTTPS service

First, of course, I thought of the solution that best fits internet intuition: temporarily set up an HTTP or HTTPS service on the Mac mini, and then let the Chicago node actively pull the file using tools like aria2c or wget -c that support multi-threaded resumable downloads. After all, HTTP is naturally suitable for file transfer, can easily handle segmented downloads, and multi-threading can also well avoid Racknerd's strategy of limiting single-connection speeds. In addition, I have long used Cloudflare Tunnel to securely expose the Mac mini's website to the public network. Even if the ICP filing domain has been canceled, I can still rely on Tunnel to provide access services, without worrying at all about domestic operators directly blocking inbound HTTP traffic.

But on second thought, doing this is somewhat like using a sledgehammer to crack a nut. My scenario is actually very simple, nothing more than transferring a SQL file of less than 100MB each time. For this, I would have to specially configure multi-site directories, add access authentication, and also consider Cloudflare's caching rules, which seems overly tedious. As for Tailscale or other WireGuard-based intranet penetration tools, they are even less suitable for this kind of cross-border high-traffic scenario—under my current line, the stability of Tailscale can no longer be relied upon, and large files often easily drop connections. I dare not bet everything on it for such an important task.

Second: Cloud storage transit

That is, simply upload the files to a cloud drive or object storage, such as Google Drive, OneDrive, Alibaba Cloud OSS, or Tencent Cloud COS, and then let the Chicago node download them from there. This method is actually very reliable because the HTTPS traffic of these big-name clouds is sufficiently ”mixed.” Even on a cheap VPS like Racknerd, their upstream ISPs wouldn't dare to arbitrarily throttle speeds or drop packets selectively, making the download process rock-solid. What's more, these object storages or cloud drives natively support resuming interrupted downloads and multi-threaded downloading; just running aria2 gets the job done, so you don't have to worry no matter how large the file is. But looking at it from another perspective, for my current scenario—where I only need to transfer a few tens of megabytes of SQL files a few times a week—this approach seems like complete overkill. Uploading and then downloading takes an extra detour; what usually takes 1 minute to finish now requires tossing back and forth, which feels a bit like ”using a sledgehammer to crack a nut” and is not worth it.

The third: Resumable transfer

This approach becomes the most suitable solution for me: still using the simplest rsync to transfer files through a single SSH connection, and then utilizing the features of “–partial –append-verify” to continue accumulating the transfer from the breakpoint after a transmission interruption. Combined with the retry mechanism added to the script, each failure waits for 10 seconds before trying again, with a maximum of 5 or 6 attempts, which almost guarantees that the file is completely transferred to the Chicago node within a limited time. Although this method will still be targeted by the single-connection rate-limiting policy of the Racknerd data center during peak hours, because the file is not large to begin with, transferring it in segments like multiple ”nibbles” actually makes it easier to get things done in a real-world environment. To be honest, the biggest advantage of this solution is that there is no need to install any extra services, no need to run an HTTP server, and no need to consider configuring complex object storage. As long as the existing SSH can run, it can be done, which is both worry-free and elegant.

So the final conclusion is very simple: if one day I want to transfer a movie library of several hundred GB, I will definitely honestly go with COS or OSS, combined with multi-threaded downloading; if I only transfer files within overseas regions, HTTP would definitely be very sweet; but like now, where my home Mac Mini exports 50M SQL files a few times a week to send to the Chicago VPS, relying on rsync's breakpoint resume to accumulate multiple times is truly the most cost-effective ”right path”.

Note: Actually, there is another easiest way, which is to avoid the peak traffic hours during the day in Chicago and honestly transfer files during other periods. But, in the spirit of the guiding ideology: ”Go ahead when there are difficulties; create difficulties and go ahead even when there are none,” of course I have to face the challenge head-on. The key is that I can also churn out a blog post~.

3 Using rsync for Resumable Transfer

1 Introduction to rsync

rsync is an extremely commonly used file synchronization and transfer tool on Unix-like systems (Linux, macOS). It is famous for its efficiency, and its core feature is that it only transfers the parts of files that have actually changed (incremental transfer), and supports powerful compression and verification mechanisms. It can be used for backups between local folders, and can also securely transfer files between different servers through the SSH protocol.

Compared to more ”foolproof” transfer tools like scp and ftp, the advantage of rsync lies in its extremely rich features, allowing for precise control in more scenarios:

• It can continue incremental resume from the interruption point when part of the file already exists on the destination side (for example, through –partial + –append-verify). Even if the network is disconnected, the command can be rerun to continue transferring from the last progress, which is very suitable for trans-national networks prone to packet loss and disconnection.
• Using the -z parameter can enable compression during the transfer process, significantly reducing the bandwidth consumption of trans-national links, and the speed is usually faster.
• It can preserve all metadata such as file timestamps, permissions, and symbolic links at once through the -a (archive) parameter, which is very suitable for precise backup or server migration.
• Adding –delete can even make the destination directory strictly synchronized to the state of the source directory, directly deleting redundant files in the destination to avoid accumulating junk files over a long period.
• Through –chown=user:group, you can directly modify the file owner and group when syncing to the target machine, which is very suitable for scenarios requiring specific permissions like Web panels, such as using it in site directories like iPanel and BT Panel, making the synced files immediately available.
• Adding –progress can display the real-time transfer progress of each file; using –bwlimit=2000 (unit KB/s) can also limit the maximum bandwidth to avoid saturating the line.
• You can also use –exclude to flexibly exclude files that do not need to be synchronized, such as –exclude=.git or –exclude=*.log, to only synchronize the required content.

rsync can also cooperate seamlessly with SSH (rsync -e ssh), providing secure transfer while possessing advanced features such as breakpoint resume and incremental verification, making it almost the preferred tool for trans-national server backup and site synchronization.

---

It should be noted that rsync is a typical dual-ended tool, which requiresrsync to be installed on both the local and remote machines, because the remote side will invoke its own rsync via ssh to participate in hash calculation and data pulling. This is also why when executing rsync, it actually implicitly invokes the remote rsync within ssh, allowing secure transfer without needing to open a separate daemon or a dedicated service port like FTP. Because of this, rsync has become the preferred tool widely adopted in cross-border server deployment, backup, and synchronization scripts.

---

2 Version Requirements for Implementing Resumable Transfer

It should be noted that rsync's implementation of resuming interrupted transfers has version requirements (the –partial feature is supported by almost all versions, the –append feature requires version 2.6.4 or above, and the –append-verify feature requires versions on both ends to be 3.x or above). Although rsync is built-in in many systems (such as macOS and certain Linux distributions), the versions vary greatly. Taking my macOS as an example, the default is 2.6.9, which cannot support the –append-verify feature:

On the other hand, the rsync version installed via apt in Debian 12 on the Chicago node is 3.2.3, which can satisfy all the features of resuming interrupted transfers:

---

One of the biggest highlights of rsync is its ability to "resume interrupted transfers," which is practically a must-have for cross-border, easily interrupted network environments. However, its implementation is actually divided into different levels, and you need to explicitly tell it how to resume through certain parameters.

1、–partial

This is the most basic and earliest supported parameter in rsync (available almost since the very early 2.x).

• Effect: If the transfer is interrupted during the process (such as a network drop or manual stop), by default rsync will delete the target file currently being transferred to avoid leaving an incomplete file.
• With –partial added, rsync will keep this ”incomplete file” to be reused during the next transfer.
• Applicable versions:Supported by all common rsync (2.x and above)。

It actually only guarantees "not deleting files after interruption," which does not mean that the next rsync will directly resume uploading from the end of the file. Instead, it will still scan, compare, and calculate deltas, and then possibly retransmit certain blocks.

2、–append

• Supported since rsync 2.6.4(2004)
• Effect: When there is already a partial file on the target side, rsync will directly append data from the end of the file without performing content verification.
• Suitable for log-like scenarios where data is only added to the end of the file, such as constantly growing log files.

The disadvantage is: it does not perform hash verification on the existing part. If the previous file was corrupted or inconsistent, it will still append from the end, which can easily lead to file corruption without being noticed.

3、–append-verify

• Supported since rsync 3.0.0Introduced in (2008).
• It is an enhanced version of –append.
• Effect: It will also resume transferring from the end of the target file, but after the transfer is complete, it will perform a verification on the file to ensure it is complete and correct. If the verification fails, it will fall back to the traditional ”block-by-block comparison and transfer” process to ensure that the target file and the source file are ultimately consistent.
• Very suitable for our scenarios that require strict consistency, such as transferring database backups, tar packages, SQL files, etc.

Common combination: –partial –append-verify

In production scripts, you will often see it written like this:

rsync -avz --partial --append-verify -e "ssh -p 12345" localfile user@remote:/path/

The roles they play respectively:

• –partial ensures that if the transfer is interrupted, the partial files already downloaded to the target machine will not be deleted.
• –append-verify directly resumes transferring from the end of the file during the next run, and automatically performs a verification after the transfer is complete to ensure the file is complete and reliable.

In this way, even with cross-border lines that are prone to disconnection, the file can be stably transferred cumulatively by executing rsync multiple times, without wasting the data blocks that have already been transferred.

---

4 rsync ≠ scp + Resumable Transfer

Many people encounter rsync for the first time often just because they need a ”scp that can resume interrupted transfers.” Just like me, after repeatedly failing to transfer files across borders using scp, I hurriedly looked for an alternative and was ultimately attracted by ”rsync supports resuming interrupted transfers.” But in fact, rsync is far more than that, which reminds me of this picture again:

It can not only resume synchronization from the breakpoint after the network is disconnected, but also efficiently compare the file contents of the source and target to achieve true incremental transfer — even for a file of several hundred megabytes, as long as only a few KB are modified, rsync will only transfer those few KB of difference blocks. This algorithmic superiority is something simple tools like scp and ftp cannot achieve at all.

Below is a comparison using the same link at the same time to transfer the same WordPress database file using scp and rsync as an example.
scp took 12 seconds:

rsync only took 4 seconds:

This is even under the condition of a very good link where the transfer is completed in one go, rsync only took 1/3 of the time of scp. If the link is very bad and scp keeps retransmitting, then rsync, relying on resuming interrupted transfers, would be unimaginably faster. Not to mention that rsync can also add compression, preserve file timestamps and permissions, delete redundant files on the target side (–delete), and even directly modify the owner of the remote files via –chown, elevating it from a file synchronization tool directly to a deployment weapon.

Since its birth, it was created to ”keep both ends precisely consistent,” and resuming interrupted transfers is just the most visible corner of all its capabilities.

5 Using rsync to Synchronize Directories

1 Overview

When many people mention rsync, they think of using it to transfer a single large file (such as wordpress.sql in this article). But in actual production and O&M scenarios,the most common use of rsync is actually to synchronize directories — such as website code, static resources, backup files, media libraries, configuration file directories, and so on.

2 The Simplest Directory Synchronization

Let's first look at a most common and simple example:

rsync -av /local/site/ user@remote:/var/www/site/

What does this command do?

• -a (archive mode) will recursively enter subdirectories and preserve file permissions, ownership, timestamps, soft links, and other metadata, which is equivalent to the collection of -rlptgoD, making it very suitable for backup or deployment.
• -v (verbose) shows the detailed process.
• The trailing “/” in /local/site/ is very critical: with ”/” means synchronizing the contents of the directory to the remote side, instead of synchronizing the site directory itself; without “/” will copy the site directory itself over, becoming /var/www/site/site/.

---

Many people will write it like this for the first time:

rsync -av /local/site user@remote:/var/www/

And then find that an extra /var/www/site/ directory has appeared on the remote machine. If you just want to put the files in site into /var/www/site/, remember to add / after the local path.

---

3 Using --delete to Maintain Strict Consistency

If you are doing online deployment (such as updating a blog or static site), the most common addition is --delete:

rsync -av --delete /local/site/ user@remote:/var/www/site/

This means: keep the remote directory strictly consistent with the local directory; if there are extra files in the remote directory (which have been deleted locally), rsync will delete them for you. It is suitable forone-master to multi-slave synchronization, or scenarios with very high requirements for deployment consistency.

4 Ensuring Consistent Ownership and Permissions

On some VPS or production servers, you need to ensure that the owner and group are correct (such as the www-data or nginx user). In this case, you can use it with --chown:

rsync -av --delete --chown=www-data:www-data /local/site/ user@remote:/var/www/site/

This way, you don't have to worry about the local user and the remote user being different.

5 Adding Compression, Suitable for Transnational Lines

If you want to sync across countries (for example, from my home data center to Chicago), you can add -z to compress during transmission, which significantly reduces bandwidth consumption and increases speed:

rsync -avz --delete /local/site/ user@remote:/var/www/site/

6 Summary

Therefore, several common combinations of rsync in directory synchronization scenarios are:

• -a preserves meta-information (permissions, timestamps, symlinks, owner/group, etc.)
• -v prints the synchronization process
• -z enables compression, suitable for cross-border transfers
• –delete makes the target directory strictly consistent with the source
• –chown ensures the owner and group are consistent

This is also why, in almost all website automated deployments, NAS synchronizations, and backup scripts,rsync is used instead of tools like scp and ftp.

6 Installing the Latest rsync on macOS

Since the rsync version installed on the Debian 12 Chicago node already meets the requirements, there is no need to fuss with it. I only need to use Homebrew to install the new version of rsync on my Mac mini:

brew install rsync

The new rsync will be installed in /opt/homebrew/bin/rsync (Apple Silicon) or /usr/local/bin/rsync (Intel).

Confirm the rsync version with the following command:

/opt/homebrew/bin/rsync --version

Currently, the latest version of rsync installed via brew is 3.4.1:

If you don't want to use the absolute path (such as /opt/homebrew/bin/rsync) every time to call the new version of rsync installed by Homebrew, you can modify the shell's environment variable file to make it take effect automatically.

The specific method is:

Find your ~/.zprofile (Zsh login configuration file) or ~/.zshrc (Zsh interactive configuration file), and open it with vim ~/.zprofile or open -e ~/.zprofile (or any editor).

Add the following line to the file,and make sure it is written at the very end of the file(to avoid being overwritten by other exports later):

export PATH="/opt/homebrew/bin:/opt/homebrew/sbin:$PATH"

This line means putting Homebrew's executable path at the very front of $PATH, so the system will prioritize finding software installed by brew. However, never write it like this:

export PATH="/opt/homebrew/bin:/opt/homebrew/sbin"

Because this will discard all original system paths, which may cause basic commands like ls and cat to fail.

After saving the changes, return to the terminal and execute

source ~/.zprofile
hash -r

Finally, check with the following command:

which rsync
rsync --version

Confirm that the output is /opt/homebrew/bin/rsync and the version is 3.4.x, which means the configuration is successful. Taking my Mac mini as an example:

Note: To install and use rsync directly on Debian, use theapt update && apt install rsynccommand, so I won't write a separate chapter for it.

7 Using rsync's Resumable Transfer Feature to Transfer Files over SSH

After installing rsync on both my local home data center Mac mini and the remote Chicago node Debian 12, and correctly configuring SSH (it is recommended to first configure SSH public key access from local to remote; friends who are not familiar with this can refer to the article:Debian Series: Configuring SSH Public Key Login), you can directly use rsync to start transferring files. First, set the export path and target server information as follows:

# 设置导出路径，以我要导出的macmini本地的wordpress.sql文件为例：
EXPORT_PATH="/Volumes/data/docker/wordpress/db/wordpress.sql"

# 目标芝加哥vps信息，包括ssh用户名、目标vps的IP地址，ssh端口和远程路径
REMOTE_USER="root"
REMOTE_HOST="107.xx.xx.xx"
REMOTE_PORT="12345"
REMOTE_PATH="/docker/wordpress/db/wordpress.sql"

The final rsync command is as follows:

/opt/homebrew/bin/rsync -az --progress --partial --append-verify -e "ssh -p REMOTE_PORT" "EXPORT_PATH" "REMOTE_USER@REMOTE_HOST:$REMOTE_PATH"

Of course, you can also write it directly without defining variables, but it will look very long:

/opt/homebrew/bin/rsync -az --progress --partial --append-verify -e "ssh -p 12345" "/Volumes/data/docker/wordpress/db/wordpress.sql" "[email protected]:/docker/wordpress/db/wordpress.sql"

If an error occurs during transfer, simply run the command again to resume the transfer, and the file transfer will be successfully completed after repeating this multiple times.

8 Advanced: Using Shell Scripts for Automatic Retransmission

Although manually running the command repeatedly to achieve resumable transfer works, from a technical perspective, it is a bit low-tech. It's fine to use occasionally, but doing this frequently would definitely drive me crazy. Therefore, I ultimately had to use a shell script. Below is the content of the script I used for your reference:

#!/bin/bash"# 加载环境变量（供 ssh 调用时生效）source ~/.bash_profile # 设置导出的源路径EXPORT_PATH="/Volumes/data/docker/wordpress/db/wordpress.sql" # 目标服务器信息REMOTE_PORT" "EXPORT_PATH" "REMOTE_USER@REMOTE_USER="root"REMOTE_HOST="107.xx.xx.xx" REMOTE_PORT="12345"REMOTE_PATH="/docker/wordpress/db/wordpress.sql.tmp" FINAL_PATH="/docker/wordpress/db/wordpress.sql"# rsync 重试配置MAX_RETRY=10"REMOTE_PORT" "REMOTE_USER@RETRY_DELAY=10  1TP6 REMOTE_PATH | awk '{print \$1}'")LOCAL_MD5="(/sbin/md5 -q "EXPORT_PATH") echo "Local MD5:"LOCAL_MD5"echo "Remote MD5:"REMOTE_PORT" "REMOTE_USER@REMOTE_MD5" if [[ "LOCAL_MD5" == "

This is an automated script specially designed for unstable cross-border network scenarios, used to export the WordPress MariaDB database from a local Docker container and back it up to a remote node in Chicago. The process is as follows:

First, generate a complete SQL file of the WordPress database locally.

Then, leveraging rsync's resumable transfer (`--partial + --append-verify`) mechanism, automatically retry multiple times when encountering network fluctuations or speed limits that cause interruptions, maximizing the chance of a successful transfer.

Once the file transfer is complete, the script will calculate MD5 hash values locally and remotely for strict verification to ensure the file contents are completely identical.

Then, perform an atomic rename operation on the remote server to ensure that only complete and usable database files always exist on the target server.

After all processes are completed, the script will also send multi-channel push notifications to your phone via Bark, allowing you to stay updated on the backup results without having to stare at the terminal.

The entire process, from export, transfer, and verification to the final notification, is fully automated, greatly reducing the need for manual intervention and avoiding potential data consistency risks caused by unstable cross-border lines.

9 Afterword

Frustratingly, just as I was about to finish writing the article and was preparing to use a real-world case to verify the power of the rsync resumable transfer optimization solution, things completely went off the rails — the cross-border line suddenly became exceptionally smooth: transferring a 50+ MB SQL file not only succeeded on the first try, but the speed was ridiculously fast, almost instantly maxing out the bandwidth, finishing in just 3 seconds at its fastest:

I was completely dumbfounded on the spot. I even went out of my way to try it repeatedly after 23:30 at night, hoping to wait for the line to return to its usual peak-hour congestion so I could take a few typical screenshots of ”resumable transfer in action.” However, it was rock-solid every single time, finishing the job in less than 10 seconds at most. The speed was comparable to transferring files between domestic data centers, and the export script also succeeded on the first try:

This was so unusual that it almost made me laugh and cry at the same time — the optimization solution I had spent so much effort working out couldn't even be demonstrated in its most shining scenario simply because the network conditions were too good.

However, calming down and thinking about it, this isn't really some metaphysical phenomenon: the instability of cross-border networks is inherently reflected in their ”unpredictability.” Often, it is not necessarily congested during peak hours; instead, it is frequently related to temporary routing scheduling and ISPs” QoS (Quality of Service) policies. For example, backbone lines occasionally experience brief bandwidth releases, or data centers might not trigger single-connection speed limit thresholds during certain periods. Coupled with transit nodes like Cloudflare / HE.net potentially switching lines temporarily, this can occasionally lead to such unusual ”high-speed smoothness."

On the other hand, it might also be related to my local egress network temporarily hitting a higher-quality return path. Even at the international egress, if congestion at a certain peering point eases, it can instantly bring a massive boost to transfer speeds. This is also why we say ”cross-border networks are never something that can be fully predicted with scientific rigor”; they are always in a state of multiple dynamic schedulings and priority competitions.

Therefore, this test happened to hit the perfect storm of timing, location, and harmony, resulting in the rsync optimization solution failing to demonstrate its resumable transfer advantage in a ”worst-case scenario.” While it's a bit of a pity, it also shows from another angle: making these robustness designs is inherently a backup plan for the 20% of the time when one unfortunately encounters the worst lines. In most cases, it might just be redundant insurance, but when the day comes that I actually encounter problems and need it, I won't be caught in a flurry like before.