Preface

Actually, at first thought, the demand for folder synchronization seems to have many solutions online. After searching for a while and combining my own experience, I simply divide these related solutions into 3 categories:
Single-app solutions
Represented by Sync Folders, available for Windows and macOS within a LAN, simple to use. The disadvantage is that it can only be used within a LAN and only supports Windows and macOS. To be fair, it is still very easy to use:

Cloud drive solutions
Represented by Nextcloud:

This type of personal cloud drive application often requires deploying the server side first, and then other nodes that need to sync data connect to the server side by installing the client side. The advantage is that it can be used across WANs. The disadvantages are that the client side is limited to desktop systems, deploying the server side requires a public IP address, and the client sync speed is limited by the connection speed between the client and server sides (speed issues can be resolved by deploying a CDN for the server side).
Backup and sync solutions based on third-party storage

Represented by rclone and Duplicati, these solutions store files that need to be backed up or synced on third-party storage, and then deploy applications on each node to achieve the purpose of backing up or syncing files by accessing the third-party storage: for example, rclone supports over 40 types of cloud storage, ranging from common cloud drives (OneDrive, Google Drive, Dropbox, etc.) to cloud object storage (Amazon S3, Alibaba Cloud OSS, etc.) and various transmission protocols (FTP, SFTP, WebDav).

Among these 3 types of solutions, the third one has the widest adaptability. Taking rclone as an example: it can support cross-WAN and simultaneously support both desktop systems and CLI interfaces. However, the trouble is that it requires using third-party storage as a transit (or having a public IP to self-host FTP, SFTP, WebDav). Essentially, the main purpose of this type of solution is to back up local data to cloud storage or pull data from cloud storage back to local. Its primary purpose is still backup and recovery. Although cloud drive data can be mounted locally through methods like Alist (rclone itself supports mounting), this is still different from the traditional demand for folder data synchronization.

So, if I just want to simply and regularly sync a few directories across WAN between different systems (macOS, Linux, Windows) without messing around with other things, is there a way?

Yes, there is, and that is the professional network-based synchronization tool to be introduced in this article: Syncthing.

Syncthing implementation principles

Syncthing is actually just an executable program that does not require installation. It does not need to deploy a server side in advance like Nextcloud, nor does it need to rely on third-party storage like rclone. You only need to run Syncthing directly on each device that needs to sync data, and then configure it simply.

Both being executable programs, how does Syncthing compare to the previously mentioned Sync Folders in terms of applicability? This depends on whether the network environment that needs synchronization is within a LAN or a WAN, and whether the client side is a desktop client or a CLI client.
LAN environment
1. Pure desktop client
In this case, to be fair, if all nodes requiring folder synchronization are desktop clients, I think Sync Folders seems to work better. Syncthing does not have much of an advantage in this scenario.
2. Mixed desktop and CLI clients
In this case, you can use Syncthing: you can deploy Syncthing on all clients; or deploy Sync Folders on desktop clients and Syncthing on CLI clients, and finally use one client to deploy both Sync Folders and Syncthing to connect everything. However, this method requires paying attention to the synchronization direction: for example, you can set Syncthing on one CLI client to Send Only, and Syncthing on other CLI clients to Receive Only, and then sync the content to all desktop clients through the client that deploys both Sync Folders and Syncthing (Sync Folders can also achieve Send Only or Receive Only effects by setting the sync type).
WAN environment
In this case, whether it is a desktop client or a CLI client, just deploy Syncthing on all of them directly.

Why can Syncthing achieve cross-WAN synchronization directly without relying on third parties? This actually relies on Syncthing's discovery servers and relay servers.

The "Settings" - "Connections" interface of Syncthing is as follows:

After enabling the global discovery function, it can provide STUN services to help users perform NAT traversal when crossing WANs.

---

The two options "Global Discovery" and "Enable Relaying" mentioned in the figure above contain two important functions: Syncthing's connection announcement service and relay service.
Connection announcement service:
The role of the connection announcement service is that when one Syncthing endpoint actively adds another Syncthing endpoint, the other endpoint will receive a notification sent by the connection announcement service asking whether to accept. If this endpoint chooses to accept, the two endpoints will successfully add each other as remote devices.
Relay service:
The relay service actually contains two independent parts: the STUN service and the TURN service. The STUN service will receive information sent by both parties (network card IP, listening port, etc.). Of course, this information generally cannot be used directly (it may be intranet information). The STUN service needs to combine the real source address and port of the Syncthing data packets it receives to judge the NAT type (NAT1, 2, 3, 4) in front of the locations of the two endpoints, and determine whether hole punching is possible. If hole punching is successful, both parties can directly establish a peer-to-peer connection based on the information provided by STUN, and this file synchronization speed will be relatively fast. If hole punching is not possible, it can only be handled entirely by the TURN server, and this file synchronization speed will be relatively slow, depending on the bandwidth allocated to each user by the relay server (the specific speed needs to be divided by 2) and the number of people using it.

Because Syncthing has already deployed public discovery servers and relay servers on the Internet, the Syncthing we deploy can be used directly on the WAN. However, the user experience depends on our own NAT type, the communication effect between our deployed Syncthing client and the discovery and relay servers, etc. (the default discovery and relay servers provided are abroad, so the user experience in China will certainly not be very good~).

---

By the way, in the advanced settings of Syncthing, you can actually see the STUN-related settings. The STUN Server can also be directly set to the STUN server you built yourself:

This is because NAT traversal is actually based on the STUN service (for the principles of STUN and how to build your own STUN server, please refer to my other article:Docker Series: Using Docker to Build Your Own STUN/TURN Server Based on Coturn), so if you want to get the best results and you are not afraid of tinkering, you can use your own self-hosted STUN server, self-hosted discovery server, and self-hosted relay server. I will cover this separately in a later chapter.

Deploying Syncthing

Directly using the Syncthing executable program

The download link on GitHub is:https://github.com/syncthing/syncthing/releases

Actually, to be honest, syncthing on Linux is just an executable file. After downloading, just grant it execution permissions and run it directly. You don't necessarily have to mess with Docker, and you won't encounter any permission issues. Just note that it's best to run it using a non-root account with general permissions.

I still chose Docker for my deployment method, simply because unifying everything on Docker makes it easier to manage and remember: after all, as I get older, I'm afraid I won't remember how many scripts I've deployed in the future...

Setting up Syncthing using Docker

Creating directories on the host machine that need to be mapped inside the container

mkdir -p /docker/syncthing/config #用于存放syncthing配置文件
mkdir -p /docker/syncthing/data #用于存放需要同步的数据

Setting up Syncthing

The command in docker run format is as follows:

docker run --name syncthing  -d --restart=always \
  --hostname=syncthing #可选项 \
  -e PUID=1000 -e PGID=1000 \ #可选项，如果docker访问挂载的目录出现权限问题的时候可以采用，在宿主机里的使用id username命令来获取PUID和PGID
  -e TZ=Asia/Shanghai \ #指定时区
  -p 8384:8384 \ # 管理gui的端口，请根据自己实际环境情况指定
  -p 22000:22000/tcp \ #指定宿主机上tcp的监听端口，请根据自己实际情况修改，最好不变
  -p 22000:22000/udp \ #指定宿主机上udp的监听端口，请根据自己实际环境修改，最好不变
  -p 21027:21027/udp \ #发现协议使用的端口，请根据自己实际环境修改，最好不变
  -v /docker/syncthing/config:/config \ #挂载宿主机上的指定目录到容器内部特定路径，这里是配置文件所在目录
  -v /docker/syncthing/data:/data1 \ #挂载宿主机上的指定目录到容器内部特定路径，这里主要是需要同步的数据所在的目录,可以指定多个，比如data2，data3等，也可以直接将宿主机上原有的需要同步的目录直接映射过来，不过需要注意权限问题，比如755改成777，否则可能会同步不了，不要问我怎么知道的。
  lscr.io/linuxserver/syncthing:latest

docker run --name syncthing -d --restart=always \

--hostname=syncthing # optional \

docker run --name syncthing  -d --restart=always --net=host \
  --hostname=syncthing \
  -e TZ=Asia/Shanghai \
  -v /docker/syncthing/config:/config \
  -v /docker/syncthing/data:/data1 \
  lscr.io/linuxserver/syncthing:latest

-e TZ=Asia/Shanghai \ # specify time zone \-p-p 8384:8384 \ # admin GUI port, please specify according to your actual environment \-p-p 22000:22000/tcp \ # specify the TCP listening port on the host machine, please modify according to your actual situation, preferably keep unchanged \

Configuring Syncthing

Initialization

Usehttp://宿主机ip:8384-p 22000:22000/udp \ # specify the UDP listening port on the host machine, please modify according to your actual environment, preferably keep unchanged \

-p 21027:21027/udp \ # port used by the discovery protocol, please modify according to your actual environment, preferably keep unchanged \


-v /docker/syncthing/config:/config \ # mount the specified directory on the host machine to a specific path inside the container, here is the directory where the configuration files are located \

-v /docker/syncthing/data:/data1 \ # mount the specified directory on the host machine to a specific path inside the container, here is mainly the directory where the data to be synchronized is located. You can specify multiple, such as data2, data3, etc., or you can directly map the existing directory on the host machine that needs to be synchronized, but pay attention to permission issues, such as changing 755 to 777, otherwise it may not synchronize, don't ask me how I know. \

lscr.io/linuxserver/syncthing:latest

Note: Please delete # and the comment content following it yourself.

Adding remote devices

Actually, if it is mainly used within a local area network (LAN) and requires active LAN discovery, and the ports 8384 tcp, 22000 tcp/udp, and 21027 udp are not in use, you can directly deploy using the host network type (Note: this deployment method is valid under Linux, but invalid under macOS). The docker run command format is as follows:

Adding the syncthing-intermini node

docker run --name syncthing -d --restart=always --net=host \

--hostname=syncthing \

---

-e TZ=Asia/Shanghai \

---

-v /docker/syncthing/config:/config \

-v /docker/syncthing/data:/data1 \

lscr.io/linuxserver/syncthing:latest

Create 3 syncthing nodes in sequence using Docker: syncthing-macmini, syncthing-intermini, and syncthing-tc. syncthing-macmini is deployed on my M1 Mac mini 8G base model, using

port mapping deployment method (Docker on macOS does not support host mode deployment); syncthing-intermini is deployed on my high-end mini PC with a 13th gen i5 CPU, located in the same LAN subnet as syncthing-macmini, using host mode deployment (for subsequent testing); syncthing-tc is deployed on a Tencent Cloud Lighthouse server, and like syncthing-macmini, usesport mapping deployment method (because even deploying in host mode would be useless).Access syncthing-macmini. Upon first access, the following prompt will pop up. You can choose freely, either Yes or No is fine:

After entering the interface, you first need to create an administrator username and password:

After the setup is complete, a login authentication page will pop up. Log in using the username and password you just created:

Adding the syncthing-tc node

Then enter the official configuration page:

Initialize syncthing-intermini and syncthing-tc in the same way.

Adding shared folders

Adding the shared folder on the main node first

Note: When initializing all nodes, do not add shared folders yet; we will set them up together later.

Start adding the other two nodes on the syncthing-macmini side.

Directly click "Add Remote Device" in the red box of the Remote Devices section:

Note: syncthing will create a new directory named ".stfolder" in the added sync folder. If this directory cannot be added due to permission issues, a prompt will appear on the web page indicating that the folder cannot be created. When you see this error, you should know that a permission issue has occurred: if it is a Docker environment, you should consider using-e PUID= -e PGID=parameters to specify reasonable real user permissions of the host machine, or change the permissions of the directory mapped from the host machine to the container from 755 to 777; if you are directly running the syncthing executable file, you can directly use the method of modifying directory permissions.

Sharing the shared folder on the main site with other nodes

Since I have already configured version control, ignore patterns, advanced, and other options when creating the folder earlier, here I can just select the 2 remote devices in Sharing and save directly, as shown in the figure below:

Then, the other two nodes will respectively receive shared folder requests from syncthing-macmini:


After clicking "Add" in the red box, you will enter the options for adding a sync folder:



Similarly, complete the same steps on the syncthing-tc node. Once finalized, you can see the synchronization status on the syncthing-macmini node:

Note: The sync speed depends on whether the syncthing nodes can choose the optimal path. If the optimal path is not chosen, it may happen that even though they are in the same local area network (LAN) where intranet transmission is the fastest, they end up using the slowest relay server. This is because the syncthing nodes misjudge each other's locations. This situation is very common in Docker deployment. However, if it is a Linux system deployed in host mode, the probability of correct identification within the same intranet segment is very high. But it doesn't matter; as long as you follow the method we discussed earlier, you can manually specify the correct address in the address list of the remote devices on each node.

Advanced Usage 1: Self-hosting discovery servers and relay servers

Theoretically, as long as we deploy syncthing in an internet-connected environment, we can generally connect to the public syncthing discovery servers and relay servers on the internet. However, some special environments cannot be ruled out, such as pure intranet environments, or ISP network restrictions that prevent connection to public discovery and relay servers. In such cases, if we have the conditions (home broadband with public IPv4, cloud hosts with fixed public IPs, or a controllable deployment environment in a pure intranet), we can build our own discovery and relay servers. I won't go into detail here; I will just paste the Docker commands, and those interested can research it themselves.
Discovery server docker run command format:

docker run --name syncthing_discovery_server  -d --restart=always \
   -e PUID=1000 -e PGID=1000 \ #和前面一样，也是可选项
   -p 8443:8443 \
   -v /docker/syncthing/discosrv:/var/stdiscosrv \
   syncthing/discosrv:latest

-e PUID=1000 -e PGID=1000 \ # Same as before, this is also optional

docker run --name syncthing_relay_server  -d --restart=always \
   -e PUID=1000 -e PGID=1000 \
   -e pools="" \ #这里很重要，如果不加这个，你搭建的中继服务器默认会直接发布到网上公用中继服务器的列表中
   -p 22067:22067 \
   -v /docker/strelaysrv:/var/strelaysrv \
   syncthing/relaysrv:latest

---

The commands and images for the discovery server and relay server above are officially provided, split into 2 independent Docker containers. Some people online have also combined these two into one Docker container, with the command as follows:

docker create  \
   --name=syncthing-relay-discosrv \
   -p 22067:22067 \
   -p 22070:22070  \
   -p 8443:8443  \
   --restart unless-stopped  \
   johngong/syncthing-relay-discosrv:latest

I haven't tried it, but those interested can give it a try.

---

The device IDs of the self-built discovery server and relay server can be found in the logs of the corresponding Docker containers, usingdocker log container_name, for example, the device ID of the discovery server I set up:

The device ID of the relay server:

In Syncthing's "Settings" - "Connections" interface, change the original "default" to the corresponding discovery server and relay server addresses. You can also keep "default" and write the relay server and discovery server addresses as "default,relay://" and "default,https://" respectively:

Note: If it is a pure intranet environment, even if it is not in the same network segment, you only need to set up a discovery server, and a relay server is not needed. However, you may need to manually add it using "tcp://peer_ip:22000" in "Remote Devices" - "Advanced" - "Address List".

Advanced Usage 2: Directly using in conjunction with virtual networking

Taking the virtual networking implemented by Tailscale as an example, directly use the Tailscale address in the address list of the remote device syncthing-tc on syncthing-macmini:

The advantage of this method is that you don't need to open any firewall ports on the Tencent Cloud server, because Tailscale directly enters the device kernel through WireGuard VPN, bypassing the traditional TCP protocol. If all nodes have Tailscale deployed, they can directly add each other using their respective Tailscale addresses, completely eliminating the need for a relay server.

Afterword

If purely from the perspective of synchronization, Syncthing is indeed the most convenient: it doesn't require deploying a server side, doesn't rely on third-party storage, supports both GUI and CLI, and is actually very simple to use.

I looked at various problems encountered by netizens, which are either permission issues or a lack of understanding of the working mechanism of the discovery server and relay server, resulting in slow speeds due to unoptimized configurations (not manually modifying the peer IP in the address list). In fact, if configured correctly, Syncthing is still very pleasant to use (for me, intranet communication goes through the intranet IP, and wide area network communication goes through the Tailscale IP).

My current 3 nodes: syncthing-macmini is actually the main WordPress site, syncthing-intermini is the hot standby site of the main WordPress site in the intranet, and syncthing-tc is the mirror site of the main site deployed on the Tencent Cloud host. The current effect achieved is that the main site is set to automatically back up once a week to the local directory (/docker/wordpress/html/wp-content/wpvividbackups):

After Syncthing detects that the main site's WPvivid has completed its scheduled backup, it will automatically sync to the same path of the WPvivid plugin on the hot standby site and mirror site. Then, you only need to click the scan button in the red box below in the WPvivid plugin of the hot standby site and mirror site:

The backup synced from the main site will then appear:

Then simply click restore to update to the content of the most recent backup of the main site. This is a bit more technically elegant than manually uploading the backup and then restoring it, even though the time difference is not much.

Having this need was unavoidable, because when WPvivid's scheduled tasks back up to a remote specific site, you can only choose one site (I currently have 2: a hot standby site and a mirror site, and maybe more in the future), and it can only be via FTP or SFTP. The most annoying part is that when scanning and restoring on the remote site, it requires entering the FTP or SFTP information (IP, port, username, and password), which is so annoying, especially since I often forget it... Now with Syncthing, this pain point is finally resolved, and I feel much better.