This article was last updated 280 days ago. The information in this article may have developed or changed. If it is invalid, please leave a message in the comment section.

Contents

1 Preface
2 Syncthing implementation principle
3 Deploy syncthing
- 3.1 Use the syncthing executable program directly
- 3.2 Use Docker to build syncthing
  - 3.2.1 Create a directory on the host that needs to be mapped to the container
  - 3.2.2 Building Syncthing
4 Configure syncthing
5 Afterword

Preface

In fact, at first glance, there seem to be many solutions for the folder synchronization requirement on the Internet. After searching for a long time and combining my own experience, I simply divided these solutions into three categories:
1. Single APP solution
Sync folders is a representative example. It is available in the Windows and MacOS systems within the LAN. It is easy to use. The disadvantage is that it can only be used within the LAN and only supports Windows and MacOS. To be honest, it is still very useful:

2. Network disk solutions
Take nextcloud as an example:

This type of personal network disk application often requires the deployment of the server first, and then other nodes that need to synchronize data can use it by installing the client to connect to the server. The advantage is that it can be used across a wide area network. The disadvantage is that the client can only be used as a desktop system, and the server deployment requires a public IP address. At the same time, the speed of client synchronization is limited by the connection speed between the client and the server (the speed problem can be solved by deploying a CDN for the server).
3. Backup and synchronization solution based on third-party storage

Represented by rclone and duplicati, this type of solution stores the files that need to be backed up or synchronized on a third-party storage, and then deploys applications on each node to achieve the purpose of backing up or synchronizing files by accessing the third-party storage: for example, rclone supports more than 40 cloud storages, from ordinary network disks (OneDrive, Google Drive, Dropbox, etc.) to cloud object storage (Amazon S3, Alibaba Cloud OSS, etc.) to various transmission protocols (FTP, SFTP, WebDav).

Among these three types of solutions, the third one has the widest adaptability. Take rclone as an example: it can support cross-WAN, desktop system and cli interface at the same time. However, the trouble is that you need to use a third-party storage as a transfer (or you can build your own FTP, SFTP, WebDav if you have a public IP), and the main purpose of this type of solution is to back up local data to cloud storage or pull data from cloud storage back to local. Its main purpose is actually backup and recovery. Although cloud disk data can be mounted locally through methods such as alist (rclone itself supports mounting), this is still different from the traditional folder data synchronization requirements.

What if I just want to simply synchronize a few directories between different systems (macos, linux, win) across a wide area network at a fixed time, and don't want to bother with other things, is there any way?

Yes, there is, and that is the professional synchronization tool based on the network itself that this article will introduce: Syncthing.

Syncthing implementation principle

Syncthing is actually just an executable program that does not need to be installed. It does not require the server to be deployed in advance like nextcloud, nor does it require the use of third-party storage like rclone. You only need to run Syncthing directly on each device that needs to synchronize data, and then simply set it up.

As both are executable programs, how is SyncThing more applicable than the previously mentioned Sync Folders? This depends on whether the network environment to be synchronized is a LAN or WAN, and whether the user end is a desktop or a CLI end.
1. LAN environment
1.1. Pure desktop
In this case, to be honest, if the nodes that need to synchronize folders are all desktops, I think sync folders seems to be more useful. In this case, syncthing does not have much advantage.
1.2. Mixed desktop and CLI
In this case, you can use syncthing: you can deploy syncthing on all ends; you can also deploy sync folders on the desktop and syncthing on the CLI, and finally deploy sync folders and syncthing on one end at the same time, then you can connect them all. However, this method requires attention to the direction of synchronization: for example, you can set syncthing on a CLI end to send only, and syncthing on other CLI ends to receive only, and then synchronize the content to all desktop ends by deploying sync folders and syncthing at the same time (sync folders can also achieve the effect of only sending or only receiving by setting the synchronization type).
2. WAN environment
In this case, whether it is the desktop or the CLI, just deploy syncthing directly.

Why can Syncthing directly synchronize across the WAN without the help of a third party? This actually depends on Syncthing's discovery server and relay server.

The interface of "Settings"-"Connection" of Syncthing is as follows:

After enabling the global discovery function, the STUN service can be provided to help users perform NAT penetration across the WAN.

The two options "Global Discovery" and "Enable Relay" mentioned in the above figure include two important functions: SyncThing's connection notification service and relay service.
1. Connection notification service:
The function of the connection notification service is that when one syncthing endpoint actively adds another syncthing endpoint, the other endpoint will receive a notification from the connection notification service and ask whether to accept it. If this endpoint chooses to accept, the two endpoints will successfully add each other as remote devices.
2. Relay service:
The relay service actually consists of two independent parts: the stun service and the turn service. The stun service will receive information (network card IP, listening port, etc.) from both parties. Of course, this information cannot be used directly (it may be intranet information). The stun service needs to combine the real source address and port of the syncthing data packet it receives to determine the NAT type (NAT1, 2, 3, 4) in front of the two endpoints and whether a hole can be made. If a hole can be made successfully, both parties can directly connect point-to-point based on the information provided by stun, and the speed of synchronizing files will be faster; if a hole cannot be made, it can only be handled by the turn server, and the speed of synchronizing files will be slower, depending on the bandwidth allocated to each user by the relay server (the specific speed must be divided by 2) and the number of people using it.

Because Syncthing has deployed public discovery servers and relay servers on the Internet, the Syncthing we deployed can be used directly on the wide area network. However, the usage experience depends on our own NAT type, the communication effect between the Syncthing end and the discovery server and relay server we deployed, etc. (The default discovery server and relay server are all foreign, and the domestic usage experience is definitely not much better~).

By the way, you can actually see the stun related settings in the advanced settings of syncthing. The Stun Server can also be directly set to the stun server you built yourself:

This is because NAT penetration is actually based on the stun service (the principle of stun and how to build your own stun server can be found in my other article:Docker series Use Docker to build your own stun/turn server based on coturn), so if you want to achieve the best results and you are not afraid of tossing, you can use your own stun server, your own discovery server, and your own relay server. I will talk about this separately in the following chapters.

Deploy syncthing

Use the syncthing executable program directly

The download link on github is:https://github.com/syncthing/syncthing/releases

In fact, to be honest, syncthing is just an executable file on Linux. After downloading it, give it execution permissions and run it directly. You don't have to bother with docker, and you won't encounter any permission issues. You just need to pay attention that it is best to use a non-root general permission account to run it.

I still choose Docker for deployment, just because it is convenient to manage and remember everything on Docker. After all, I am getting older, and I am afraid that I will not be able to remember how many scripts I have deployed in the future. . .

Use Docker to build syncthing

Create a directory on the host that needs to be mapped to the container

mkdir -p /docker/syncthing/config # is used to store the syncthing configuration file mkdir -p /docker/syncthing/data # is used to store the data that needs to be synchronized

Building Syncthing

The docker run command format is as follows:

docker run --name syncthing -d --restart=always \ --hostname=syncthing #Optional\ -e PUID=1000 -e PGID=1000 \ #Optional. If docker accesses the mounted directory and there is a permission problem, you can use the id username command in the host to get the PUID and PGID -e TZ=Asia/Shanghai \ #Specify the time zone-p 8384:8384 \ #Manage the port of the gui. Please specify it according to your actual environment-p 22000:22000/tcp \ #Specify the listening port of tcp on the host. Please modify it according to your actual situation. It is best not to change it. -p 22000:22000/udp \ #Specify the listening port of udp on the host. Please modify it according to your actual environment. It is best not to change it. -p 21027:21027/udp \ # discovers the port used by the protocol. Please modify it according to your actual environment. It is best not to change it. -v /docker/syncthing/config:/config \ # mounts the specified directory on the host to a specific path inside the container. This is the directory where the configuration file is located. -v /docker/syncthing/data:/data1 \ # mounts the specified directory on the host to a specific path inside the container. This is mainly the directory where the data to be synchronized is located. You can specify multiple directories, such as data2, data3, etc. You can also directly map the original directories on the host that need to be synchronized. However, you need to pay attention to permission issues, such as changing 755 to 777, otherwise it may not be synchronized. Don't ask me how I know. lscr.io/linuxserver/syncthing:latest

Note: Please delete # and the following comments.

In fact, if it is mainly used in the LAN and needs to use LAN active discovery, and the ports 8384 tcp, 22000 tcp/udp, and 21027 udp are not used, you can directly use the host type deployment (note: this deployment method is valid under Linux, but invalid under MacOS). The docker run format command is as follows:

docker run --name syncthing -d --restart=always --net=host \ --hostname=syncthing \ -e TZ=Asia/Shanghai \ -v /docker/syncthing/config:/config \ -v /docker/ syncthing/data:/data1 \ lscr.io/linuxserver/syncthing:latest

Create three syncthing nodes in docker mode: syncthing-macmini, syncthing-intermini, and syncthing-tc. syncthing-macmini is deployed on my m1 macmini 8g beggar version, using-pDeployed by mapping ports (docker does not support host deployment on macos); syncthing-intermini is deployed on my high-end mini host with 13th generation i5 cpu, located in the same LAN segment as syncthing-macmin, and deployed by host (for later experiments); syncthing-tc is deployed on Tencent Cloud lightweight server, and is deployed by-pDeploy by mapping the port (because it is useless even if deployed in host mode).

Configure syncthing

initialization

usehttp://host ip:8384Visit syncthing-macmini. The following question will pop up when you first visit. You can choose yes or no:

After entering the interface, you first need to create an administrator username and password:

After the settings are completed, the login authentication page will pop up. Log in using the username and password you just created:

Then enter the formal configuration page:

Similarly, initialize syncthing-intermini and syncthing-tc.

Note: When initializing all nodes, you don’t need to add the synchronization folder first. You will set it up later.

Adding a Remote Device

Start adding the other two nodes on the syncthing-macmini side.

Add syncthing-intermini node

Simply click "Add Remote Device" in the red box in the remote device section:

Because syncthing-intermini is deployed in host mode, it can be seen directly from nearby devices in the figure below:

If you do not use host deployment, you cannot see it in nearby devices, so you need to copy the device ID on syncthing-intermini separately:

Then log in to the syncthing-intermini node, and you can see the add request from syncthing-macmini:

After clicking Add Device, the detailed information of syncthing-macmini will appear. At this time, just click Save in the lower right corner:

Then you can see that the remote device of syncthing-intermini already has syncthing-macmini:

Then we can see that the remote device of syncthing-intermini has been successfully added to syncthing-macmini:

The reason why there are so many addresses in the above picture is that when we added intermini on macmini, we selected the default dynamic address list. Therefore, the default stun server and relay server on the public network are used to determine which connection methods are available, resulting in a large number of options below, some of which are available and some are not. In fact, for devices in the same LAN, it is not necessary to be so complicated, so we can directly modify the address list of syncthing-intermini in the remote device on syncthing-macmini, and change the default dynamic totcp://intermini LAN IP:22000, as shown below:

Then you can see that the address of syncthing-intermini has changed in the remote device:

At the same time, in syncthing-intermini, the address of the remote device syncthing-macmini also changed:

Add syncthing-tc node

Similarly, add the syncthing-tc node, but because this is a node on the Tencent Cloud lightweight server, if you need to use a public IP connection, you need to open ports 22000 tcp/udp, 8384 tcp, and 21027 udp in the console firewall in advance:

So far, two other nodes have been successfully added to syncthing-macmini:

Add Sync Folder

Add the synchronization folder on the primary node first

Assuming that the content we need to synchronize is located in the /data1 directory in the syncthing-macmini container, we need to first add it on the master node syncthing-macmini:

If you checked Add Ignore Mode, the following interface will appear:

Then the sync folder is added successfully:

Note: syncthing will create a new directory named ".stfolder" in the added synchronization folder. If this directory cannot be added due to permission issues, a prompt will be displayed on the web page saying that the folder cannot be created. When you see this error, you should know that there is a permission issue: If it is a docker environment, you should consider using-e PUID=<number of columns> -e PGID=<number of columns>The parameters specify reasonable real user permissions of the host machine, or change the directory permissions mapped from the host machine to the container from 755 to 777; if you run the syncthing execution file directly, you can directly use the method of modifying the directory permissions.

Share the sync folder on the master site to other nodes

Because I have already set up version control, ignore mode, advanced and other options when creating the folder, I just need to select two remote devices in the share and save it directly, as shown below:

Then the other two nodes will recognize the shared folder request received from syncthing-macmini:

After clicking "Add" in the red box, you will enter the option of adding a synchronization folder:

Similarly, complete the same steps on the syncthing-tc node. After completion, you can see the synchronization status on the syncthing-macmini node:

Note: The synchronization speed depends on whether the optimal path can be selected between syncthing nodes. If the optimal path is not selected, it may happen that the nodes are in the same LAN and the transmission is fastest through the intranet, but the nodes go through the slowest relay server. This is because syncthing misjudges each other's location. This situation is very common in docker deployment. If it is a linux system and deployed in host mode, the chance of correct identification in the same intranet segment is very high. But it doesn't matter. Just follow the method we mentioned earlier and manually specify the correct address in the address list of each node's remote device.

Advanced Use 1: Self-built discovery server and relay server

Theoretically, as long as we deploy syncthing in an Internet-connected environment, we can generally connect to the public syncthing discovery server and relay server on the Internet. However, there are some special environments, such as a pure intranet environment, or operator network restrictions that make it impossible to connect to the public discovery server and relay server. At this time, if we have the conditions (home broadband with a public IPv4, a cloud host with a fixed public IP, or a pure intranet environment with a controllable deployment environment), we can build our own discovery server and relay server. I'm not going to say more here, I'll just paste the docker command, and you can study it yourself if you are interested.
Discover the server docker run format command:

docker run --name syncthing_discovery_server -d --restart=always \ -e PUID=1000 -e PGID=1000 \ # Same as before, also optional -p 8443:8443 \ -v /docker/syncthing/discosrv:/var/stdiscosrv \ syncthing/discosrv:latest

Relay server docker run command format:

docker run --name syncthing_relay_server -d --restart=always \ -e PUID=1000 -e PGID=1000 \ -e pools="" \ # This is very important. If you don't add this, the relay server you build will be published directly to the list of public relay servers on the Internet by default -p 22067:22067 \ -v /docker/strelaysrv:/var/strelaysrv \ syncthing/relaysrv:latest

The above discovery server and relay server commands and images are officially provided and are divided into two independent dockers. Some people on the Internet have combined these two into one docker. The commands are as follows:

docker create \ --name=syncthing-relay-discosrv \ -p 22067:22067 \ -p 22070:22070 \ -p 8443:8443 \ --restart unless-stopped \ johngong/syncthing-relay-discosrv:latest

I haven't tried it, but if you are interested, you can try it.

The device IDs of the self-built discovery server and relay server can be found in the corresponding docker logs, usingdocker log container nameFor example, the device ID of the discovery server I built is:

Device ID of the relay server:

In the "Settings"-"Connection" interface of Syncthing, change the original default to the corresponding discovery server and relay server addresses, or keep the default and write the relay server and discovery server addresses as "default,relay://" and "default,https://" respectively:

Note: If it is a pure intranet environment, even if it is not a network segment, you only need to establish a discovery server, and you don't need a relay server. However, you may need to manually add it in "Remote Device" - "Advanced" - "Address List" using "tcp://other party ip:22000".

Advanced use 2: Directly use with virtual networking

Taking the virtual networking implemented by tailscale as an example, directly use the address of tailscale in the address list of the remote device syncthing-tc of syncthing-macmini:

The advantage of this method is that the firewall ports on Tencent Cloud servers do not need to be opened, because TailScale directly enters the device kernel through Wireguard VPN, not the traditional TCP protocol. If all nodes are deployed with TailScale, they can directly add each other using the TailScale address of the other party, without the need for a relay server.

Afterword

If we only talk about synchronization, Syncthing is indeed the most convenient: there is no need to deploy a server, no need to rely on third-party storage, and it supports both GUI and CLI. It is also very simple to use in practice.

I looked at the various problems encountered by netizens, either permission issues, or misunderstanding of the working mechanism of the discovery server and relay server, and the slow speed caused by not optimizing the configuration (manually modifying the peer IP in the address list). In fact, if the correct configuration is performed, Syncthing is still very easy to use (like me, I use the intranet IP for intranet communication and the tailscale IP for WAN communication).

I now have three nodes: syncthing-macmini is actually the main site of wordpress, syncthing-intermini is the hot standby site of the main site of wordpress in the intranet, and syncthing-tc is the mirror site of the main site deployed on the Tencent cloud host. The effect achieved now is that the main site is set up with automatic backup once a week to the local directory (/docker/wordpress/html/wp-content/wpvividbackups):

After Syncthing detects that the WPvivid of the main site has completed the regular backup, it will automatically synchronize to the same path of the WPvivid plug-in of the hot backup site and the mirror site. Then you only need to click the scan button in the red box below in the WPvivid plug-in of the hot backup site and the mirror site:

The backup synchronized from the master site will appear:

Then just click Restore to update to the content of the most recent backup of the main site. This is definitely more technically advanced than manually uploading a backup and then restoring it, although the time difference is not much.

There is no way to solve this problem, because when WPvivid schedules tasks to back up to a remote site, you can only choose one site (I currently have a hot backup site and a mirror site, and there may be more in the future), and it can only be done by FTP or SFTP. The most annoying thing is that when scanning and restoring on the remote site, you are required to enter the FTP or SFTP information (IP, port, username and password), which is so annoying, and I often forget it... Now with syncthing, this pain point has finally been solved, and I feel relieved.