A few days ago, seeing that the Tailscale versions running on my Tailscale devices were all over the place, driven by my OCD, I did a mass upgrade for all devices running Tailscale (except for the QNAP NAS and OpenWrt, because the latest official installation packages for these two are currently only 1.58) to version 1.78.1:

Unexpectedly, among so many upgraded devices (including macOS, iPhone, iPad, and all Linux hosts), macOS was the only one that had an issue: a normal upgrade only updates the Tailscale version and does not affect the corresponding fixed private IP address starting with 100. However, in this upgrade, all three macOS devices (including the base-model M1 Mac mini running the main blog site) were affected, and their fixed private IP addresses all changed. As a result, in the ”Machines” interface on the official Tailscale website, two records could be seen for each macOS device (one old record showing offline, and one new record showing online).

At that time, I simply deleted the old records and didn't think too much about it. However, I forgot a crucial point: the disaster recovery site running on my Tencent Cloud host uses the reachability of the previous fixed private IP address of the main blog site's Mac mini as the key indicator to determine whether the home data center is down (for details, please refer to the article:Home Data Center Series: Flexibly utilizing Cloudflare Tunnel to achieve automatic takeover by disaster recovery site when the main WordPress site fails)。

This led to a very serious consequence. Since I upgraded the Tailscale version of the main blog site's Mac mini, the invalidation of the original fixed private IP address caused the detection script of the Tencent Cloud host's disaster recovery site to constantly assume that the home data center was down. Therefore, the disaster recovery site remained enabled, meaning that during this period, there were two connectors simultaneously in the Cloudflare Tunnel where the blog domain is located, whereas normally there should only be one:

Of course, a single Cloudflare Tunnel can also support multiple connectors. This is actually the multi-origin load balancing feature bundled with Cloudflare Tunnel (for details, see the article:Home Data Center Series Cloudflare Tutorial (9) Introduction to Common Zero Trust Features and Multi-Scenario Usage Tutorial), but there is a major prerequisite: the network structures of the multiple origin sites must be completely identical. Generally speaking, using localhost works fine, which is also the most standard way for Cloudflare Tunnel to support multi-origin load balancing.

However, because my home data center has a complex structure (the host running Cloudflare Tunnel, the host running the intranet WAF, and the host running the main blog site are all independent), and its network planning is definitely different from the IP address of the Tencent Cloud Lighthouse server, some technical ”special handling” was required for the disaster recovery site on the Tencent Cloud host to take over the blog service when the home data center goes down (it can only take over the blog service; nothing else can be done since the performance of the Lighthouse cloud host is very limited).

I won't delve into the specific technical details, but in short, the final result was that accessing the blog became a matter of luck. Some requests were routed to the main blog site in the home data center, while others were routed to the disaster recovery site on Tencent Cloud. Meanwhile, due to the ”special handling,” Cloudflare's detection of the origin site would intermittently fail, frequently resulting in 502 errors indicating that the origin site was unreachable:

Moreover, this error was quite random. When accessing the same article, some friends would encounter it, while others could access it normally, making it seem like a ”ghostly issue.”

Actually, I also felt that blog access was a bit abnormal over the past few days. However, because I had adjusted many intranet policies during this period (including iKuai's multi-line DNS feature) to play around with some setups (using two Apple TVs as backup proxy gateways and a unified Tailscale exit node respectively), I kept thinking it was due to my tinkering. Plus, I could access it normally using cellular data (looking back, I just got lucky when accessing via cellular), so I didn't pay much attention to it.

Speaking of which, I would like to thank two friends in the group (jdejdndns and RadiantHope) for their reminders:

If it weren't for their reminders, I wouldn't have been able to confirm that there was indeed a problem with blog access. Thank you both here!

In addition, this experience fully validates a piece of wisdom in cutover operations: do not make multiple changes during a single cutover, otherwise, if something goes wrong, you won't even know where to start troubleshooting.