Attack Post-Mortem

When I got up at 5 AM, I happened to glance at my emails:

Holy crap, targeting me again? And last time was at 02:16, this time at 02:21. So around 2:20 AM is their working hours? Let me quickly check the traffic report:

Let's look at the duration:

It completely ended at 2:21:37. The packet content was similar to last time, so I won't post it. Simply put, it was a crazy GET flood on my blog homepage based on HTTP/2 requests. This attack lasted for 5 minutes (16 minutes last time), with a total of 6.49 million requests (10 million last time) and a request bandwidth of 23.7G (35G last time). As you can see, this attack achieved 2/3 of the previous effect in 1/3 of the time. What does this mean? It means my response capacity this time was twice that of last time. Why is that? Because I changed my optimization method. Remember the notification about the Worker quota in the email during the last attack?

There was no such warning this time, because last time I felt that using a self-built Workers-KV approach to optimize access was too fragile. Although under normal circumstances a quota of 100,000 requests per day is more than enough, once some bored idiot launches this kind of low-tech DDoS attack, it gets ugly. So, I decided to try the official version of the self-built Workers-KV approach: APO for WordPress. Actually, its implementation principle is similar to self-built Workers-KV—both cache WordPress content as static HTML in edge network caches, and both rely on Workers-KV. However, this official method doesn't seem to have a limit on the number of Worker executions, and it integrates extremely well with the caching functionality.

But, although the response capacity doubled, this time I discovered an issue I hadn't noticed last time. It's like the famous quote from Fist of the North Star: "You may look alive, but you are already dead!" Why do I say this? Because I found that although the website could be opened, the content looked like this:

While my normal site looks like this:

Why did all the content change?? Then it hit me: my main WordPress site must have died, and the load balancer directly routed the traffic to the backup WordPress site, which I hadn't had time to sync data to yet... Then I checked the Uptime monitoring, and sure enough:

This was because I had just modified the Uptime monitoring page address to point tohttps://blog.tangwudi.com/methis link, which I had just created on the main WordPress site a few days ago. The backup site didn't have this link. So, after the load balancer switched traffic to the backup site, even though the site was running normally, the monitoring showed it as down. (As for why there were a few green spots in between, I guess it's because the main site was mostly dead but still had a bit of life left. It occasionally revived to respond to a few requests, so the load balancer would occasionally route a few requests back. Since the monitoring page existed at those moments, it showed green, but then it died again immediately, the load balancer switched back to the backup site, and the monitoring page disappeared again~). This can also be seen from the statistics of Chaitin SafeLine at that time:

The 61.11% of 404 errors were caused when the main site died before the traffic was switched to the backup site (there is a timeout wait time before nginxWebUI switches traffic to the backup site). This can also be indirectly confirmed by one of my emails:

Why does this email prove the above speculation? Because WordPress uses PHP, and due to PHP's working mechanism, scheduled tasks and similar operations can only be executed when triggered by access requests, which includes WordPress's automatic updates. Normally, when the main site is running fine, the backup site receives no access requests, so even if WordPress's automatic updates are configured, it is useless. However, at 2:21, the main site was knocked down (the attack traffic also ended at that time; did it end because it detected that my site was unresponsive? Quite resource-saving indeed). After the timeout period, the blog traffic was switched to the backup site. Once there were requests, they triggered the automatic update mechanism of the backup site's WordPress, and then the upgrade was completed at 2:23. PS: My main site had already automatically completed its version upgrade the day before yesterday:

And the direct evidence is that when accessing the main site using the intranet address, it was basically inaccessible, but returned to normal after restarting the main site's Docker container.

Looking back now, I happened to have synchronized the data to the backup site just before the last attack, so the content of the main and backup sites looked exactly the same, and I didn't notice this issue. Later, due to my mental OCD, I restarted the main site's Docker again, which brought the dead main site back to life to take over the traffic again, making it even harder to detect.

Alright, the analysis is mostly clear. Based on the information at hand, let's summarize the pros and cons of the current blog service architecture.
Advantages

• APO is indeed awesome; the response speed for normal access and the ability to handle attacks have both been significantly improved.
• Cloudflare's architecture based on Anycast + unlimited caching worked wonders again when facing a DDoS attack. In the end, only 0.16% of the attack requests and 1.6% of the attack bandwidth actually entered the home data center.
• The Nginx-based load balancing is working normally, achieving basic failover.
• The active-standby redundancy architecture of WordPress within the same data center still played its part.
  Disadvantages
  • The current performance bottleneck is WordPress. The attack traffic entering the home data center via Cloudflare Tunnel posed almost no pressure on Nginx (SafeLine WAF and nginxWebUI load balancing), but instead, the main WordPress site itself was knocked down.
  • The load balancing provided by nginxWebUI has no issues in terms of performance (currently using the stream-based Layer 4 TCP proxy mode), but its features are simply too basic. The default health check (which barely qualifies as one) seems to rely on whether a TCP port connection can be established, which is a bit crude. Moreover, the lack of statistical charts is very inconvenient (of course, this is also because I used all default values, but there is really no motivation to research or tinker with such a simplified load balancing feature).
  • The data synchronization frequency between the active and standby WordPress sites in the home data center is an issue, but for now, once a week seems about right, and missing 1 or 2 articles is acceptable.

Process Optimization

The purpose of process optimization is to address points 1 and 2 of the aforementioned disadvantages.

Point 1:

The so-called performance bottleneck of WordPress only appears when it is attacked by abnormal traffic. Under normal circumstances, after APO caching, the requests that need to go back to the origin are already much fewer. Plus, my tiny site has barely any traffic to begin with—rounded off, it basically means there is almost no traffic at all normally. Therefore, we only need to solve the "abnormal traffic" issue during attacks. So, where does this abnormality mainly lie? Since I configured a single-IP rate limit for the entire site in Cloudflare:

And also configured a single-IP high-frequency access limit on SafeLine WAF:

Therefore, the main cause of the abnormality is not multiple visits from a single IP, but rather the number of concurrent connections. As long as concurrent connections are limited, the maximum number of requests reaching WordPress at the same time is fixed. Then, no matter how many abnormal requests there are, it makes no difference to WordPress. SafeLine WAF Community Edition does not have this feature, and I don't want to modify SafeLine's underlying Nginx. Although nginxWebUI can be modified, I want to directly replace it with a dedicated open-source load balancer. So after much thought, I decided to deploy another Nginx (my aaPanel is back in action again) before SafeLine WAF, specifically to limit concurrent connections for WordPress:

Then point it to SafeLine WAF via aaPanel's reverse proxy.

Point 2:

Given that nginxWebUI's features are indeed too simple, I will officially enable Zevenet as a replacement and complete the cutover. The steps are as follows:

Similarly, add the backup site. The only difference is that the Priority needs to be set. The final result is as follows:

Next, you only need to change the IP and port on the Chaitin SafeLine WAF that originally pointed to the nginxWebUI load balancer to point to the VIP and port on Zevenet:

Then, finally, you can get rid of the awkwardness of nginx load balancing where nothing is visible by default:

Summary

After the above process optimization, the request route for accessing the blog inside the home data center via Cloudflare Tunnel has now become:
Cloudflare Tunnel—>BT Panel Nginx—>SafeLine WAF—>Zevenet Load Balancer—>WordPress Main Site.

By controlling the overall concurrent connections entering the blog through the BT Panel's Nginx, the first problem was solved; and by using the community edition of the professional load balancer Zevenet (Zevenet is quite famous among open-source load balancers) to replace the previous Nginx, it brought stronger scalability, and O&M through the GUI interface has also become much more convenient. Although it has fewer features compared to the commercial version, it is more than enough for an average person's environment, which also solved the second problem.

The optimization of the blog's internal access flow is complete. Next, I look forward to the arrival of the next attack to see if it can take down my main site again (actually, it doesn't even need to be taken down; as long as it forces me to switch to the backup site, I lose~).