Project Background Introduction
In the past, only 2 or 3 friends watched my home Emby, so I didn't think too much about it. I just used an unregistered domain name and directly did port mapping on iKuai, mapping it straight to the intranet HTTP port of Emby installed on the NAS, and that was it. Originally, this was no big deal, but ever since that incident with Telecom (see previous article for details:Home Data Center Series: Precautions for Building a Website with Home Broadband Having a Public IP), I felt that using an unregistered domain name plus plaintext HTTP was too dangerous, so I had the idea of optimizing the Emby publishing method. Then, considering how precious public IPv4 addresses are nowadays, I had to guard against Telecom forcibly reclaiming public IPv4 addresses under the banner of complying with the national "cyber powerhouse" policy. So, I decided to optimize the overall architecture of my home data center to prepare in advance for future website building with only public IPv6 addresses. Finally, I thought, since the project scale has grown so large, I might as well take the opportunity to sort out the traffic access logic of the home data center. As a result, this project was born: Home Data Center v4/v6 Dual-Stack Network Architecture and Application Access Flow Optimization.
I originally just wanted to write a casual post, but ended up making the project so big by accident?? Caught completely off guard, is this "a tragedy triggered by an Emby"?
Original Environment Analysis
The traffic logic of the original environment was quite chaotic, as shown in the figure below:
Tencent Cloud CDN origin-pull requests reached iKuai via Telecom line 1, Cloudflare Tunnel access requests reached iKuai via Telecom line 3, and Emby HTTP requests reached iKuai via Telecom line 2. The key issue was that the access traffic entering the home data center through the iKuai soft router had no unified traffic processing logic.
As can be seen from the figure above, there are 3 paths for access traffic to reach intranet applications:
Directly map ports on iKuai to the application (represented by the Emby application), still running plaintext HTTP (Telecom saw it but just kept quiet).
iKuai port mapping to the designated port (HTTPS) of the v4 gateway (which is the Pagoda Linux Panel), and then the v4 gateway performs SSL offloading and points directly to the application via reverse proxy.
iKuai port mapping to WAF, and then WAF performs security checks on access requests and points to the application via reverse proxy. The applications here also include the v4 gateway. Why? Because for some applications, I previously deployed them directly via source code on the Pagoda Panel for convenience...
In summary, the traffic logic of the entire network was actually a mess... Of course, this is mainly a legacy issue, and you can't just say I was lazy~.
In addition to the messy traffic logic, I didn't utilize IPv6 at all, wasting the v4/v6 dual-stack environment. At the same time, there were no adequate security plans. For example, what to do if hit by a DDoS attack? How to divert attack traffic to traffic scrubbing (Cloudflare)? In short, there were many problems.
Logical Topology After Optimization and Reconstruction
After some effort, the optimized and reconstructed logical network architecture is shown in the figure below:
Inbound Access Traffic Line Allocation
Tencent Cloud CDN origin-pull traffic enters from Telecom line 1
This goal is achieved by using iKuai's "Advanced Applications" - "Dynamic Domain Name" function to update the public IPv4 address of Telecom port 1 to Domain B in real-time:
Point the origin-pull address of the corresponding accelerated domain name on Tencent Cloud CDN to Domain B:
Cloudflare Tunnel access traffic enters from Telecom line 2
This goal is achieved by using iKuai's "Flow Control" - "Port Shunting" function to specify that the device deploying Cloudflare Tunnel goes out from Telecom line 2, so Cloudflare will connect using the public IPv4 address of Telecom port 2:
Alibaba Cloud CDN IPv6 origin-pull traffic enters from Telecom line 3
This goal is achieved by using iKuai's "Network Settings" - "IPv6" - "IPv6 Settings" page, which supports enabling IPv6 lines in a multi-line environment, thereby enabling IPv6 on Telecom line 3. This way, IPv6 traffic can only enter from Telecom line 3:
Since IPv6 uses public routing, the original IPv4 port mapping function is no longer effective, and iKuai does not yet support IPv6 -> IPv4 address port forwarding. Therefore, IPv6 origin-pull traffic reaching the v4/v6 gateway needs to be implemented in cooperation with iKuai's dynamic domain name function (it can also be run by the v4/v6 gateway itself, which is more accurate; iKuai's acquisition of other devices' IPv6 addresses depends on the situation—if it is a stateful IPv6 address, the accuracy is fine, but if it is stateless, it can only be obtained through the MAC address of other devices, which sometimes fails).
From a security perspective, you can enable the IPv6 firewall function on iKuai, specifying that only access requests matching the IPv6 address suffix of the v4/v6 gateway and specific destination ports (such as reverse proxy ports, TCP, UDP port forwarding ports) can pass through, while requests accessing other internal public IPv6 addresses are blocked. This can greatly improve the security of intranet IPv6 addresses. This goal is achieved through iKuai's "Security Settings" - "ACL Rules" function:
Other Inbound Traffic Line Allocation
The Emby access entry is now allocated to Telecom line 3, accessed using a registered domain name, while the data entry for Transmission (used for PT) is allocated to Telecom line 1. The reason for this allocation is that the domain name accessed by Emby will expose the public IP of my Telecom line 3. If subjected to a large-scale DDoS attack, I can directly disconnect Telecom line 3, since Telecom line 3 is mainly used for Emby and as an IPv6 backup line anyway. Telecom line 1 was originally used as the origin-pull line for Tencent Cloud and usually doesn't have much traffic, so I might as well use Transmission to put it to good use.
Cloudflare inbound traffic accessing my .com domain goes through Telecom line 2, which is also my internet access line. Disconnecting Telecom lines 1 and 3 will not affect access to my .com domain, nor will it have any impact on my normal internet access.
v4/v6 gateway
Optimize the v4/v6 gateway (the original Pagoda Linux Panel):
Migrate source-deployed applications
Migrate all source-deployed applications to Nginx in the Docker area (for deploying Nginx+PHP via Docker, see:Docker Series: Single-container Nginx, Single-container PHP (one version) Multi-site Sharing)。
Enable IPv6 support for all sites
The Nginx on Pagoda Linux Panel has the "--with-ipv6" module added by default during installation, so you only need to add the following code to the corresponding site configuration:
listen [::]:443 ssl http2; #具体端口大家根据自己实际情况修改，比如本例中是55555As shown in the figure below:
If Nginx is installed separately, you can enter the Nginx file directory and use./nginx -V to view the installed modules. If there is--with-ipv6, it means this module is installed; otherwise, you need to recompile and install it.
Enable SSL encryption for all sites
To cope with an environment that will only become increasingly severe in the future, all sites must enable SSL encryption (including Emby). For certificates, just use the Let's Encrypt certificate built into the Pagoda Panel. If the number of hosts under the main domain exceeds 20, you can use OHTTPS (see article:Home Data Center Series: One-Stop SSL Certificate Management Tool OHTTPS Tutorial), the v4/v6 gateway provides a unique site listening port to the outside (e.g., TCP 55555, and all sites must add this listening port), which is used for port mapping of the Telecom public IPv4 address on the iKuai soft router and the port corresponding to the Alibaba Cloud CDN IPv6 origin-pull address.
All SSL decryption must be completed on the v4/v6 gateway. In theory, only HTTP runs on the intranet, except for a few special applications (such as DoH).
Alibaba Cloud CDN IPv6 origin address
Actually, why did I introduce Alibaba Cloud CDN when I already have Tencent Cloud CDN? Because Tencent Cloud CDN currently only supports IPv4 addresses for origin-pull:
This is a real bummer, whereas Alibaba Cloud CDN supports IPv6 origin-pull:
So if public IPv4 addresses really become unavailable in the future and Tencent Cloud still doesn't support IPv6 origin-pull by then, I will be forced to fully switch to Alibaba Cloud CDN.
Alibaba Cloud's origin address needs to point directly to the dynamic domain name corresponding to the public IPv6 address of the v4/v6 gateway. This also needs to be implemented through iKuai's "Advanced Applications" - "Dynamic Domain Name" feature:
iKuai's IPv6 dynamic domain name resolution (DDNS) can obtain the IPv6 address of an intranet host based on either its MAC address or DUID, and update this IPv6 address to Domain A, which is used for Alibaba Cloud CDN origin-pull. Therefore, you just need to enter Domain A in the origin address field of Alibaba Cloud CDN, and point the port to the listening address 55555 of the v4/v6 gateway:
Note 1: Actually, you can also add the Alibaba Cloud address as a hot standby origin in Tencent Cloud CDN. This way, if the public IPv4 address is suddenly reclaimed one day, it won't affect the access to my domestic domain names.
Note 2: For the configuration of IPv6 and IPv6 dynamic domain names on the iKuai soft router, please refer to:iKuai Soft Router Series: iKuai IPv6 Feature Configuration Tutorial。
iKuai soft router port mapping
Delete all port mappings of the iKuai soft router's public IPv4 address that point directly to the inside of the home data center, and create only a single port mapping pointing to the site listening port of the v4/v6 gateway, which in this example points to TCP port 55555.
Note: The port 55555 pointed to by Tencent Cloud CDN origin-pull is actually the port mapped on Telecom Line 1 of the iKuai soft router. In fact, the port on the v4/v6 gateway does not have to be 55555. However, the TCP port 55555 pointed to by Alibaba Cloud CDN origin-pull is indeed the port on the v4/v6 gateway. For the sake of uniformity, we might as well set both the IPv4 and IPv6 listening ports on the v4/v6 gateway to 55555.
Deploy v6 -> v4 port forwarding on the v4/v6 gateway
For non-HTTP applications (such as the World of Warcraft server side), deployment via reverse proxy is not possible. Instead, port forwarding must be used to forward directly from a specific port of the public IPv6 address to a specific port of the intranet private IPv4 address. This requires deploying Lucky to achieve (for detailed Lucky deployment and configuration, see:Docker Series: Using Docker to Build a Simplified Public IPv6 Inbound Access Gateway Based on Lucky)。
Next-hop destination of the v4/v6 gateway
All inbound access requests, after undergoing SSL decryption at the v4/v6 gateway, need to be directed to the next-hop WAF device via reverse proxy for security inspection (for the reverse proxy configuration steps on the Baota Linux panel, see:Linux Panel Series: Configuring Reverse Proxy and Publishing Using Non-443 Ports）。
Although in principle all inbound requests received by the v4/v6 gateway should be directed to the downstream WAF for inspection via reverse proxy, some special applications need to bypass the WAF (such as Emby traffic). This is because the traffic returned from the Emby server is too large; if it all goes through the WAF, the WAF probably won't be able to handle it. Actually, the best way would be the DR (Direct Routing) mode, where the traffic returned from the Emby server directly bypasses the WAF and returns to the v4/v6 gateway. However, going through so much trouble just for one application is not very cost-effective in my opinion, so I'll just have to live with it for now.
Therefore, as a final compromise, Emby access requests bypass the WAF via reverse proxy directly from the v4/v6 gateway and point straight to the Emby server.
Hardening of the v4/v6 gateway itself
Actually, Nginx itself also needed to be hardened. However, on one hand, since it goes through the CDN, the origin server's IP address is not exposed; on the other hand, the Nginx on the v4/v6 gateway only handles SSL decryption and reverse proxy (applications deployed via source code have already been migrated to the Nginx in the Docker area), so there is not much need to bother. The only thing that can be done is to configure a fake default site to deal with IP scanning and avoid leaking the site's domain name. There are many tutorials on the internet for this, so I won't go into detail here—just remember to use a fake SSL certificate.
WAF
Since Cloudflare comes with built-in traffic scrubbing and WAF functions, I am quite comfortable with access requests entering the intranet through the tunnel, and they can be pointed directly to the applications in the Docker area. However, for requests coming in from domestic CDNs, we must be very careful, which makes the WAF extremely important: all access to internal applications on the v4/v6 gateway, except for Emby, must first point to the WAF and undergo WAF inspection.
The specific configuration varies depending on the WAF. I am using the community free version of Chaitin SafeLine, mainly for semantic analysis and high-frequency access/attack restriction:
For traffic coming from the .com domain, especially for highly important applications (such as blogs), I also route it through the WAF to compare the traffic volume with that from domestic domains and to provide the final line of defense:
(Optional) NGINX Load Balancing
Since I have Docker Zone 1 (Mac mini) and Docker Zone 2 (Intel mini host), both of which have Nginx deployed, I have a need for load balancing and high application availability. As for whether to use Nginx, HAProxy, or Zevenet CE for load balancing here, it depends on your personal preference.
To be honest, the interface of Zevenet CE is quite decent, and most importantly, the community edition is free. It has all the basic server load balancing features (the large screenshot is a bit blurry, so please bear with it and just get a general idea):
However, I ultimately chose Nginx's stream module to implement Layer 4 load balancing (Layer 7 load balancing is not needed for now) because it feels more lightweight, and I used nginxWebUI to solve the lack of a graphical interface:
Since the current traffic is not very high, I adopted an active-standby deployment to achieve high availability for the blog.
Docker Series: Using Docker to Build a Graphical Nginx Based on nginxWebUI
Security Plan
This security contingency plan is mainly aimed at responding to high-traffic DDoS attacks (non-high-traffic attacks should be handled by devices like WAFs). In typical projects, if a user has purchased cloud scrubbing services from a provider (telecom carrier or cloud service provider), once abnormal traffic is detected, the provider will divert the abnormal traffic to traffic scrubbing equipment for mitigation, and then route the clean traffic back to the user.
But as a budget-strapped blogger, I don't have the money for that. So what can I do? I have to find ways to make the most of the resources I currently have. My .com domain is hosted on Cloudflare, so it's safe from DDoS attacks. The traffic that needs a security plan is mainly domestic traffic accessing the .cc domain via Tencent Cloud or Alibaba Cloud CDN. So, how do we set this up?
Utilize certain CDN features to defend against attacks
I configure IP rate limiting for everything; 100 QPS is more than enough for normal applications:
Usage capping: once the set cap is exceeded, the CDN is disabled immediately. You will need to set this based on your website's actual traffic:
Disable TLS 1.0; in fact, you can also consider disabling TLS 1.1:
WAF rate limiting and high-frequency attack restriction
After the initial filtering by the CDN's rate limiting, the WAF can perform another round of filtering while restricting high-frequency attack behaviors.
Cloudflare traffic scrubbing
With a simple configuration, the Cloudflare Free plan can perform traffic scrubbing. The only downside is that the default CDN IPs provided are not optimized, making user access significantly slower than domestic CDNs. However, in critical moments, being slow is better than being completely inaccessible (and you can manually test and configure optimized IPs). Therefore, Cloudflare can be used as a last resort defense when existing security policies fail to stop an attack.
How do we do this? By adding the domestic domain to Cloudflare as a Custom Hostname:
Normally, DNS resolution is split into domestic and overseas: domestic traffic is routed via CNAME to the domestic CDN provider's address, while overseas traffic is routed directly to 1.0.0.5 via an A record:
This way, once a DDoS attack is detected, you can simply pause the CNAME record and change the routing of the A record from "Overseas" to "Default". Any subsequent attack traffic resolving your domain will resolve directly to 1.0.0.5, hitting Cloudflare's Anycast IP and entering Cloudflare's traffic scrubbing pipeline.
However, this method is ineffective against ongoing attacks because the attackers have already resolved your domain and are targeting your current IP directly. For me, there is an additional step: disconnect Telecom Line 1 and redial. Thanks to the dynamic IP mechanism, Telecom Line 1 will get a new IPv4 address, and the subsequent attacks will no longer reach me.
Compared to purchasing expensive cloud scrubbing services or buying dedicated DDoS mitigation hardware, this method has a slower response time and requires manual intervention, but its biggest advantage is that it is free. Of course, this setup is only suitable for personal websites; if it is a commercial site, do not try to save on these minor expenses.
Afterword
Through this article, I find that an old saying really rings true: "A good memory is not as good as a bad pen." The knowledge mentioned above is commonly used in daily life, and I didn't think much of it, but when I write them down and need to write some summary conclusions, I still find some details that I usually didn't notice or didn't think deeply about. Sure enough, keeping a blog is indeed very beneficial.
P.S.: On April 17, I woke up to find that multi-dialing was gone. Although I knew this day would come sooner or later, I didn't expect it to be so soon. At this rate, public IPv4 addresses might also be gone soon. Although it doesn't affect me much, after all these years, it's still a bit sad.