1 Why Do We Still Need a “Guess What You Think”?”

Because it overlapped with the newly added ”Right Menu” feature on the article page, after some consideration, I decided to remove the ”Guess What You Think” feature. However, this article is still kept as a reference example for implementing similar features.

In previous articles (see:Building a “Lightweight Knowledge Index” for My Blog (Part 3): Design and Implementation of a Lightweight WordPress Recommendation System Based on Pre-computed Semantic Index), I added a new ”Related Articles” feature below the ”Content Structure Prompt” (it was previously called ”Recommended Reading”, but I later felt that ”Recommended Reading” was too broad, so I changed the name to the more accurate ”Related Articles”).

After running it for a while, I am quite satisfied with this feature—since the recommendation logic is built on explicit structures such as article titles, tags, categories, and body content, in most cases, it can recommend content highly relevant to the current article.

However, there is one issue that has always bothered me: a large blank area is left on the right side of ”Related Articles”, which always makes it look somewhat unfilled:

Since it's empty anyway, I naturally wondered if I could put some other recommended content there. But the problem is that “Related Articles” itself has already done a great job of ”finding similar articles.” If I continue to make recommendations based on explicit structures like categories, tags, series articles, and knowledge maps, the final results will highly likely overlap with ”Related Articles,” or even result in a lot of duplicate content. In other words, if the new feature just uses a similar algorithm to recommend the same articles again, its existence wouldn't make much sense.

Since the ”Related Articles” on the left is responsible for mining explicit associations, the content on the right should perhaps take on another responsibility: trying to discover those potential associations hidden beneath the surface of articles that cannot be directly described by categories and tags. Among the currently mature technical solutions, the most suitable one for this task is precisely the semantic space formed after vectorization (for those unfamiliar with the concept of vectors, you can refer to my other article:Vectors: The Universal Language in the AI World)。

Therefore, I decided to add a brand-new recommendation module on the right—“Guess What You Think.” Unlike ”Related Articles,” which relies on explicit structures, it will be based entirely on vector data generated by article Embeddings, using semantic similarity to find articles that may be related but do not necessarily belong to the same category or tag system.

2 Implementation Ideas for the “Guess What You Think” Feature

1 Why Not Directly Use the Existing article-index.json?

Following the ideas of the previous articles in this series, whether it is “Related Articles” or “Series Article Cards,” they are ultimately based on WordPress article content to generate semantic-index.json, which is then organized to obtain article-index.json, and finally, the WordPress front-end directly reads article-index.json to complete the display.

Intuitively, “Guess What You Think” seems to be able to directly reuse this structure: article-index.json already contains information such as article titles, URLs, and summaries. As long as some fields are added, it could theoretically be used for Embedding calculations as well. However, after actual evaluation, I decided to abandon this approach. The problem is not whether there are enough fields, but that its design goal itself is not suitable for Embedding.

article-index.json is essentially a data structure oriented towards front-end display. Its core value is “displayability” rather than “semantic integrity.” It retains more organized outcome information, such as titles, links, summaries, category tags, series relationships, and topic spaces. This content is highly suitable for building “Related Articles” because it relies on explicit structural relationships.

However, the input requirements for Embedding are completely different; it relies on the original semantic expression of the article, rather than results that have already been structured. If we continue to use article-index.json, it will bring a direct problem: the semantics have already been compressed once before entering the Embedding. For example, HTML tags, code blocks, and unstructured content have already been cleaned up or weakened at this layer, and contextual information has also been summarized. While this is reasonable for a display system, for semantic modeling, it means information loss.

More importantly, article-index.json itself already contains a complete explicit structural system (categories, tags, series, knowledge maps, etc.). This system is an advantage in “Related Articles” because it makes recommendations stable and explainable, but in “Guess What You Think,” it becomes a constraint—because the goal of this feature is precisely to break out of explicit structures and discover potential associations that cannot be expressed through categories or tags. If we still rely on these structural fields, the final results are highly likely to converge again into patterns like the same category, same tag, or same series. Consequently, the difference between “Guess What You Think” and “Related Articles” would be significantly weakened.

If we trace back to the semantic-index.json generated in the first step of the ”Related Articles” feature, technically speaking, it actually already meets the conditions for Embedding, as it still retains information such as article titles, body content, and summaries. However, the design goal of semantic-index.json is not for Embedding, but rather to exist as an intermediate layer for the subsequent semantic indexing system. In addition to the body content, it also mixes data structures oriented towards indexing and recommendation systems, such as keywords, tags, and related articles.

In other words, although semantic-index.json can be used for Embedding, it is essentially a piece of “multi-purpose index data” rather than a semantic input layer specifically prepared for vectorization.

Therefore, there are actually two different design approaches here: the first is to directly reuse semantic-index.json, extracting the content or summary fields to generate vectors; the second is to separately build a data source specifically oriented towards Embedding.

Ultimately, I chose the latter—instead of continuing to expand on the existing indexing system, I rebuilt an independent data processing pipeline dedicated to Embedding:

WordPress 原始内容
    ↓
semantic-source.json
    ↓
semantic-vector.json
    ↓
semantic-path.json

↓

semantic-source.json

↓

2 semantic-source.json: Building a Semantic Source Suitable for Embedding

In this new data pipeline, the first step is to generate semantic-source.json. It is not a continuation of semantic-index.json, but rather bypasses the original index system to re-extract a semantic input layer specifically for Embedding from the raw WordPress content.

The reason why such an intermediate layer is needed is that the raw article content saved in WordPress is not suitable for direct use in Embedding calculations. In addition to the main text, an article often contains a large number of HTML tags, code blocks, image links, shortcodes, and various formatting information. While this content is essential for web display, a significant portion of it is considered noise data for semantic models.

Therefore, before entering the Embedding stage, it is first necessary to perform a preprocessing step on the article content specifically oriented towards semantic calculation, retaining as much of the content actually expressed by the article as possible, while eliminating parts that have little to do with semantic understanding.

After processing, each article will eventually be organized into a very simple structure:

{
  "id": "14257",
  "title": "为博客构建“轻量级知识索引”（三）：基于预计算语义索引的 WordPress 轻量推荐系统设计与实现",
  "url": "/technology/homedatacenter14257/",
  "content": "..."
}

Where:

• id is used to associate articles later;
• title stores the article title;
• url is used for the final redirection;
• content stores the semantic text used to generate the Embedding.

The core of the entire file actually lies in this content field.

If we simply export the full text of the article, although vectors can still be generated, the results may not be ideal: on one hand, blog articles vary greatly in length, with some being only a few hundred words and others potentially reaching tens of thousands of words; on the other hand, not all content has the same value for semantic modeling.

Taking technical articles as an example, although a large number of configuration files, command-line outputs, and code examples are important to readers, their contribution to determining the article's topic is often far less than that of the article title, core arguments, and key paragraphs.

The goal of semantic-source.json is not to save the full text of the article, but to extract an information summary that can represent the core semantics of the article.

In the design, I adopted a relatively simple strategy: first, retain the article title; then, extract the most representative content from the body; for excessively long articles, control the final output scale through length limits to avoid a single article occupying too much context space. The purpose of doing this is not to pursue absolute completeness, but to find a balance between semantic fidelity and processing efficiency.

From the results, semantic-source.json is more like a “semantically streamlined article library” specially prepared for Embedding. It retains the most core expressive content of the articles while avoiding a large amount of noise information unrelated to semantic computation.

With this organized semantic source data, the next step is to further convert these texts into vector representations, thereby entering the real Embedding stage.

3 semantic-vector.json: Converting Articles into Vectors

Once semantic-source.json is ready, the next step is very clear: convert these text contents into vectors that machines can understand and compute.

For humans, understanding an article relies on language, experience, and context. For example, when we see terms like “home data center”, “PVE virtualization platform”, and “Cloudflare Tunnel”, we naturally associate the relationships between them, and can even judge whether two articles are discussing similar topics.

But computers do not have this capability; in their eyes, text is essentially just a sequence of characters, and the concept of “meaning” does not exist. Therefore, if we want the system to automatically discover potential associations between articles, we must first map the text to a representation that can be mathematically computed.

What the Embedding model does is essentially complete this conversion. It reads the article content and encodes it into a high-dimensional vector. For the nomic-embed-text model in Ollama, each article will eventually be converted into a 768-dimensional array of floating-point numbers. This array itself is not readable, but its position in the vector space can reflect the semantic features expressed by the article.

Simply put, the closer the semantics of the articles, the closer their distance in the vector space usually is; while articles with greater differences in topics will be distributed further apart.

For example:

• Two articles about Cloudflare Tunnel will usually have vector positions that are relatively close;
• Articles about PVE virtualization and home data centers may also form a relatively concentrated area;
• In contrast, non-technical topics such as Buddhist reflections and life insights tend to be distributed far away from these technical articles.

This is also the core reason why Embedding can achieve semantic recommendation: the system no longer relies on categories, tags, or series relationships, but directly searches for related content based on the semantic similarity of the articles themselves.

After vectorization processing, semantic-vector.json will eventually be generated.

Compared with semantic-source.json, this file no longer saves the full text, but instead saves the vector results corresponding to the articles and necessary metadata. For example:

{
  "id": "14289",
  "title": "AI时代的创作重构：从信息处理到认知协作",
  "url": "/technology/cognition14289/",
  "vector": [0.0123, -0.0841, ...]
}

Where:

• id is used to establish a correspondence with WordPress articles;
• title and url are mainly used for subsequent display and debugging;
• vector is the semantic vector generated by the Embedding model.

semantic-vector.json can be understood as the “computation layer” in the entire semantic recommendation system: if semantic-source.json stores the original semantic expression of the articles, then semantic-vector.json stores the coordinate positions of these semantics in the vector space—its responsibility is not to display content, but to provide basic data for subsequent similarity calculations.

However, at this stage, the system still cannot directly provide recommendation results to users. This is because semantic-vector.json only records the vector of each article, but has not yet calculated the relationship network between articles.

Therefore, the final step of processing is still needed: based on the distance between these vectors, pre-calculate the closest neighbor nodes for each article and save the results.

This is the semantic-path.json to be introduced in the next section.

4 semantic-path.json: Vector-Based Semantic Associations

Once semantic-vector.json is generated, the semantic computation chain has actually completed the most critical step: each article has been mapped into a high-dimensional vector space.

But if we only stop at this layer, the system is still unusable. The reason is simple: vectors themselves cannot be directly used for recommendation. It can only express “what this article is”, but cannot answer “which one is it closer to”. What “Guess What You Think” really needs is the relationship between articles, rather than the semantic description of a single article.

Therefore, the problem to be solved at this layer is very clear: convert the distance relationships in the vector space into directly usable structured associations. The implementation method is essentially a batch similarity calculation. The system will use each article as a baseline to perform vector comparisons with all articles, usually using cosine similarity as the metric, and select the Top-K most similar articles as its semantic neighbors.

In the actual storage structure, semantic-path.json will solidify this relationship into a very direct data form. For example, the structure of an article might look like this:

{
  "id": "14289",
  "links": [
    {
      "target_id": "14136",
      "score": 0.87
    },
    {
      "target_id": "14228",
      "score": 0.82
    },
    {
      "target_id": "14164",
      "score": 0.79
    }
  ]
}

Where:

• id represents the current article;
• links represents the set of articles with the closest semantics to it;
• target_id is the associated article;
• score is the matching score obtained from the vector similarity calculation, used for sorting and filtering Top-K results.

In this way, the “distance relationship” that originally existed in the high-dimensional space is compressed into a structured data format, which is the origin of semantic-path.json.

From the perspective of system design, the role of this layer is to shift the similarity queries, which would otherwise need to be calculated in real-time during access, forward to the build phase. At runtime, no vector calculations are performed; instead, the pre-calculated relationship results are read directly. Without this layer, “Guess What You Think” would have to load all vectors and perform full calculations and sorting on every request. This is not a big problem when the content scale is small, but as the number of articles grows, it will quickly become a performance bottleneck.

And the significance of semantic-path.json lies in moving this step “forward”: trading offline computation for online performance, turning the recommendation logic into pure data reading. Of course, this design also implies a trade-off: the relationships between articles are solidified at the time of generation. Once built, subsequent recommendation results will not adjust in real-time with subtle changes in the model or data, but will rely on the structure calculated at that time.

But for a system like a personal blog with limited content scale and a relatively stable update rhythm, this trade-off is reasonable. What it brings in return is higher stability, simpler runtime logic, and more controllable system complexity.

Up to this layer, the entire Embedding chain is actually closed:

WordPress 原始内容
        ↓
semantic-source.json   （语义输入层）
        ↓
semantic-vector.json   （向量表示层）
        ↓
semantic-path.json     （语义关系层）
        ↓
猜你所想 / 语义推荐系统

Abstractly speaking, these three steps correspond to language expression, mathematical representation, and relationship modeling, respectively, and semantic-path.json is the layer that goes from “vector space” back to “usable structure”.

At this point, the underlying data foundation for “Guess What You Think” is complete. Next, we only need to let WordPress utilize these relationship results to build a stable and explainable recommendation presentation mechanism.

3 From Semantic Computation to Recommendation Display

1 semantic-source.json Generation Script

In the previous chapter, we introduced the role and data structure of semantic-source.json. It is the starting point of the entire Embedding data chain, responsible for extracting input data suitable for semantic computation from WordPress original content.

Therefore, the first step to complete is to read the post content from the database and generate semantic-source.json. The code for the corresponding script ”build_semantic_source.py” is as follows:

import requests
import re
import html
import json
import os
from urllib.parse import urlparse

# 根据实际环境修改
BASE_URL = "http://127.0.0.1"

# ⭐ semantic content 总长度限制
MAX_CONTENT_LENGTH = 8000

# ⭐ 单段最大长度
MAX_PARAGRAPH_LENGTH = 800


# -------------------------
# Step 1: 获取 WordPress 文章
# -------------------------
def fetch_posts(base_url):
    posts = []
    page = 1
    total_pages = None

    while True:
        url = f"{base_url}/wp-json/wp/v2/posts?page={page}&per_page=100"

        resp = requests.get(url)

        if resp.status_code != 200:
            raise Exception(
                f"Request failed at page {page}, status: {resp.status_code}"
            )

        data = resp.json()

        if total_pages is None:
            total_pages = int(resp.headers.get("X-WP-TotalPages", 1))
            print(f"Total pages: {total_pages}")

        posts.extend(data)

        print(
            f"Fetched page {page}, total posts: {len(posts)}"
        )

        if page >= total_pages:
            break

        page += 1

    return posts


# -------------------------
# URL -> path
# -------------------------
def extract_path(url: str) -> str:
    if not url:
        return ""

    return urlparse(url).path


# -------------------------
# HTML 清洗（保留章节结构）
# -------------------------
def clean_html(raw_html):
    text = raw_html or ""

    text = re.sub(
        r"<pre.*?>.*?</pre>",
        " ",
        text,
        flags=re.DOTALL | re.IGNORECASE
    )

    text = re.sub(
        r"<code>.*?</code>",
        " ",
        text,
        flags=re.DOTALL | re.IGNORECASE
    )

    text = re.sub(
        r"<script.*?>.*?</script>",
        " ",
        text,
        flags=re.DOTALL | re.IGNORECASE
    )

    text = re.sub(
        r"<style.*?>.*?</style>",
        " ",
        text,
        flags=re.DOTALL | re.IGNORECASE
    )

    # shortcode
    text = re.sub(r"\[/?[^\]]+\]", " ", text)

    # 标题
    text = re.sub(
        r"</?(h1|h2|h3|h4|h5|h6)[^>]*>",
        "\n",
        text,
        flags=re.IGNORECASE
    )

    # 段落
    text = re.sub(
        r"</?p[^>]*>",
        "\n",
        text,
        flags=re.IGNORECASE
    )

    # br
    text = re.sub(
        r"<br\s*/?>",
        "\n",
        text,
        flags=re.IGNORECASE
    )

    # 删除剩余 HTML
    text = re.sub(r"<[^>]+>", " ", text)

    text = html.unescape(text)

    # URL
    text = re.sub(r"https?://\S+", " ", text)

    # 特殊符号
    text = re.sub(
        r"[丨|｜•·■◆►▶●]+",
        " ",
        text
    )

    # 压缩空格（保留换行）
    text = re.sub(r"[ \t]+", " ", text)

    # 压缩连续换行
    text = re.sub(r"\n{3,}", "\n\n", text)

    return text.strip()


# -------------------------
# 将正文分段
# -------------------------
def split_paragraphs(text):
    paragraphs = [
        p.strip()
        for p in re.split(r"\n+", text)
        if p.strip()
    ]

    result = []

    for p in paragraphs:
        while len(p) > MAX_PARAGRAPH_LENGTH:
            result.append(p[:MAX_PARAGRAPH_LENGTH])
            p = p[MAX_PARAGRAPH_LENGTH:]

        if p:
            result.append(p)

    return result


# -------------------------
# Step 2: semantic content 构建
# -------------------------
def build_semantic_content(title, clean_text):
    paragraphs = split_paragraphs(clean_text)

    if not paragraphs:
        return title

    filtered = []

    for p in paragraphs:
        p = p.strip()

        # 太短
        if len(p) < 80:
            continue

        # 纯章节标题
        if re.match(r"^\d+(\.\d+)*\s+", p):
            if len(p) < 40:
                continue

        filtered.append(p)

    if not filtered:
        filtered = paragraphs

    total = len(filtered)

    selected = []

    # 开头部分
    selected.extend(filtered[:3])

    # 全文均匀采样
    if total > 10:
        sample_ratios = [
            0.10,
            0.20,
            0.30,
            0.40,
            0.50,
            0.60,
            0.70,
            0.80,
            0.90
        ]

        for ratio in sample_ratios:
            pos = int(total * ratio)

            if 0 <= pos < total:
                selected.append(filtered[pos])

    # 结尾部分
    if total >= 2:
        selected.extend(filtered[-2:])
    elif total >= 1:
        selected.append(filtered[-1])

    # 去重
    unique_paragraphs = []
    seen = set()

    for p in selected:
        if p not in seen:
            unique_paragraphs.append(p)
            seen.add(p)

    semantic_text = (
        f"{title}\n"
        + "\n".join(unique_paragraphs)
    )

    return semantic_text[:MAX_CONTENT_LENGTH].strip()


# -------------------------
# Step 3: 构建 semantic-source
# -------------------------
def build_semantic_source(posts):
    articles = []

    for post in posts:
        raw_content = post.get("content", {}).get("rendered", "")

        clean_text = clean_html(raw_content)

        title = post.get("title", {}).get("rendered", "")

        semantic_content = build_semantic_content(
            title,
            clean_text
        )

        article = {
            "id": str(post.get("id", "")),
            "title": title,
            "url": extract_path(post.get("link", "")),
            "content": semantic_content
        }

        articles.append(article)

    return articles


# -------------------------
# 输出 JSON
# -------------------------
def write_json(data, output_path=None):
    if output_path is None:
        base_dir = os.path.dirname(os.path.abspath(__file__))
        output_path = os.path.join(
            base_dir,
            "semantic-source.json"
        )

    with open(output_path, "w", encoding="utf-8") as f:
        json.dump(
            data,
            f,
            ensure_ascii=False,
            indent=2
        )

    print(f"\nJSON written to: {output_path}")


# -------------------------
# Main
# -------------------------
if __name__ == "__main__":
    posts = fetch_posts(BASE_URL)

    print(f"\nTotal posts fetched: {len(posts)}")

    semantic_source = build_semantic_source(posts)

    print(
        f"Semantic source built: "
        f"{len(semantic_source)} articles"
    )

    write_json(semantic_source)

    print("\nSample article:\n")

    if semantic_source:
        print(
            json.dumps(
                semantic_source[0],
                ensure_ascii=False,
                indent=2
            )
        )

The logic of this script is actually not complicated.

First, it retrieves all published posts via the WordPress REST API, and then extracts the post titles, URLs, and body content. Since Embedding calculations will be performed later, the focus here is not on the page display effect, but on the semantic information expressed by the posts themselves.

After retrieving the body content, the script first performs a content cleanup. WordPress posts typically contain a large amount of presentation-related information, such as HTML tags, code blocks, shortcodes, style content, and various links. While this content is necessary for page rendering, it often acts as noise for semantic calculations. Therefore, at this stage, the natural language expression of the posts is preserved as much as possible, while content unrelated to semantics is removed, thereby improving the quality of subsequent vectorization processing.

After completing the cleanup, the script splits the body content by paragraph and filters out content that is too short or headings that only contain section numbers. The purpose of doing this is to ensure that the content participating in subsequent Embedding has a relatively complete semantic expression as much as possible, rather than being disrupted by a large amount of scattered headings or navigation information.

Subsequently, instead of directly using the entire post, the script extracts a set of representative content snippets from the full text. In addition to retaining the beginning of the post to reflect the thematic background, it also performs uniform sampling from different positions throughout the text at a fixed ratio, and additionally retains the content of the ending. Compared to simply intercepting the first few paragraphs or a fixed-length text, this approach can better cover the overall structure of the post, ensuring that the generated semantic representation not only reflects the topics discussed at the beginning of the post but also retains important information from the middle and end sections.

Finally, the script combines the title and these filtered content snippets into a unified semantic text and writes it to semantic-source.json. The subsequent Embedding process will no longer directly process the original WordPress posts, but will instead perform vectorization calculations based on this preprocessed and compressed semantic data.

2 semantic-vector.json Generation Script

The prerequisite for performing the operations in this section is to have an embedding model that can generate Embeddings.

If there are no special requirements for cost and deployment methods, you can directly call OpenAI's text-embedding-3-small or other Embedding APIs provided by mainstream large model service providers. These commercial models usually have good Chinese comprehension and semantic expression capabilities, and are relatively simple to implement.

However, for scenarios like personal blogs, the number of posts tends to grow continuously. If calling a cloud API is required every time the semantic index is regenerated, it will generate additional costs over the long term and increase dependence on external services.

Since I have previously written an article about deploying the local embedding model nomic-embed-text via Ollama in a home data center, this article will directly use nomic-embed-text to complete the vector generation work (readers who are not familiar with this can refer to my previous article:Making Embedding an Infrastructure: Deploying Independent Embedding Services in PVE + LXC. Of course, it would be even better to deploy Ollama on an M-series Mac mini, as the embedding speed will be much faster. For example, in the actual production environment, I directly use the Qwen3-embedding:8B embedding model. For the specific setup and configuration methods, please refer to my previous article:Deploying Llama 3.2 on Mac mini (M4 Pro): A Complete Guide to Efficient Running and Cross-Origin Access Optimization via Ollama）。

It should be noted that the focus of this article is not on which specific Embedding model to use (such as OpenAI's commercial models, locally deployed nomic-embed-text, or the Qwen series embedding models, etc.), but rather on the construction method of the semantic indexing pipeline. Therefore, as long as the model can provide a standard Embedding interface, the overall implementation approach remains consistent.

However, different models may vary in semantic expression capability, computational resource consumption, and adaptation to Chinese contexts, but these differences are implementation-level choices and do not affect the overall architectural design.

After completing semantic-source.json, the next step is to convert the articles into vector representations. Therefore, the task at this stage is very clear: read semantic-source.json and generate the corresponding vector data for each article.

The code for the corresponding script build_semantic_vector.py is as follows:

import json
import requests
import os
import time
import numpy as np
import hashlib

# -------------------------
# Ollama embedding API
# -------------------------
# 根据实际环境进行修改
OLLAMA_URL = "http://127.0.0.1:11434/api/embed"
MODEL_NAME = "nomic-embed-text"

# -------------------------
# config
# -------------------------
MAX_SEGMENT_LENGTH = 1500
MAX_SEGMENTS = 8


# -------------------------
# load data
# -------------------------
def load_semantic_source(input_path=None):
    if input_path is None:
        base_dir = os.path.dirname(os.path.abspath(__file__))
        input_path = os.path.join(base_dir, "semantic-source.json")

    with open(input_path, "r", encoding="utf-8") as f:
        data = json.load(f)

    print(f"Loaded semantic source: {len(data)} articles")
    return data


# -------------------------
# load cache
# -------------------------
def load_existing_vectors(output_path=None):
    if output_path is None:
        base_dir = os.path.dirname(os.path.abspath(__file__))
        output_path = os.path.join(base_dir, "semantic-vector.json")

    if not os.path.exists(output_path):
        return {}

    with open(output_path, "r", encoding="utf-8") as f:
        data = json.load(f)

    return {item["id"]: item for item in data}


# -------------------------
# hash
# -------------------------
def make_hash(text):
    return hashlib.md5(text.encode("utf-8")).hexdigest()


# -------------------------
# split text
# -------------------------
def split_text_for_embedding(text):
    text = text.strip()
    if not text:
        return []

    paragraphs = []

    for line in text.split("\n"):
        line = line.strip()
        if not line:
            continue

        while len(line) > MAX_SEGMENT_LENGTH:
            paragraphs.append(line[:MAX_SEGMENT_LENGTH])
            line = line[MAX_SEGMENT_LENGTH:]

        if line:
            paragraphs.append(line)

    return paragraphs[:MAX_SEGMENTS]


# -------------------------
# embedding
# -------------------------
def get_embedding(text, retries=3):

    payload = {
        "model": MODEL_NAME,
        "input": text
    }

    for i in range(retries):
        try:
            resp = requests.post(OLLAMA_URL, json=payload, timeout=120)

            if resp.status_code == 200:
                data = resp.json()
                emb = data.get("embeddings", [])
                if emb:
                    return emb[0]

            print(f"[Retry {i}] HTTP {resp.status_code}: {resp.text}")

        except Exception as e:
            print(f"[Retry {i}] Exception: {e}")

        time.sleep(1.5 * (i + 1))

    raise Exception("Embedding failed")


# -------------------------
# build vectors (incremental)
# -------------------------
def build_semantic_vectors(articles, existing):
    vectors = []
    total = len(articles)

    for index, article in enumerate(articles, start=1):

        title = article.get("title", "")
        content = article.get("content", "")
        article_id = article.get("id", "")

        content_hash = make_hash(title + content)

        cached = existing.get(article_id)

        # -------------------------
        # skip unchanged
        # -------------------------
        if cached and cached.get("hash") == content_hash:
            print(f"[{index}/{total}] Skip: {title}")
            vectors.append(cached)
            continue

        print(f"[{index}/{total}] Embedding: {title}")

        segments = split_text_for_embedding(content)

        if not segments:
            print(f"[WARN] Empty content: {title}")
            continue

        embeddings = []

        for seg in segments:
            print(f"Segment length: {len(seg)}")

            emb = get_embedding(seg)

            if emb:
                embeddings.append(np.array(emb))

            time.sleep(0.1)

        if not embeddings:
            continue

        avg_vector = np.mean(embeddings, axis=0).tolist()

        vectors.append({
            "id": article_id,
            "title": title,
            "hash": content_hash,
            "vector": avg_vector
        })

    return vectors


# -------------------------
# write
# -------------------------
def write_json(data, output_path=None):
    if output_path is None:
        base_dir = os.path.dirname(os.path.abspath(__file__))
        output_path = os.path.join(base_dir, "semantic-vector.json")

    with open(output_path, "w", encoding="utf-8") as f:
        json.dump(data, f, ensure_ascii=False)

    print(f"\nJSON written to: {output_path}")


# -------------------------
# main
# -------------------------
if __name__ == "__main__":

    articles = load_semantic_source()
    existing = load_existing_vectors()

    vectors = build_semantic_vectors(articles, existing)

    write_json(vectors)

    print("\nSample vector:\n")

    if vectors:
        print({
            "id": vectors[0]["id"],
            "title": vectors[0]["title"],
            "vector_dimensions": len(vectors[0]["vector"])
        })

Unlike the previous section where semantic-source.json was generated, this step truly enters the Embedding calculation phase.

The script first reads semantic-source.json the organized article content in it, and attempts to load the existing vector cache (semantic-vector.json), if not found, a new one is created. For each article, the script calculates the hash value of its title and content to determine whether the article has already had a semantic vector generated and its content remains unchanged. Only articles with updated content, or brand new articles, will be sent to the locally deployed Ollama service, where the nomic-embed-text model generates the corresponding semantic vector; unchanged articles will directly reuse the cached vectors, thereby saving redundant computation time.

Considering that some articles are quite long, submitting the entire content at once would not only increase the processing pressure on the model but might also affect the vector quality. Therefore, the script will first segment the content, splitting the article into multiple smaller semantic fragments, and perform Embedding calculations on each separately.

Once all segments are vectorized, these segment vectors are averaged to obtain the final semantic vector representing the entire article.

This combination of incremental computation and caching mechanism does not aim for absolute precision, but rather achieves a relatively reasonable balance among computational cost, response time, and semantic coverage. For scenarios with limited content scale like personal blogs, this approach is already capable of obtaining sufficiently stable semantic expression capabilities.

At this level, the articles are no longer text in the traditional sense, but have been converted into vector representations that computers can use for mathematical operations and similarity comparisons. The subsequent semantic-path.json generation is precisely built upon these vector data.

Note:

This implementation was initially built based on the nomic-embed-text embedding model during the debugging phase. In the design of a semantic indexing system, the embedding model essentially acts as the “definer of the semantic space.” Therefore, replacing the embedding model is not a simple “model name switch,” but a system-level change that may affect the distribution of the entire semantic retrieval results.

Different models (such as nomic-embed-text used during the debugging phase and qwen3-embedding:8b which I currently use in practice) differ in the following aspects:

• Input tokenization methods
• Vector space distribution structures
• API parameters and return formats
• Compression strategies for expressing semantics

This code retains the nomic-embed-text version as a baseline implementation, whose primary value lies in:
Lower resource consumption, suitable for local debugging and rapid validation;
Relatively stable semantic expression, making it easy to build an initial indexing system;
Serving as a lightweight baseline model, helping to compare the performance differences of different embedding models.

In production environments (such as the currently used qwen3-embedding:8b), adaptation and adjustments to the embedding requests and index generation logic must be made based on actual model characteristics, rather than just replacing the model name.

3 semantic-path.json Generation Script

After completing semantic-vector.json, the next step is to establish semantic relationships between articles.

Therefore, the task at this stage is also very clear: read the article vectors saved in semantic-vector.json, calculate the similarity between articles, and finally generate semantic-path.json and save it in the following path of WordPress:

`wp-content/themes/主题目录/cache/semantic-path.json`

Since I am using the Argon theme, the corresponding save path is:

wp-content/themes/argon-theme-master/cache/semantic-path.json

The script corresponding to this step is build_semantic_path.py:

import json
import os
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

# -------------------------
# 配置
# -------------------------
TOP_K = 6
SEMANTIC_THRESHOLD = 0.72

# 排除一些不需要处理的url，这里是我博客的那些地图
EXCLUDED_PATH_KEYWORDS = [
    "/cloudflaremap/",
    "/aimap/",
    "/singingmap/",
    "/roadmap/"
]

# -------------------------
# Step 1
# -------------------------
def load_semantic_source(input_path=None):
    if input_path is None:
        base_dir = os.path.dirname(
            os.path.abspath(__file__)
        )
        input_path = os.path.join(
            base_dir,
            "semantic-source.json"
        )

    with open(
        input_path,
        "r",
        encoding="utf-8"
    ) as f:
        data = json.load(f)

    print(
        f"Loaded semantic source: {len(data)} articles"
    )

    return data

# -------------------------
# Step 2
# -------------------------
def load_semantic_vector(input_path=None):
    if input_path is None:
        base_dir = os.path.dirname(
            os.path.abspath(__file__)
        )
        input_path = os.path.join(
            base_dir,
            "semantic-vector.json"
        )

    with open(
        input_path,
        "r",
        encoding="utf-8"
    ) as f:
        data = json.load(f)

    print(
        f"Loaded semantic vectors: {len(data)} articles"
    )

    return data

# -------------------------
# 过滤地图页
# -------------------------
def is_excluded(article):
    url = article.get("url", "")

    return any(
        keyword in url
        for keyword in EXCLUDED_PATH_KEYWORDS
    )

# -------------------------
# Step 3
# -------------------------
def build_article_map(
    source_data,
    vector_data
):
    vector_map = {
        item["id"]: item
        for item in vector_data
    }

    article_map = {}

    for item in source_data:
        if is_excluded(item):
            continue

        article_id = item["id"]

        vector_item = vector_map.get(article_id)

        if not vector_item:
            continue

        article_map[article_id] = {
            "id": article_id,
            "title": item.get("title", ""),
            "url": item.get("url", ""),
            "vector": vector_item.get(
                "vector",
                []
            )
        }

    print(
        f"Merged articles: {len(article_map)}"
    )

    return article_map

# -------------------------
# semantic similarity
# -------------------------
def calculate_semantic_similarity(a, b):
    va = np.array(
        a["vector"]
    ).reshape(1, -1)

    vb = np.array(
        b["vector"]
    ).reshape(1, -1)

    return float(
        cosine_similarity(va, vb)[0][0]
    )

# -------------------------
# semantic recommendation
# -------------------------
def build_semantic_recommendation(
    current_article,
    article_map
):
    scored = []

    for article_id, article in article_map.items():
        if article_id == current_article["id"]:
            continue

        if is_excluded(article):
            continue

        sim = calculate_semantic_similarity(
            current_article,
            article
        )

        scored.append({
            "id": article["id"],
            "title": article["title"],
            "url": article["url"],
            "score": sim
        })

    scored.sort(
        key=lambda x: x["score"],
        reverse=True
    )

    filtered = [
        item
        for item in scored
        if item["score"] >= SEMANTIC_THRESHOLD
    ]

    if len(filtered) < TOP_K:
        filtered = scored[:TOP_K]

    return filtered[:TOP_K]

# -------------------------
# 主流程
# -------------------------
def build_semantic_path(article_map):
    results = {}

    total = len(article_map)

    for idx, (
        article_id,
        article
    ) in enumerate(
        article_map.items(),
        start=1
    ):
        print(
            f"[{idx}/{total}] "
            f"{article['title']}"
        )

        items = build_semantic_recommendation(
            article,
            article_map
        )

        results[article_id] = {
            "article_id": article_id,
            "article_title": article["title"],
            "article_url": article["url"],
            "items": items
        }

    return results

# -------------------------
# 输出 JSON，需要根据实际场景进行修改，这里我是直接输出到自用的wordpress的主题目录下。
# -------------------------
def write_json(data, output_path=None):
    if output_path is None:
        output_path = (
            "/docker/wordpress/html/"
            "wp-content/themes/"
            "argon-theme-master/cache/"
            "semantic-path.json"
        )

    output_dir = os.path.dirname(
        output_path
    )

    if (
        output_dir
        and not os.path.exists(output_dir)
    ):
        os.makedirs(
            output_dir,
            exist_ok=True
        )

    with open(
        output_path,
        "w",
        encoding="utf-8"
    ) as f:
        json.dump(
            data,
            f,
            ensure_ascii=False,
            indent=2
        )

    print(
        f"\nJSON written to: {output_path}"
    )

# -------------------------
# Main
# -------------------------
if __name__ == "__main__":
    source_data = load_semantic_source()

    vector_data = load_semantic_vector()

    article_map = build_article_map(
        source_data,
        vector_data
    )

    semantic_path = build_semantic_path(
        article_map
    )

    write_json(semantic_path)

    first_key = next(
        iter(semantic_path)
    )

    print("\nSample:\n")

    print(
        json.dumps(
            semantic_path[first_key],
            ensure_ascii=False,
            indent=2
        )
    )

The script first reads all article vectors saved in semantic-vector.json, and then calculates the semantic similarity between articles. Here, Cosine Similarity is used as the distance metric to measure how close two articles are in the semantic space.

For each article, the system iterates through the vectors of other articles and sorts them by similarity from highest to lowest. It then filters out several of the most relevant articles to serve as future recommendation candidates for that article.

Considering that there are various special pages in the blog, such as navigation pages and learning maps, although these contents participate in the site structure construction, they are not suitable as recommendation targets. Therefore, the script actively excludes these pages during the calculation process.

In the final generated semantic-path.json, each article will save a set of pre-calculated semantic relationships.

4 WordPress Front-End Integration

At this point, the entire semantic calculation pipeline is actually complete. As for the WordPress frontend, it does not need to understand Embeddings, vector spaces, or similarity algorithms. The only thing it needs to do is very simple: find the corresponding list of related articles from semantic-path.json based on the current article ID, and then display the results. The entire frontend phase is essentially just a data reading process.

The corresponding PHP code in WordPress is as follows (the left side is the ”Related Articles” feature, and the right side is the ”Guess What You Want” feature):

function add_recommend_list_after_content(content) {

    if (!is_single() || !in_the_loop() || !is_main_query()) {
        returncontent;
    }

    post_id = get_the_ID();

    // =========================
    // 地图页 / 结构页过滤
    // =========================map_slugs = [
        '/map/',
        '/cloudflaremap/',
        '/aimap/',
        '/roadmap'
    ];

    current_url =_SERVER['REQUEST_URI'] ?? '';

    foreach (map_slugs asslug) {
        if (strpos(current_url,slug) !== false) {
            return content;
        }
    }

    // =========================
    // 读取 article-index.json
    // =========================article_index_file = get_template_directory() . '/cache/article-index.json';

    if (!file_exists(article_index_file)) {
        returncontent;
    }

    article_index = json_decode(
        file_get_contents(article_index_file),
        true
    );

    if (!isset(article_index[post_id])) {
        return content;
    }

    // =========================
    // 读取 semantic-path.json
    // =========================semantic_path_file = get_template_directory() . '/cache/semantic-path.json';
    semantic_path = [];

    if (file_exists(semantic_path_file)) {
        semantic_path = json_decode(
            file_get_contents(semantic_path_file),
            true
        );
    }

    current =article_index[post_id];

    // =========================
    // 左侧：相关文章（article-index.json）
    // =========================related_ids = current['related'] ?? [];related_list = [];

    foreach (related_ids asrelated_id) {
        if (!isset(article_index[related_id])) {
            continue;
        }
        related_list[] = [
            'title' =>article_index[related_id]['title'],
            'url'   =>article_index[related_id]['url']
        ];
    }related_count = 5;
    related_list = array_slice(related_list, 0, related_count);

    // =========================
    // 右侧：延伸阅读（semantic-path.json）
    // =========================semantic_list = [];
    semantic_count = 3;

    if (!empty(semantic_path) && isset(semantic_path[post_id])) {
        semantic_items =semantic_path[post_id]['items'] ?? [];

        foreach (semantic_items as item) {
            // 去重：避免和相关文章重复duplicate = false;
            foreach (related_list asrelated_item) {
                if (related_item['url'] ===item['url']) {
                    duplicate = true;
                    break;
                }
            }
            if (duplicate) {
                continue;
            }

            semantic_list[] = [
                'title' =>item['title'],
                'url'   => item['url']
            ];

            if (count(semantic_list) >= semantic_count) {
                break;
            }
        }
    }

    // =========================
    // 最终为空
    // =========================
    if (empty(related_list) && empty(semantic_list)) {
        returncontent;
    }

    // =========================
    // UI
    // =========================
    html = '<div class="post-recommend-wrapper" style="
        display:flex;
        gap:0;
        margin:28px 0;
        padding:16px 18px;
        background:#f0f7ff;
        border-left:4px solid #3b82f6;
        border-radius:6px;
        font-size:14px;
        line-height:1.7;
        color:#444;
    ">';

    // =====================================================
    // 左侧：相关文章
    // =====================================================html .= '<div style="
        flex:1;
        padding-right:16px;
        min-width:0;
    ">';

    html .= '<div style="
        font-size:13px;
        color:#666;
        margin-bottom:8px;
        font-weight:600;
    ">
        📎 相关文章
    </div>';

    if (!empty(related_list)) {
        foreach (related_list asitem) {
            html .= '<div style="margin:6px 0;">
                <a href="' . esc_url(item['url']) . '" target="_blank" style="
                    color:#1d4ed8;
                    text-decoration:none;
                    font-weight:500;
                ">
                ' . esc_html(item['title']) . '
                </a>
            </div>';
        }
    } else {html .= '<div style="color:#999;">
            暂无相关文章
        </div>';
    }

    html .= '</div>';

    // =====================================================
    // 中间分割线
    // =====================================================html .= '<div style="
        width:1px;
        background:#dbeafe;
        margin:0 10px;
    "></div>';

    // =====================================================
    // 右侧：猜你所想
    // =====================================================
    html .= '<div style="
        flex:1;
        padding-left:16px;
        min-width:0;
    ">';html .= '<div style="
        font-size:13px;
        color:#666;
        margin-bottom:8px;
        font-weight:600;
    ">
        🔗 猜你所想
    </div>';

    if (!empty(semantic_list)) {
        foreach (semantic_list as item) {html .= '<div style="margin:6px 0;">
                <a href="' . esc_url(item['url']) . '" target="_blank" style="
                    color:#2563eb;
                    text-decoration:none;
                    font-weight:500;
                ">
                ' . esc_html(item['title']) . '
                </a>
            </div>';
        }
    } else {
        html .= '<div style="color:#999;">
            暂无猜你所想
        </div>';
    }html .= '</div>';

    html .= '</div>';

    returncontent . $html;
}

add_filter('the_content', 'add_recommend_list_after_content', 20);

The above PHP code can be added via the Code Snippets plugin or in functions.php.

Looking at the overall logic, this code mainly completes three steps.

First, the system determines whether the current page is an article page and filters out special pages such as knowledge maps and learning maps. These pages themselves serve navigation and indexing functions and are not suitable as participants in the recommendation system, so they are directly excluded from the recommendation logic.

Subsequently, the program reads two data files, article-index.json and semantic-path.json, under the path ”wordpress/html/wp-content/themes/argon-theme-master/cache/”. Among them, article-index.json stores structured relationships used to generate the “Related Articles” on the left; while semantic-path.json stores semantic relationships calculated based on Embeddings, used to generate the “Guess What You Want” on the right.

Once the current article ID is obtained, the system extracts the corresponding recommendation results from both data files and assembles them into the final display content. To avoid duplicate recommendations, if the same article has already appeared in the “Related Articles” list, it will not appear in the “Guess What You Want” section again.

From an implementation perspective, the role WordPress plays here is actually very simple. It is neither responsible for vector generation nor similarity calculation, but merely reads the pre-generated JSON data and outputs it to the article page according to a preset style.

Therefore, no matter what Embedding model is used in the backend, and no matter how the semantic associations are generated, the frontend code does not need to change. For WordPress, semantic-path.json is essentially just an ordinary data file.

At this point, the entire implementation chain of the “Guess What You're Thinking” feature has completed its closed loop. Starting from the original article content, through semantic organization, vectorization, and similarity calculation, it ultimately forms semantic associations between articles and presents them to readers on the frontend in the form of a recommendation list.

5 “Guess What You Think” Effect Display and Recommendation Analysis

To observe the differences between the two recommendation mechanisms more intuitively, let's take the article "Understanding RAG from Scratch (Part 1): Principles and Complete Process Analysis“ as an example. Below this article, the recommendation results given by the traditional ”Related Articles" section are as follows:

• Understanding RAG from Scratch (II): Running a Local RAG Demo on Mac mini—A Practical Guide to the Minimal Architecture
• Building a Self-Hosted Embedding Model with Ollama + Chatbox Knowledge Base in Practice
• The Most Convenient AI App Frontend: Comprehensive Introduction to Chatbox + User Guide
• Vectors: The Universal Language in the AI World
• Starting Your AI Journey: A Detailed Introduction to Local LLM UIs and LLM API Providers

Meanwhile, the recommendation results given by the “Guess What You're Thinking” section are:

• Building a “Lightweight Knowledge Index” for a Blog (Part 2): JSON Structure and Generation Script Implementation
• Building a “Lightweight Knowledge Index” for a Blog (Part 1): Structure Design and Build Process
• Building a “Lightweight Knowledge Index” for My Blog (Part 3): Design and Implementation of a Lightweight WordPress Recommendation System Based on Pre-computed Semantic Index

At first glance, this result seems somewhat similar to the traditional “Related Articles” because the recommended content is also focused on related topics such as knowledge indexing, semantic retrieval, and recommendation systems, but the underlying implementation logic of the two is actually different.

“Related Articles” essentially relies on the article structure. Whether it is categories, tags, series relationships, or manually curated knowledge maps, it focuses on “which system this article belongs to,” so the recommendation results usually continue along a set knowledge path. On the other hand, “Guess What You're Thinking” focuses on something else: what the article is actually discussing.

For example, although “Building a ”Lightweight Knowledge Index' for a Blog (Part 3)" and the first two articles in the series have an inherent series relationship, from a semantic level, they also revolve around semantic index construction, data organization methods, and recommendation system design. In the semantic space established by Embedding, these articles are naturally clustered in close proximity, so even without relying on categories, tags, or series relationships at all, there is a high probability they will be linked together.

This is also one of the biggest differences between semantic recommendation and traditional related articles: it does not care which category an article is placed in, but instead tries to understand the actual content expressed by the article, and then establishes associations based on semantic proximity.

Of course, in some cases, this association may also cross the original categorization system. For example, although two articles have different topics, if both discuss similar concepts such as knowledge organization, system design, or cognitive structures, they may still be close to each other in the semantic space and thus recommended together. However, whether such cross-domain associations appear, and to what extent, largely depends on the Embedding model used and how the semantic space is constructed.

From this perspective, “Related Articles” is more like continuing along a preset path on an established map, while “Guess What You're Thinking” is about finding the nearest neighbors in the semantic space. The two are not mutually exclusive but rather establish connections between content from different dimensions.

Actually, at the very beginning, I just wanted to use an embedding model to build a “Guess What You're Thinking” feature, recommending related content through vector similarity. However, due to insufficient consideration of chunking strategies, cleaning rules, and semantic coverage in the early stages, the performance of “Guess What You're Thinking” was unstable. So, I took a step back at that time and implemented a “Related Articles” module based on more intuitive structural information to ensure a basic content association experience.

Later, as the overall pipeline was gradually optimized—including adjustments to text cleaning, chunking strategies, and vector construction methods—the performance of “Guess What You're Thinking” gradually met expectations. However, a very natural phenomenon occurred: its recommendation results were quite close to those of “Related Articles” in many cases.

In principle, this is not surprising—both are essentially making “content relevance” judgments, with one based on semantic vectors and the other on structural or explicit relationships. Therefore, when the content quality is high and the topic is clear, it is normal for the results to converge.

I once considered whether to keep only one of the modules, but ultimately chose to have both coexist. The reason is simple: although their recommendation results are similar, their focus is not entirely the same. “Guess What You're Thinking” leans more toward potential interest expansion in the semantic space, while “Related Articles” leans more toward deterministic associations at the structural level. The former provides space for exploration, while the latter provides reading anchors. Overlaying the two actually makes content navigation more stable and layered.

However, to distinguish primary and secondary importance, ”Guess What You're Thinking” only displays a maximum of 3 recommendation results.

4 Summary

Over the past few years, technologies like semantic retrieval, vector databases, and RAG have gradually become important infrastructure for AI applications. But for a personal blog, the problems that actually need to be solved are often not that complex. Most cases, what we need is not a real-time updated semantic retrieval system that supports hundreds of millions of data points, but simply to establish associations between articles that better align with the content itself.

Therefore, throughout the implementation process, I always adhered to one principle: keep the complexity in the build phase as much as possible, and leave the simplicity to the runtime phase. Through a three-layer structure of semantic-source.json, semantic-vector.json, and semantic-path.json, all semantic calculations are completed in advance, and ultimately WordPress is only responsible for reading the results and displaying the content.

Although this approach loses real-time computing capabilities, it yields extremely low running and maintenance costs. For scenarios like personal blogs, where the update frequency is far lower than the access frequency, this trade-off is often more reasonable. From the results, “Related Articles” is responsible for maintaining the knowledge structure, while “Guess What You're Thinking” is responsible for discovering semantic associations. Together, they form a recommendation mechanism that balances explainability and exploration.

However, for me, the greatest value of this system may not lie in the recommendations themselves. What is truly meaningful is the semantic association network established by Embedding—in traditional blog systems, connections between articles usually come from categories, tags, series articles, and manually maintained knowledge maps; but after introducing semantic associations, articles begin to have a connection relationship established based on the content itself.

This relationship does not rely on pre-designed categorization systems or manual curation, but is naturally formed by the actual content expressed in the articles. It can be said that categories, tags, and knowledge maps describe the knowledge structure in the author's eyes, while the semantic associations established by Embedding attempt to discover the relationships presented by the content itself.

In a sense, this can also be seen as an attempt to gradually evolve the blog from an information collection into a knowledge network.