1. Questions arising from the right-hand menu

In my previous article on implementing the "right-side menu" feature (see:Building a Lightweight Knowledge Index for Blogs (Part 7): Right-hand Menu – A Unified Organization Mechanism for Semantic Space and Classification SystemThis paper utilizes a semantic similarity-based classification and recommendation algorithm. The core idea is not complex: first, a semantic vector is generated for each article in the blog; then, based on the overall distribution of article vectors within the same category, a central vector representing the theme of that category is calculated.

When a visitor opens an article, the system calculates the cosine similarity between the semantic vector of the current article and the central semantic vectors of all categories, and selects the three most similar categories as the recommended categories.

However, after I actually got the system up and running, new questions arose. Calculating the category center vector isn't complicated; it's essentially just aggregating the vectors of articles within the same category. What truly intrigued me was: where exactly are the semantic vectors of an article derived? Why can an article of several thousand or even tens of thousands of words be compressed into a vector composed of a few thousand numbers? Furthermore, how can these seemingly random numbers represent the article's theme and meaning? If we can't answer these questions, then we've only learned to use vectors, but haven't truly understood them.

In the previous article of this series,Exploring the Semantic Space of AI (Part 1): Vectors—The Universal Language of the AI WorldIn my previous article, I introduced that vectors are like a universal language for AI. Whether it's text, images, or sound, they can ultimately be converted into vector form and enter a unified mathematical space. However, the previous article mainly addressed: what are vectors, and why can vectors become the universal language in the AI world?

This article aims to address the following questions: How does AI generate these vectors? How does a word become a vector? How does a sentence express its complete semantics? Why can an article ultimately have its own vector representation? Furthermore, why can a category also form its own central vector?

Once we understand this process, we can truly see what the Embedding model does; why semantic search works; why RAG needs a vector database; and the principles behind the right-hand menu function in my blog articles.

2. Starting with a single word: How AI understands language

Now that we know that a word will eventually have its own semantic vector, a new question naturally arises:Who assigned this vector to it?

When many people first encounter semantic vectors, they instinctively think that the model has a huge "dictionary" stored inside, recording a set of numbers corresponding to each word. For example, "sun" corresponds to one set of numbers, "moon" corresponds to another set of numbers, and when the model encounters these words, it only needs to look them up in this dictionary to get the corresponding vector.

This idea sounds reasonable, but it's not actually how modern AI understands language. The reason isn't hard to understand: natural language itself is constantly evolving, with new words appearing almost daily. If we relied on manually maintaining such a dictionary, no matter how large it became, it would be impossible to cover all words, let alone different languages, different fields, and constantly evolving new concepts. For AI, this approach is neither realistic nor a true way to understand language.

Modern AI is closer to the process of human language learning: a newborn child doesn't truly understand the meaning of "sun" simply because someone tells them its definition. Instead, they gradually discover that "sun" often appears alongside words like "sky," "daytime," "light," and "warmth" through listening to others speak, reading texts, and observing the real world, thus gradually understanding the connections between these concepts. After a long period of accumulation, even without specific explanation, they can develop an understanding of the concept of "sun."

The approach to AI language learning is actually quite similar: during training, the model reads massive amounts of text—from news reports and encyclopedic knowledge to technical documents, novels, and forum discussions; almost any large-scale text available can become training data. The model doesn't memorize every sentence, but rather gradually learns the relationships between different words through continuous reading of these texts.

For example, it finds that "programmer" often appears alongside "code," "programming," and "software"; "hospital" often appears alongside "doctor," "patient," and "treatment"; and "Cloudflare" often appears alongside technical terms such as "CDN," "DNS," "WAF," and "Tunnel." These numerous recurring linguistic patterns ultimately constitute the model's understanding of word meanings.

in other words,AI doesn't learn a fixed definition from a dictionary, but rather the complex relationships between other concepts formed through extensive language use..

This is why the meanings of words are not designed manually, but learned bit by bit from massive amounts of language data. Different models may use different specific training methods, from the early Word2Vec to today's various embedding models based on Transformers, all of which have undergone continuous development and evolution. But regardless of the changes in the underlying algorithms, they all share a common goal:This allows the model to learn the semantic relationships behind words from a large amount of language data, rather than relying on manually maintaining a static dictionary.

However, after understanding these concepts, a new question arises: if AI has already learned the meanings of words like "sun," "moon," and "cloudflare," how does this abstract understanding transform into a set of concrete numbers, ultimately forming what we call "word vectors"? Furthermore, why can these numbers participate in subsequent mathematical calculations and ultimately combine to form the semantic vectors of a sentence, an article, or even a category?

3 From Understanding to Numbers: How AI Expresses Semantics

As we've discussed, AI doesn't understand language by consulting a pre-prepared dictionary, but rather by continuously learning the semantic relationships between words while reading massive amounts of text. In other words, the model has gradually developed semantic representations for concepts like "sun," "programmer," and "Cloudflare."

However, new problems arise. Understanding is a very abstract concept. Humans can say "I understand this word," but for computers, there is no such storage form as "understanding." Computers do not directly store concepts like "the sun represents a star" or "programmers are responsible for writing code" internally because they simply cannot directly process these abstract meanings.

In fact, whether it's text, images, sound, or video, computers can ultimately only process numbers. The text we see is essentially just a string of numbers stored according to certain encoding rules for a computer. For example, a computer doesn't know what the Chinese characters "太" (tài) and "阳" (yáng) represent; it only knows their corresponding encoded values. When a program reads these characters, it simply reads the corresponding numbers and doesn't automatically understand the meaning they contain.

This also means that simply teaching AI to speak is far from enough. If we want the model to be able to further compare whether two words are similar, calculate the meaning of a sentence, or even perform various tasks such as semantic search and article recommendation, it must ultimately store the semantics it has learned in a form that computers can process.

The most direct method is to convert semantics into numbers. However, these numbers don't mean assigning a unique number to each word. For example, we could stipulate:

Sun → 0001 Moon → 0002 Earth → 0003

This method is indeed digitization, but it can only serve as an "identity identifier." For computers, there is no semantic relationship between 0001 and 0002, just like a person's ID number; it is only used to distinguish different objects and does not indicate whether they are similar to each other.

Therefore, a word ultimately corresponds not to a single number, but to a set of numbers. In AI, this set of numbers is usually referred to as a...vectorEach number contained in the vector can be considered as one of its components.Dimension.

Therefore, when we say that a model generates... 768-dimensional vector,1024-dimensional vectoror 1536-dimensional vectorThis doesn't mean there are 768, 1024, or 1536 different types of vectors, but rather that the vector consists of 768, 1024, or 1536 numerical values, each corresponding to one of the dimensions. For example, suppose an embedding model generates a vector for the word "sun". 1536-dimensional semantic vectorIts structure can be simply represented as follows (the numbers above represent the dimension numbers, and the values below represent the values corresponding to that dimension):

These values, taken individually, are almost meaningless, because no single dimension can represent a specific meaning on its own. What is truly meaningful is the whole they form together, which collectively determines the word's position in the semantic space and expresses the word's overall meaning.

In other words, what we really care about is not a particular number itself, but...This entire set of numbers together constitutes a unified representation.This is why we usually don't interpret what a single dimension of a vector represents, but rather treat the entire vector as an indivisible whole. When the vectors of two words are close to each other, it means that they express more similar meanings; the greater the distance, the greater the semantic difference between them.

At this point, we finally arrived at the concept that was repeatedly mentioned in the previous article—semantic vectorsIt is not a string of numbers defined by humans, nor is it a simple number; rather, it is a semantic representation that AI has created after learning a language in order to perform mathematical calculations.

However, a new question arises here. If each word has its own semantic vector, then how do a sentence, a paragraph, or even an article of several thousand words obtain its own semantic vector? Is it simply a matter of adding up the vectors of all the words to get the vector of the entire article?

4 From Text to Vector: How Embedding Generates Semantic Representations

In the previous chapter, we learned that AI ultimately represents semantics as a vector. This raises a new question: are these vectors pre-stored, or are they calculated in real-time by the model based on the input?

If we follow the simplest approach, we might think of a method: prepare a fixed vector for each word. When the model sees "sun," it directly retrieves the vector corresponding to "sun"; when it sees "moon," it retrieves the vector corresponding to "moon."

Early word vector models did indeed use this approach—each word corresponded to a fixed vector. However, a fundamental problem was quickly discovered. Consider the previous example: "Apples are delicious"; "Apple released a new generation of iPhones." The "apple" in these two sentences clearly has different meanings. If the same fixed vector is used, they appear identical to the model.

It is precisely to address this problem that modern AI models have begun to adopt a new approach:Instead of preparing a fixed vector for each word, we dynamically generate a semantic representation in the current context by combining the entire context. This process is precisely the modern Embedding The core work accomplished by the model.

However, generating such context-aware semantic representations requires more than just traditional embedding layers; it also necessitates a context-modeling structure like the Transformer. The Transformer can be understood as the model's "reader." When text is input into the model, it doesn't process each word independently like a traditional program, but instead observes the entire sentence simultaneously, analyzing the relationships between each word and other words.

Take the two examples above: Apples are delicious; Apple released a new generation of iPhones. Although both sentences contain the word "apple," after analyzing the context, Transformer will find that in the first sentence, "apple" is more closely associated with concepts such as "delicious" and "fruit"; while in the second sentence, it forms different semantic associations with concepts such as "release," "iPhone," and "technology company."

Therefore, the model ultimately generates two different semantic representations. In other words, in modern AI, we no longer consider "apple" to have only one fixed vector, but rather...Each time the word "apple" appears in a different context, it may have a different semantic vector. In other words, modern AI is no longer truly representing the "word" itself, but...The semantics of a word in its current context.

Therefore, although we are used to saying "word vectors," in modern Transformer models, it more accurately represents...Contextual RepresentationThis is also one of the biggest differences between modern embedding and early word vector models:Vectors are no longer statically stored, but dynamically generated after the model analyzes the contextual relationships.

Understanding this allows us to explain a question that has long puzzled many beginners: why can modern embedding models recognize polysemy and even distinguish subtle semantic differences in the same word across different contexts? The answer isn't that the model internally stores many different sets of word vectors, but rather that it rereads the current context each time and recalculates the semantic representation of the current word based on the entire text.

So far, we know that the Transformer can combine context to generate a corresponding semantic representation for each word. However, the embeddings actually used for semantic search, knowledge base retrieval, or RAG are not hundreds of vectors corresponding to hundreds of words, but rather...The entire text ultimately corresponds to only one vector.

So how are the semantic representations of so many words further integrated and finally compressed into a semantic vector that can represent the entire article?

5 From Words to Articles: How AI Organizes Semantics

Based on the preceding discussion, we already know that Transformer takes into account the context and applies it to each element in the text.词Dynamically generate corresponding semantic representations. However, the term "word" here is merely a simplified way of speaking for ease of understanding. Strictly speaking, modern AI models do not actually process words, but rather... TokenA token can be a complete word, a Chinese character, part of a word, or even a punctuation mark, so it is not entirely equivalent to the "word" we understand in everyday life.

In other words, when an article is input into the model, the model does not ultimately obtain a semantic representation, but a set of semantic representations corresponding to the number of tokens—each token has its own semantic representation.

Note: In the following discussion, when referring to the internal processing of the model, we will consistently use... Token This term.

However, this raises a new problem: if an article contains two thousand tokens, the model will ultimately generate two thousand semantic representations. When performing semantic search, similar article recommendation, or RAG retrieval, we don't compare each of these two thousand semantic representations individually; instead, we aim to use...A vectorThis represents the entire article.

So, where does this article vector come from? The most obvious method is to store all the semantic representations corresponding to all tokens. However, this not only requires a lot of storage space, but also because different articles have different lengths, some with only a few dozen tokens and others with thousands of tokens. This means that each article will have a different number of semantic representations, making it difficult to directly compare them.

Therefore, the model needs to complete a new task:The semantic representation of a variable number of tokens is compressed into a fixed-length article vector. This process is usually called Pooling.

The "compression" here doesn't simply mean discarding information, but rather preserving as many of the most important semantic features of the entire article as possible and condensing this information into a fixed-length vector. In this way, whether the original text contains only a few sentences or tens of thousands of words, the final result will be an article vector with identical dimensions.

For example, assuming an embedding model outputs a 1536-dimensional vector, then whether it's a 500-word blog post or an ebook of hundreds of thousands of words, after pooling, it will produce a 1536-dimensional text vector of the same length.

In this way, texts of different lengths have a unified mathematical representation, and their semantic similarity can be directly calculated.

In real-world scenarios, different embedding models may employ different pooling strategies, such as taking all... Token semantic representation The methods used include averaging, using semantic representations at specific locations, or employing more complex aggregation methods. However, for most modern text embedding models, the goal is consistent:The goal is to use a fixed-length vector to represent the semantics of the entire article as completely as possible.

For users, we usually don't need to care about which pooling method the model uses internally, because that's all done automatically by the model. What we really need to understand is:The article vector is not a semantic representation of a single token, but rather a holistic representation obtained by organizing, compressing, and abstracting the semantic representations of all tokens in the entire article.

This explains why an article of several thousand words can ultimately be compressed into a vector consisting of only a little over a thousand values. It doesn't simply store every token from the article intact into the vector; rather, it compresses the entire article into a fixed-length representation while preserving its overall semantic features. Therefore, articles with similar semantics will ultimately be located closer together in the vector space.

For the right-hand menu in my blog, this step is the foundation of the entire recommendation system. Each article generates a corresponding article vector, and subsequent calculations, whether of similarity between articles or generating category center vectors, are all based on these article vectors. However, at this point, we still only have...Single articleThe semantic representation of .

If a category contains dozens or even hundreds of articles, how can these article vectors be further abstracted into a central vector that can represent the entire category's theme?

6 From Article to Topic: How AI Abstracts Semantics

Based on the preceding discussion, we now know that an article can ultimately be represented as a fixed-length article vector. Regardless of the length or content of the original text, after embedding and pooling, we can obtain a mathematical representation that can represent the semantics of the entire article.

However, a new question arises: if an article can have its own semantic vector, can a larger topic, such as "Cloudflare", "Linux", or "AI", also have its own semantic vector?

At first glance, this seems unbelievable, because an article is a real text that can be directly input into a model to generate vectors; while a topic like "Cloudflare" is not a specific article in itself, but rather a higher-level concept formed by many related articles. So, how does AI represent such an abstract concept?

The answer is actually simpler than you might think: if you consider each article as a point in a semantic space, then articles discussing the same topic will usually be located in close proximity. Although they may focus on different aspects—for example, some might introduce Cloudflare Tunnel, some Cloudflare Workers, and others Cloudflare DNS—they are all centered around the same theme, so their semantic distance is usually not too far.

In other words, a topic is not a single point, but rather a region in a semantic space.

Given this, a natural thought arises: if we can find the "center" of this area, then could this central location represent the entire theme?

In practical applications, this center vector is typically calculated by aggregating the vectors of all articles under that topic. The most common method is to directly calculate the average of all article vectors. Although different systems may employ more complex aggregation algorithms, their goals are consistent: to find a center vector that best represents the overall semantics of the entire topic.

In my application scenario, the semantic vector of a topic is not directly generated by the model, but is obtained by further aggregating the article vectors—the articles describe the specific content, while the topic extracts the common semantics behind this content.

This idea is used in the right-hand menu of my blog posts. The system first generates a corresponding article vector for each article, and then calculates the corresponding topic center vector based on the vectors of all articles under the same topic. When a visitor opens an article, the system does not directly compare all articles, but first calculates the cosine similarity between the vector of the current article and the center vectors of all topics, finds the top 3 most similar topics, and then uses the same approach to find the top 4 most similar articles within that topic.

The entire process can be simplified as follows:

   Current Article │ ▼ Article Semantic Vector │ ▼ Calculate Cosine Similarity with All Topic Center Vectors │ ▼ Select the 3 Most Similar Topics │ ▼ Generate First-Level Navigation of the Right-Hand Menu │ ▼ Calculate Cosine Similarity with All Article Vectors in the 3 Similar Topics │ ▼ Select the 4 Most Similar Articles │ ▼ Generate Second-Level Navigation of the Right-Hand Menu

In fact, this "from concrete to abstract" approach isn't limited to my blog. We see similar processes in many AI applications: large amounts of concrete data are continuously aggregated to ultimately form a holistic semantic representation that can represent a certain type of thing. For example, topic organization, document clustering, content recommendation in knowledge bases, and even the knowledge organization methods in many RAG systems are essentially built upon this idea.

Looking back, this article has actually completed a full path:

 Words | ▼ Token | ▼ Token Semantic Representation | ▼ Article Vector | ▼ Topic Vector

From a single word to a vector representing an entire topic, AI doesn't complete the work all at once. Instead, it continuously organizes, compresses, and abstracts semantics at different levels. Therefore, whether it's a word, an article, or a topic, from AI's perspective, it can essentially be represented as a vector in a semantic space, only at different semantic levels.

7. Summary: How semantics is formed step by step

In the previous article, we learned about vectors as a representation method and why AI needs to use vectors to process semantics. This article further answers another question:How exactly do these semantics gradually transform into semantic vectors?

Starting with a single word, the model combines context to understand its meaning in the current context; from the semantic representation of tokens to the semantic vector of the entire article; and then further abstracting multiple article vectors into a semantic representation that can represent the entire topic, what we see is not a few independent technologies, but a complete process of continuously organizing, compressing, and abstracting semantics.

The greatest significance of understanding this process lies not in learning a few new technical terms, but in establishing a new perspective.

Many AI technologies seem unrelated when you first encounter them: Embedding, vector databases, semantic search, RAG, related article recommendations, knowledge organization... It seems that you need to understand a new set of concepts every time you learn a new technology.

But once you understand how semantics are formed step by step, you'll find that these technologies are all built on the same foundation:This allows semantics at different levels to be represented, organized, and ultimately mapped to the same semantic space for computation.

Therefore, once you truly understand the formation process of semantic vectors, when learning these specific technologies, the focus is no longer on "which interface to call" or "which framework to use," but rather on a more fundamental question:How is semantics represented in the system? And how is it gradually organized and abstracted as information continues to accumulate?

For me, this is also the original intention behind writing this series of articles. Rather than introducing a specific framework or implementing a specific function, I'd rather understand the common principles behind these technologies. Because specific models are constantly being updated, frameworks are constantly iterating, and technologies popular today may be replaced by new solutions in a few years; but...How to represent, organize, and abstract semanticsThis entire set of ideas, however, often has a longer lifespan.

Once you truly establish this perspective, when you look back at semantic search, RAG, knowledge graphs, and even the right-hand menu and knowledge map features in my blog, you'll find that although they appear in different forms, they are essentially just specific applications of the same idea in different scenarios.

From this moment on, seemingly independent technologies such as Embedding, RAG, vector databases, and semantic search are no longer fragmented pieces of knowledge, but begin to become different components within the same cognitive system.