AI Semantic Space Exploration (Part 2): From a Word to an Article—How Semantic Vectors Are Generated
本文最后更新于 136 天前,其中的信息可能已经有所发展或是发生改变,如有失效可到评论区留言。
Article Abstract
The article explores how AI transforms natural language into semantic vectors through in-context learning, revealing a hierarchical abstraction process from words to articles and then to themes. Based on the Transformer model, AI dynamically generates context-relevant vectors for words by analyzing massive amounts of text, then uses pooling techniques to compress articles into fixed-length semantic representations, and finally forms thematic center vectors by aggregating vectors of similar articles. This mechanism not only explains the underlying principles of embedding models, semantic search, and RAG systems, but also supports the category recommendation function of the blog's right-hand menu, demonstrating the multi-level organizational capability of semantic vectors from concrete text to abstract themes in practical scenarios.
Qwen3-14B · 2026-07-08

1 Questions Arising from the Right-Hand Menu

In my previous article implementing the “right-hand menu” feature (see:Building a “Lightweight Knowledge Index” for My Blog (Part 7): Right Menu—The Unified Organizational Mechanism of Semantic Space and Classification System), I used a category recommendation algorithm based on semantic similarity. Its core idea is not complicated: first, generate a corresponding semantic vector for each article in the blog, and then, based on the overall distribution of article vectors under the same category, calculate a center vector that can represent the theme of that category.

When a visitor opens an article, the system calculates the cosine similarity between the semantic vector of the current article and the center semantic vectors of all categories, and selects the three most similar categories as recommended categories.

But once I actually got this system running, new questions arose. Calculating the category center vector is not complex; it is essentially just aggregating the article vectors under the same category. What really made me curious was: where do the semantic vectors of articles actually come from? Why can an article of several thousand or even tens of thousands of words ultimately be compressed into a vector consisting of just a few thousand numbers? Furthermore, why can these seemingly random numbers represent the theme and meaning of the article? If we cannot answer these questions, then we have only learned how to use vectors without truly understanding them.

In the previous article of this series, "AI Semantic Space Exploration (Part 1): Vectors—The Universal Language in the AI World", I already introduced that for AI, vectors are like a universal language. Whether it is text, images, or sound, everything can ultimately be converted into vector form to enter a unified mathematical space. However, the previous article mainly addressed: what vectors are, and why they can become the universal language in the AI world.

The problem this article aims to solve is: how does AI generate these vectors? How does a word become a vector? How does a sentence express complete semantics on this basis? Why can an article ultimately have its own vector representation? Furthermore, why can a category also form its own center vector?

Only after understanding this process can we truly comprehend: what embedding models are doing; why semantic search works; why RAG needs vector databases; and the principles behind the right-hand menu feature of the articles in my blog.

2 Starting from a Word: How AI Understands Language

Since we already know that a word will ultimately have its own semantic vector, a new question naturally arises:Who assigned this vector to it?

When many people first encounter semantic vectors, they subconsciously assume that the model has a huge ”dictionary” saved inside beforehand, which records a set of numbers corresponding to each word. For example, “sun” corresponds to one set of numbers, and “moon” corresponds to another. When the model encounters these words, it only needs to look up this dictionary to get the corresponding vector.

This idea sounds reasonable, but it is actually not how modern AI understands language. The reason is not hard to grasp: natural language itself is constantly evolving, and new vocabulary appears almost every day. If we truly relied on manually maintaining such a dictionary, no matter how large it was made, it could never cover all words, let alone different languages, different fields, and constantly changing new concepts. For AI, this approach is both unrealistic and incapable of truly understanding language.

Modern AI is closer to the process of how humans learn language: a newborn child does not truly understand the meaning of the word ”sun” just because someone tells them its definition. Instead, through constantly listening to others speak, reading text, and observing the real world, they gradually discover that ”sun” often appears together with words like ”sky,” “daytime,” “light,” and “warmth,” thereby gradually understanding the connections between these concepts. After long-term accumulation, even without anyone specifically explaining it, they can build a cognitive understanding of the concept of the ”sun.”

The way AI learns language is actually very similar: during the training process, the model reads massive amounts of text—from news reports and encyclopedic knowledge to technical documents, novels, and forum discussions, almost any large-scale text that can be obtained could become training data. The model does not memorize every sentence; instead, through constantly reading these texts, it gradually learns the relationships between different words.

For example, it will find that ”programmer” often appears with ”code,” “programming,” and “software”; “hospital” often appears with ”doctor,” “patient,” and “treatment”; and “Cloudflare” often co-occurs with technical terms like ”CDN,” “DNS,” “WAF,” and “Tunnel.” These frequently recurring linguistic patterns ultimately form the model's understanding of word meanings.

In other words,What AI learns is not a fixed definition in a dictionary, but rather the complex relationships with other concepts formed through extensive language use.。

Because of this, the meanings of words are not manually designed, but are learned bit by bit from massive amounts of language data. The specific training methods used by different models may vary, from early Word2Vec to today's various Transformer-based Embedding models, all of which have undergone continuous development and evolution. But no matter how the underlying algorithms change, they all share a common goal:To enable the model to learn the semantic relationships behind words through large amounts of language data, rather than relying on a manually maintained static dictionary.

However, after understanding this, a new question arises: if AI has already learned the meanings of words like ”sun,” “moon,” and “Cloudflare,” how does this abstract understanding turn into a set of concrete numbers, ultimately forming what we call ”word vectors”? Furthermore, why can these numbers participate in subsequent mathematical calculations and eventually combine into the semantic vector of a sentence, an article, or even a category?

3 From Understanding to Numbers: How AI Expresses Semantics

Through the previous discussion, we already know that AI does not understand language by consulting a pre-prepared dictionary, but rather by continuously learning the semantic relationships between words while reading massive amounts of text. In other words, semantic representations for concepts like ”sun,” “programmer,” and “Cloudflare” have gradually formed within the model.

However, new questions follow. Understanding itself is a very abstract concept; humans can say ”I understand this word,” but for computers, there is no such storage form as ”understanding.” Computers do not directly save concepts like ”the sun represents a star” or “programmers are responsible for writing code” internally, because they simply cannot process these abstract meanings directly.

In fact, whether it is text, images, sound, or video, what computers can ultimately process are only numbers. The text we usually see is, in essence, just a string of numbers stored according to certain encoding rules for computers. For example, a computer does not know what the two Chinese characters ”太” and “阳” represent individually; it only knows the encoding values corresponding to these two characters. When a program reads these characters, it only reads the corresponding numbers and does not automatically understand the meaning contained within them.

This also means that simply letting AI learn language is far from enough. If we want the model to be able to further compare whether two words are similar, calculate the meaning of a sentence, or even complete various tasks such as semantic search and article recommendation, it must ultimately save the semantics it has learned in a form that computers can process.

The most direct way is to convert semantics into numbers. However, the numbers here do not mean assigning an ID to each word. For example, we could completely stipulate:

太阳 → 0001
月亮 → 0002
地球 → 0003

This approach is indeed a form of digitization, but it only serves as an ”identifier.” For a computer, there is no semantic relationship between 0001 and 0002; just like a person's ID card number, it is merely used to distinguish different objects and does not indicate whether they are similar to each other.

Therefore, what a word ultimately corresponds to is not a single number, but a set of numbers. In AI, this set of numbers is usually called aVector, and each number contained in the vector can be seen as one of itsdimension (Dimension)。

Therefore, when we say a model generates a 768-dimensional vector、1024-dimensional vectoror 1536-dimensional vector, it does not mean that there are 768, 1024, or 1536 different types of vectors, but rather that this vector consists of 768, 1024, or 1536 numerical values, with each value corresponding to one of its dimensions. For example, assuming a certain Embedding model generates a 1536-dimensional semantic vector, then its structure can be simply represented as follows (the numbers above represent the dimension numbers, and the values below represent the values corresponding to those dimensions):

image.png

Looking at these values individually makes almost no sense, because no single dimension can represent a specific meaning on its own. What is truly meaningful is the whole formed by combining them together, which collectively determines the position of the word in the semantic space and expresses the overall semantics of the word.

In other words, what we really care about is not a single number itself, butthe overall representation formed collectively by this entire set of numbers. Because of this, we usually do not try to explain what a single dimension in a vector represents, but rather view the entire vector as an indivisible whole. When the vectors of two words are close to each other, it indicates that the semantics they express are more similar; the further the distance, the greater the difference in semantics.

At this point, we have finally arrived at the concept repeatedly mentioned in the previous article—semantic vectorIt is not a string of numbers artificially prescribed, nor is it a simple ID; rather, it is a semantic representation established by AI after learning language in order to perform mathematical calculations.

However, a new question arises here. If every word has its own semantic vector, how does a sentence, a paragraph, or even an article of several thousand words get its own semantic vector? Is it really as simple as adding up the vectors of all the words to get the vector of the entire article?

4 From Text to Vectors: How Embeddings Generate Semantic Representations

In the previous chapter, we already learned that AI ultimately represents semantics as a vector. Now, a new question arises: are these vectors pre-stored, or are they calculated in real-time by the model based on the input content?

If we follow the simplest line of thought, we might think of a method: prepare a fixed vector for each word. When the model sees ”sun”, it directly retrieves the vector corresponding to ”sun”; when it sees ”moon”, it retrieves the vector corresponding to ”moon”.

In early word vector models, this approach was indeed used—each word corresponded to a fixed vector. But soon, a fundamental problem was discovered. Take the previous example again: ”Apples are delicious” and "Apple released a new generation of iPhone." The word "Apple" in these two sentences clearly does not have the same meaning. If we still use the same fixed vector to represent them, they would be indistinguishable in the eyes of the model.

It is precisely to solve this problem that modern AI models have begun to adopt a new approach:Instead of preparing a fixed vector for each word, they dynamically generate the semantic representation under the current context by combining the entire context. This process is precisely the core work accomplished by modern Embedding models.

However, to generate this context-combined semantic representation, relying solely on the traditional Embedding layer is not enough; it also requires a context modeling structure like the Transformer. The Transformer can be understood as the model's ”reader.” When a piece of text is input into the model, it does not process it word by word independently like traditional programs; instead, it observes the entire sentence simultaneously, analyzing the relationship between each word and the other words.

Take the two previous examples: ”Apples are delicious” and “Apple released a new generation of iPhone.” Although both sentences contain the word ”Apple,” after analyzing the context, the Transformer will find that in the first sentence, “Apple” is more closely connected to concepts like ”delicious” and “fruit”; whereas in the second sentence, it forms different semantic associations with concepts like “released,” "iPhone," and "tech company."

Therefore, what the model ultimately generates are actually two different semantic representations. That is to say, in modern AI, we no longer believe that ”Apple” has only one fixed vector, but rather thatevery time ”Apple” appears in a different context, it may have a different semantic vector. In other words, modern AI truly represents is no longer the ”word” itself, but ratherthe semantics of the word in the current context。

Therefore, although we are used to saying ”word vectors,” in modern Transformer models, more accurately, it representsContextual RepresentationThis is also one of the biggest differences between modern Embedding and early word vector models:Vectors are no longer statically stored, but are dynamically generated after the model analyzes contextual relationships.

Understanding this also allows us to explain a question that has long puzzled many beginners: why modern Embedding models can recognize polysemy and even distinguish subtle semantic differences expressed by the same word in different scenarios. The answer is not that the model internally stores many different sets of word vectors, but because it re-reads the current context every time and recalculates the semantic representation of the current word based on the entire text.

By now, we already know that the Transformer can combine context to generate a corresponding semantic representation for each word. However, the Embedding truly used for semantic search, knowledge base retrieval, or RAG is not the hundreds of vectors corresponding to hundreds of words individually, but ratherthe entire text ultimately corresponding to only one vector.

So, how are the semantic representations of so many words further fused and ultimately compressed into a single semantic vector that can represent the entire article?

5 From Words to Articles: How AI Organizes Semantics

Through the previous discussion, we already know that the Transformer will combine context to dynamically generate a corresponding semantic representation for each词in the article. However, ”word” here is actually just a simplified term for ease of understanding. Strictly speaking, what modern AI models truly process are not words, but rather TokenA Token can be a complete word, a Chinese character, a part of a word, or even a punctuation mark, so it is not completely equivalent to the ”word” we commonly understand.

That is to say, when an article is input into the model, what the model ultimately gets is not a single semantic representation, but a set of semantic representations corresponding to the number of Tokens—each Token has its own semantic representation.


Note: In the subsequent discussions, when referring to the internal processing of the model, we will consistently use Token this term.


However, this brings up a new problem: if an article contains two thousand Tokens, the model will ultimately generate two thousand semantic representations. Yet, when we perform semantic search, similar article recommendation, or RAG retrieval, we do not take two thousand semantic representations to compare them one by one; instead, we hope to usea single vectorto represent the entire article.

So, where does this single article vector come from? The easiest method to think of is to save all the semantic representations corresponding to all Tokens. However, doing so not only occupies a large amount of storage space, but also different articles have different lengths—some have only dozens of Tokens, while others may contain thousands of Tokens. This means each article would have a different number of semantic representations, making it difficult to compare them directly.

Therefore, the model needs to further complete a new task:compressing a variable number of Token semantic representations into a fixed-length article vector. This process is commonly referred to as Pooling。

The ”compression” here is not simply discarding information, but rather retaining the most important semantic features of the entire article as much as possible, and condensing this information into a fixed-length vector. In this way, whether the original text is just a few sentences or contains tens of thousands of words, an article vector of the exact same dimension can ultimately be obtained.

For example, assuming a certain Embedding model ultimately outputs a 1536-dimensional vector, then whether it is a 500-word blog post or a hundreds-of-thousands-of-words e-book, after Pooling, a 1536-dimensional article vector of the same length can be obtained:

image.png

In this way, texts of different lengths have a unified mathematical representation, allowing their semantic similarity to be directly calculated.

In practical scenarios, different Embedding models may adopt different Pooling strategies, such as taking the average of all Token semantic representations , using the semantic representation of a specific position, or employing more complex aggregation methods. However, for most modern text Embedding models, their goal is consistent:to let a fixed-length vector express the semantics of the entire article as completely as possible.

For users, we usually do not need to care about which specific Pooling method is adopted inside the model, because these tasks have already been automatically completed by the model. What we truly need to understand is:An article vector is not the semantic representation of a single Token, but rather the overall representation obtained after organizing, compressing, and abstracting the semantic representations of all Tokens in the entire article.

This also explains why an article of several thousand words can ultimately be compressed into a vector consisting of only a little over a thousand numerical values. It does not simply save every Token in the article intact into the vector, but rather compresses the entire article into a fixed-length representation while preserving the overall semantic features. Therefore, articles with similar semantics will ultimately be closer to each other in the vector space.

For the right-hand menu in my blog, this step is precisely the foundation of the entire recommendation system. Each article generates a corresponding article vector, and whether it is calculating the similarity between articles or generating category center vectors in the future, it is all built upon these article vectors. However, up to this point, what we have obtained is still onlya single article'ssemantic representation.

If a category contains dozens or even hundreds of articles, how are these article vectors further abstracted into a center vector that can represent the topic of the entire category?

6 From Articles to Themes: How AI Abstracts Semantics

Through the previous discussion, we already know that an article can ultimately be represented as a fixed-length article vector. Regardless of the length of the original text or the amount of content, after Embedding and Pooling, a mathematical representation that can represent the semantics of the entire article can be obtained.

However, a new question arises: if an article can have its own semantic vector, can a larger topic, such as ”Cloudflare”, “Linux”, or “AI”, also have its own semantic vector?

At first glance, this seems somewhat incredible, because an article is actual text that can be directly input into a model to generate a vector; whereas a topic like ”Cloudflare” is not a concrete article itself, but rather a higher-level concept formed collectively by many related articles. So, how does AI represent such an abstract concept?

The answer is actually simpler than imagined: if we view each article as a point in the semantic space, then articles discussing the same topic will usually be distributed in close proximity. Although the content they focus on varies—for example, some introduce Cloudflare Tunnel, some introduce Cloudflare Workers, and others introduce Cloudflare DNS—because they all revolve around the same topic, their distance in the semantic space is often not too far.

In other words, a topic is not a single point, but rather more like a region in the semantic space.

That being the case, a natural idea emerges: if we can find the ”center” of this region, would this central position be able to represent the entire topic?

In practical applications, this center can usually be calculated by aggregating all article vectors under that topic. The most common method among them is to directly calculate the average of all article vectors. Although different systems can also adopt more complex aggregation algorithms, their goals are consistent: to find a center vector that best represents the overall semantics of the entire topic.

In my application scenario, the semantic vector of a topic is not directly generated by the model, but is further aggregated on the basis of article vectors—articles describe concrete content, while topics extract the common semantics behind these contents.

For the right-hand menu of my blog articles, this very idea is utilized. The system first generates a corresponding article vector for each article, and then calculates the corresponding topic center vector based on the vectors of all articles under the same topic. When a visitor opens an article, the system does not directly compare all articles, but first calculates the cosine similarity between the current article's vector and the center vectors of all topics to find the closest TOP 3 topics, and then uses the same approach to further find the closest TOP 4 articles within those topics.

The entire process can be simplified as:

   当前文章
      │
      ▼
  文章语义向量
      │
      ▼
与所有主题中心向量计算余弦相似度
      │
      ▼
选择最相似的3个主题
      │
      ▼
生成右侧菜单第一层导航
      │
      ▼
与3个相似主题中所有文章向量计算余弦相似度
      │
      ▼
选择最相似的4篇文章
      │
      ▼
生成右侧菜单第二层导航

In fact, this ”from concrete to abstract” idea is not only applicable to my blog. In many AI applications, we see a similar process: a large amount of concrete data is continuously aggregated, ultimately forming an overall semantic representation that can represent a certain class of things. For example, topic organization in knowledge bases, document clustering, content recommendation, and even the way knowledge is organized in many RAG systems are essentially built upon this idea.

Looking back, this article of ours has actually completed a full path:

 词语
  │
  ▼
Token
  │
  ▼
Token 语义表示
  │
  ▼
文章向量
  │
  ▼
主题向量

From an initial word to a vector that can ultimately represent an entire topic, AI does not complete all the work at once, but continuously organizes, compresses, and abstracts semantics at different levels. Because of this, whether it is a word, an article, or a topic, in the eyes of AI, it can essentially be represented as a vector in the semantic space, only that they correspond to different semantic levels.

7 Summary: How Semantics Are Formed Step by Step

In the previous article, we got to know the representation method of vectors and understood why AI needs to use vectors to process semantics. This article further answers another question:How exactly these semantics turn into semantic vectors step by step.

Starting from a single word to the model understanding its meaning in the current context by combining context; from the semantic representation of a Token to the semantic vector of the entire article; and then to multiple article vectors further abstracted into a semantic representation that can represent the entire topic, what we see is not several independent technologies, but a complete process of continuously organizing, compressing, and abstracting semantics.

The greatest significance of understanding this process does not lie in learning a few new professional terms, but in establishing a new perspective of observation.

Many AI technologies seem completely unrelated to each other when first encountered: Embedding, vector databases, semantic search, RAG, related article recommendations, knowledge organization... It seems that every time you learn a technology, you need to re-understand a new set of concepts.

But once you understand how semantics are formed step by step, you will find that these technologies are actually all built on the same foundation:Allowing semantics at different levels to be represented, organized, and ultimately mapped into the same semantic space for calculation.

Because of this, when you truly understand the formation process of semantic vectors and then go on to learn these specific technologies, your focus will often no longer be on ”which interface should be called” or “which framework should be used”, but you will begin to think about another more fundamental question:How are the semantics in the system represented? And how do they gradually complete organization and abstraction as information continuously converges?

For me, this is also the original intention of writing this series of articles. Compared to introducing a specific framework or implementing a specific function, I prefer to understand the common principles behind these technologies. Because specific models will be constantly updated, frameworks will continuously iterate, and technologies popular today might be replaced by new solutions in a few years; buthow semantics are represented, organized, and abstractedthis entire set of ideas often possesses a longer vitality.

Once you truly establish this perspective, looking back at semantic search, RAG, knowledge graphs, and even functions like the right-hand menu and knowledge map in my blog, you will find that although their forms of expression are different, they are essentially just specific applications of the same idea in different scenarios.

From this moment on, seemingly independent technologies such as Embedding, RAG, vector databases, and semantic search are no longer scattered pieces of knowledge, but begin to become different components of the same cognitive system.

📚 Series Articles: Exploring AI Semantic Space (2 / 3)

View Related Categories · 3 Matches
📌 Content Structure Prompt:
This content is part of the following knowledge map, where you can view the complete content path: AI Learning Map。
📎 Related Articles
Share this article
The blog content is original, please indicate the source when reposting! The RSS address of the blog is:https://blog.tangwudi.com/feed, welcome to subscribe; if needed, you can join theTelegram Groupto discuss questions together.
No Comments

Send Comment Edit Comment


				
|´・ω・)ノ
ヾ(≧∇≦*)ゝ
(☆ω☆)
(╯‵□′)╯︵┴─┴
 ̄﹃ ̄
(/ω\)
∠( ᐛ 」∠)_
(๑•̀ㅁ•́ฅ)
→_→
୧(๑•̀⌄•́๑)૭
٩(ˊᗜˋ*)و
(ノ°ο°)ノ
(´இ皿இ`)
⌇●﹏●⌇
(ฅ´ω`ฅ)
(╯°A°)╯︵○○○
φ( ̄∇ ̄o)
ヾ(´・ ・`。)ノ"
( ง ᵒ̌皿ᵒ̌)ง⁼³₌₃
(ó﹏ò。)
Σ(っ °Д °;)っ
( ,,´・ω・)ノ"(´っω・`。)
╮(╯▽╰)╭
o(*////▽////*)q
>﹏<
( ๑´•ω•) "(ㆆᴗ
😂
😀
😅
😊
🙂
🙃
😌
😍
😘
😜
😝
😏
😒
🙄
😳
😡
😔
😫
😱
😭
💩
👻
🙌
🖕
👍
👫
👬
👭
🌚
🌝
🙈
💊
😶
🙏
🍦
🍉
😣
Source: github.com/k4yt3x/flowerhd
Kaomoji
Emoji
Little Dinosaur
Flower!
Previous
Next
       

👋 Welcome to “Wudi's Personal Blog”

Here, long-term exploration is mainly carried out around the following directions:

🧱 Personal Digital Infrastructure and Blog System Construction
☁️ Cloudflare and Network Architecture Practice
🧠 AI and Knowledge System Exploration
🛡️ Network Security and Access Optimization
🎵 Music and Sound Cognition
👁️ Cognitive Perspectives and Worldviews