1 When Computers Begin to “Understand” Humans

With the development of artificial intelligence, AI is gradually entering the daily lives of ordinary people. Compared to computer programs in the past that required learning how to operate them, today's AI has a very obvious change: you can talk to it just like you usually talk to other people, telling it what you want to know, what problems you have encountered, and what tasks you need to complete, or even just expressing an incomplete idea, and it can often understand your meaning based on the context and provide a corresponding answer.

On the surface, this seems to be just a change in the way humans interact with computers, but looking deeper, this change is actually far from being as simple as it appears:Because human language is not just a set of simple commands.

The same sentence in different contexts may have completely different meanings; a seemingly ordinary sentence may also contain jokes, sarcasm, hints, or even meanings that are not directly spoken. Many times, we even omit some information that is mutually understood by default, yet the other party is still able to understand the content we truly want to express.

But in the face of these complex expressions, today's AI is often able to provide contextually appropriate responses. This is also why, when having a conversation with AI, we easily get a very intuitive feeling:It seems to really understand.

And when this ability further extends to answering questions, analyzing information, explaining concepts, reasoning, and even continuously communicating with people around a question, we easily develop another association:Does it also possess some kind of human-like thinking ability? After all, in our daily experience, an object that is able to continuously communicate and respond to questions like this easily makes people feel thatthere is also some kind of thinking activity behind it。

This cannot help but raise a question: “Looking like it is thinking,” does it really mean it can think like a human? If not, then by what means exactly did computers acquire this human-like language ability?

To clarify these questions, perhaps we need to start with humans themselves and re-examine a question we take for granted:How exactly do humans learn to use language?

2 Familiarity with Three Hundred Tang Poems Enables One to Chant Them Even If One Cannot Write Poetry

“Familiarity with three hundred Tang poems enables one to chant them even if one cannot write poetry” is a saying very familiar to Chinese people. It is usually used to describe how a person who originally could not write poetry seems to be able to gradually acquire a certain ability to “write poetry” simply through extensive reading and familiarity with poems.

But if we think about it carefully, this matter is actually quite intriguing:Why can a person learn to write poetry just by reading a lot of poems?

According to the way of learning we usually understand, if we want to learn to do something, it seems we should first master the corresponding rules. So, if we want to learn to write poetry, we should first learn what five-character and seven-character lines are, what rhyming, tonal patterns, and antithesis are, understand the formats of different poetic styles, and how poetic lines should be organized. After mastering these rules, we then create according to the rules. This is a very clear learning path:First know the rules, then act according to the rules.

But this saying describes another path: it does not emphasize learning all poetry theories first, but rather emphasizes a very simple process:Familiarity through readingThe so-called “familiarity through reading” refers to making oneself familiar with a large number of specific poetic lines through continuous contact and repeated reading. In this process, he may gradually know what kind of sentences sound like poetry, what kind of expressions are natural when put together, and what kind of rhythm reads smoothly. Even if he cannot accurately state the rules therein, he may slowly form something we often call a “sense of language.”

For example, a person may not be able to explain clearly what tonal patterns are, but can feel that a sentence reads “wrong”; or they may not be able to accurately explain the characteristics of a certain poetic style, but can write sentences with a similar style.

In other words, the language ability formed by humans through learning is not just the mastery of explicit rules, but may also contain many laws behind specific examples that are difficult to describe directly. When these laws are gradually internalized, they are no longer just “things seen,” but begin to become experience that can be drawn upon when facing new expressions.

From this perspective, what it describes is actually a learning phenomenon we are not unfamiliar with. Take practicing spoken English as an example: learning English in China, many people start all the way from vocabulary and grammar, studying for many years, yet may not be able to communicate naturally; but some elderly people who have received almost no systematic English education, once in an English-speaking environment, facing real communication needs every day, may only need a few months to be able to buy things, ask for directions, and even bargain with others in English.

Of course, they do not necessarily master complete English grammar, nor can they necessarily explain why every sentence they say is said that way, but in the process of continuous contact and use, some expressions that originally required deliberate thinking will become familiar unconsciously, and the laws in language will slowly settle into a kind ofdirectly usable language ability。

3 When “Familiarity” Becomes an Ability

Through the previous discussion, we already know that,mastering rules first is not the only path for humans to acquire language ability. Extensive contact and repeated use can also make language laws that originally required deliberate thinking graduallyinternalize into an ability that does not need to be deliberately called upon。

When this ability is truly formed, a very important change will occur: the experience accumulated in the past is no longer only used to process content already seen, but begins to be able to act onnew expressionsEven if a person has never written on a certain topic, they can still write a poem conforming to this style based on the language experience accumulated in the past. This poem is not a simple copy of a past poem; it may even contain sentences never seen in the past, yet it can still conform to the language habits they have already formed.

And to form such new expressions, the language experience accumulated in the past first needs to participate in the understanding and association of the content in front of them. Seeing “bright moon” may lead to associations with “hometown,” “yearning,” and “night”; seeing “lonely boat” may lead to associations with “drifting,” “river water,” and “parting.” These connections are not necessarily generated through explicit reasoning, but are gradually formed in the process of long-term contact with language, and are naturally mobilized when encountering related expressions.

And these connections will further affect our judgment of subsequent content. For example, when people familiar with Tang poetry see “A bosom friend afar,” they easily think of the next line, “Brings a distant land near.” Here, the preceding words have already provided a strong hint for the subsequent content.The more one reads and the more familiar one is with this way of expression, the easier it is to judge what might appear next based on the information that has already appeared when facing new content.

At this point, we can see that the so-called “ability to chant” is not simply bringing out what was read in the past, but being able to use the language experience accumulated in the past,starting from the expression in front of us, to establish associations between content, and based on this, judge and generate subsequent content.

And the foundation of all this is that the large amount of content contacted in the past did not simply remain in memory as learning ended, but instead gradually transformed into an experience that can continue to participate in language processing.

4 How Does AI Generate Answers Step by Step?

If we apply this process to AI, the question becomes: How does AI use the vast amount of language information it has been exposed to in the past to generate new expressions it has never seen before?

If we think about it carefully, when we usually answer a question, we don't simply find a ready-made passage from our memory. For example, if someone asks us, “Why is the sky blue?”, we retrieve relevant content from the knowledge we have already mastered, determine the direction of the answer, and then organize our own expression.The question itself provides context, and the context in turn affects what we should say next.

So, when we hand a question over to AI, how does this process happen?

Glossary of Terms

AI

AI is a broader concept. The chatbots and AI assistants we usually use can all be seen as AI applications. For users, we are directly talking to “AI”.

Large Language Model

Chatbots can generate text answers based on the questions we input because a large language model is running behind them.What users see is the AI application, while what is actually responsible for predicting and generating subsequent text based on context is the large language model within it. The “prediction” and “generation” discussed next in this chapter refer specifically to this process.

For example, if we ask AI, “Why is the sky blue?”, the AI might start its answer with “After sunlight enters the atmosphere, scattering occurs.” At this point, this sentence becomes the content that has already been generated.

Next, the model will continue to predict what content is more likely to appear next based on the current context, including this sentence. Thus, “sunlight contains light of different colors” may appear next, followed by “light of different colors has different wavelengths”, and then “blue light is more easily scattered than light of other colors” may continue to appear.

This process can be simply represented as:

As we can see, AI does not retrieve answers item by item along a pre-written path; instead, after each step of generation, it continues to predict the next step based on all the content that has appeared so far.

Ultimately, what we see is a complete answer centered around this question,but this complete answer does not exist from the very beginning; rather, it is gradually formed during such a generation process.

From this perspective, the most basic way a large language model works can actually be summarized in one sentence:Predict the content most likely to appear next based on the content that has already appeared.

Of course, this is just one possible way to unfold. If the answer chooses a different starting point at the beginning, the subsequent content may also unfold in different directions.That is to say, the same question does not necessarily correspond to a fixed answer path.

5 On What Basis Can AI Predict the Next Step?

In the previous chapter, we saw that the basic process of a large language model generating answers can be summarized as:Predict the content most likely to appear next based on the content that has already appeared. But the question follows: How on earth does AI know what content is “most likely” to appear?

The “most likely” here is not a vague statement.Mathematically speaking, the prediction of a large language model is essentially a probability judgment: faced with the content that has already appeared, the model will judge the likelihood of different subsequent content appearing and choose the more likely result.

Let's look at a simple example first. Suppose we see:“The sun from the east...”, we can easily think of “rises”. Of course, it is not that only this word can appear next; it could also be “appears”, “moves”, or even other content. However, in common language expressions, “rises” is obviously more natural, which is an experience we have formed after long-term exposure to and use of language.

When faced with similar situations, large language models also need to determine what to generate next from multiple possible subsequent contents. For example:

Therefore, the “prediction” here is not simply judging whether a word is “correct” or not, but calculating the likelihood of different subsequent content appearing based on the current context, and deciding what content to generate next accordingly.

So, on what basis does the model calculate these probabilities? The answer is:TrainingDuring the training process, the model is exposed to a large number of language examples. For example, it may repeatedly see expressions like “The sun rises from the east.” As training continues, the internal parameters of the model will be continuously adjusted, and different words, expressions, and the relationships between them will gradually form stable associations within the model.

Therefore, when the model sees “The sun from the east...” again in the future, it does not need to temporarily look for a complete standard answer; instead, it can use the parameter relationships formed during the training process to calculate different subsequent content.In the current context, the reason why “rises” is more easily predicted is that these parameter relationships give it a higher probability.

Moreover, the model does not need to have seen the exact same sentence in the training data to make predictions. As long as the language relationships formed during the training process can apply to the new context, it can face specific expressions that have never appeared in the past. This is also why large language models can generate a large number of sentences that have never directly appeared in the training data:What it learns is not just what content has appeared in the past, but also the complex relationships between this content, and these relationships will further affect new predictions.

What needs to be emphasized is that,the content most likely to appear in terms of probability does not mean it is definitely correctThe model's predictions can highly match the context, and the generated content may also look very reasonable, but this does not mean that the facts therein are definitely accurate or that the logic definitely holds.Because the model first solves “what content is most likely to appear next”, rather than directly judging “what is true or correct”, which is also an important entry point for understanding the limitations of large language models.

6 How Does the Model Learn to Predict?

In the previous chapter, we saw that the reason why large language models can predict the content most likely to appear next is that during the training process, a large number of complex relationships are formed among the internal parameters of the model.

But these relationships are not something the model has from the very beginning.The model does not know at first what content is more likely to appear next; it needs to form this prediction ability bit by bit through training. So, how exactly does training happen?

Let's use the previous example again. Suppose such a sentence appears in the training data:“The sun rises from the east.” After seeing “The sun from the east”, the model needs to predict the next content most likely to appear.

At first, its prediction may not be accurate. For example, it may think that “sets” also has a certain possibility, or it may even give results that are completely out of context. At this time, training can continue:

The most critical part here is that,The model does not simply memorize the answer because it “saw the correct answer”; rather, through a process of repeated prediction and correction, it continuously adjusts its internal parameters based on the difference between the predicted and actual results, thereby changing the probability of different content appearing in subsequent predictions and making the predicted results gradually approach the actual results.

If parameters are simply understood as a large number of adjustable values within the model, they can be seen as the “internal state” gradually formed by the model during the training process.After a large amount of training, these parameters gradually form various linguistic relationships that the model has learned from the training data, further influencing its predictions when facing new content.

In other words, training is not simply saving a large amount of text, but rather letting the model repeatedly experiencethe prediction, comparison, and adjustmentprocess.Training data provides instances of learning time after time, while parameters are the ultimate results left behind by these learning processes.

As we discussed earlier, humans also gradually develop language abilities through continuous exposure, use, and correction. For example, when a child first starts to speak, they do not know which words should go together, nor do they know how a sentence should be structured; but as they are exposed to more and more, they gradually become familiar with the relationships between these expressions.

Although the training of large language models is completely different from the human learning process, both ultimately reflect a noteworthy phenomenon:What is formed during the learning process does not cease to function when learning ends, but continues to influence how new situations are handled afterward.

7 How Does the Model Utilize Context?

Earlier, we discussed how the model gradually “learns” to predict through training, but learning to predict is only the first step. When a new question enters the model, it also needs to combine the context to determine which information the current content should focus on, and use this information to decide what to predict next. So, how exactly does the model achieve this?

For example: “Xiao Ming put the book on the table because it was very old.” When we see “it,” we do not just look at the word itself, but combine it with the preceding content to determine what it refers to—the relationship between “it” and “book” is obviously more important than the relationship between “it” and “Xiao Ming.” If we change it to “because it was very sturdy,” our understanding of “it” might change again.

This shows that when understanding a sentence, we do not process each word in isolation, but rather combine it with the context to determinewhich content is more relevant to the current content.This relevance in turn affects our understanding of the current content, thereby influencing our prediction of subsequent content. Large language models also need to handle similar issues when dealing with language.

Taking the previous example again: “Why is the sky blue?” The model cannot just process “why,” “sky,” and “blue” separately, but needs to combine this content and the relationships between them to form an overall representation of the current question.When the model begins to generate a response, this already-processed contextual information will continue to influence subsequent predictions. For example, after generating “sunlight,” the model still needs to combine the preceding information such as “sky” and “blue” to continue predicting what might appear next, such as “atmosphere” and “scattering.”

At this point, a question is presented to the model:If different content in a sentence influences each other, how should the model handle the relationships between them? Transformer provides a model structure for handling such problems, enabling the model to connect information at different positions when processing input and allow these relationships to participate in subsequent calculations.

Among them, a very key mechanism is called Attention (attention mechanism).If we understand it in an intuitive way, we can think of it as:When processing a certain piece of content, the model will judge which content in the context is more relevant based on the current situation, and let this content have a greater impact on the current processing.

For example, when processing “it,” “book” may be more worthy of attention than “Xiao Ming”; when processing “blue,” “sky” may be more relevant than other content.These reinforced contextual relationships will further affect the model's representation of the current content and ultimately participate in subsequent probability predictions. It is precisely in this way that the model allows different content in the context to jointly participate in the next step of generation.

8 Returning to the Original Question

Earlier, we pursued several seemingly simple questions: Why can AI generate a response that did not previously exist based on a question? Why, after a large amount of training, can it handle new questions that did not directly appear in the training data? And why can a complete response unfold step-by-step around the current question?

At this point, these questions can actually be understood within the same chain: large language models are trained on vast amounts of language data, changing their internal parameters through continuous prediction, comparison, and adjustment; after training is complete, when faced with new input, they combine different elements in the context and their relationships to predict the most likely next content; each generated piece of content then becomes part of the subsequent prediction, ultimately forming the complete response we see.

At this point, let's look back at the question raised in Chapter 1: Why does AI give the impression that it “understands”?

The reason is: when dealing with human language, it does not simply treat the input as a set of isolated words, but is able to combine context to make coherent, contextually appropriate responses. And when this ability is continuously applied to a conversation, this performance easily makes us think of “understanding” and “thinking.”

But if we peel back the layers of the process behind this performance, we will find that it is ultimately still built on prediction after prediction.And this is precisely a key perspective for understanding large language models: do not just look at what it appears to be, but ask how it actually achieves it.