1 Introduction
In the previous article, I sorted out the core concepts and process of RAG (see article:Home Data Center Series: Understanding RAG from Scratch (Part 1): Principles and Complete Process Analysis), but that was still at the theoretical level, just talking on paper. To get RAG actually running, we must take the first step of practice. The question is: where is the most suitable place to start?
I think the breakthrough point should be “embedding models”, because in the RAG process, the retrieval effect highly depends on how the text is converted into vectors, and this step determines whether the relevant content can be recalled later.
Speaking of embedding models, many people's first reaction might be to directly call cloud service APIs, such as:OpenAI's text-embedding-3-small and text-embedding-3-large, which are the two models most widely used in the community (Microsoft Azure actually hosts the same series as well); Anthropic, Cohere, and even some domestic vendors (such as Baidu's bge series and Zhipu's embedding-2) also provide similar embedding APIs. Indeed, doing so is worry-free, as there is no need to configure the environment and you can use them directly, but the problems are also obvious:
- Cost constraints: As the data scale increases, API call fees will continue to accumulate.
- Privacy concerns: Corpus needs to be transmitted to third-party servers, which may not be suitable for internal or sensitive data.
- External dependency: Once the service adjusts prices, limits rate, or even goes offline, your system will be affected.
- Difficult to compare and replace: Switching between different providers is not easy, and the cost of experimentation is high.
For these reasons, I prefer tobuild my own embedding models, so that I can not only fully control the data and environment, but also flexibly switch models and freely compare effects, which is more suitable for learning and exploration.
In self-built solutions,Ollama(For those unfamiliar with Ollama, you can refer to my other article:Home Data Center Series: Building a Private AI: A Detailed Tutorial on Building Open-Source Large Language Models Locally Based on Ollama) is a great starting point: it is specifically designed for local deployment, encapsulating the entire process of “download-load-call” very concisely. With just a few commands, you can run an embedding model on your computer without messing with underlying inference frameworks or worrying about compatibility issues, making it perfect for readers who want to get started quickly.
However, before practicing ”building a self-built embedding model with Ollama”, we first need to understand what an embedding model is.
2 From Text to Vector: Understanding Embedding Models
1 What is an Embedding Model?
To explain it in a very simple sentence:An embedding model is a model that converts text into a string of numerical vectors, does that sound a bit abstract?
Let's take an example. Suppose you have a sentence—“Apple is a kind of fruit”. When you hand it over to the embedding model, it will return a set of numbers, which may have hundreds or even thousands of dimensions. For example:
[0.12, -0.08, 0.95, ... , 0.34]
Regarding ”dimensions”, it can be understood like this: if there are only 2 numbers, it is like marking a point on a two-dimensional plane (X, Y coordinates); if there are 3 numbers, it is a point in a three-dimensional space (X, Y, Z); and when the numbers become hundreds or thousands, it means this sentence is placed in a “high-dimensional semantic space”. It is hard for us humans to imagine what 1000 dimensions look like, but for computers, this is just having many more coordinate axes, and at the same time,The more dimensions there are, the more nuanced the representation.
Just like describing a person, if you only say “height and weight”, that is a two-dimensional feature; if you add “age, occupation, and hobbies”, the dimension is higher and the description is more accurate. The same goes for embedding models, which use hundreds or thousands of dimensions to depict text, allowing semantic details to be captured more comprehensively.
They have no intuitive meaning on their own, but they determine “where this sentence is located in the semantic space.” In other words, an embedding model is like a translator, translating natural language that we can read into “semantic coordinates” that computers can operate on.
With this representation, computers can determine the similarity between two pieces of text by comparing the distance between their coordinates. For example:
- “Apple is a fruit” and “Banana is a fruit” are very close in distance in the embedding space;
- “Apple is a fruit” and “Apple Inc. released a new phone”, although both contain “Apple”, are relatively far apart in position.
This is the magic of embedding models: they no longer rely on simple keyword matching, but can capture closeness at the “semantic level.”
In RAG, the role that the embedding model plays is:Translating your question and the documents in the knowledge base into the same semantic space, and then completing the retrieval based on “who is closer.”。
Of course, why do we need this approach instead of directly using traditional search? This involves the limitations of keyword search and the advantages of semantic vectors.
2 Why Do We Need Embedding Models?
2.1 Limitations of Keywords
In the previous section, we mentioned that embedding models can convert text into numerical vectors, and we also know that the higher the dimension of the vector, the more nuanced the semantic expression. So here comes the question: why do we need this vector representation instead of directly using traditional keyword search? This is because keyword search has its limitations; simply put, keyword search only looks at the literal words and does not understand the “meaning.” It has several obvious problems:
- Mismatching: For example, if you search for “nutritional value of apples”, and the document says “Apple Inc. releases a new product”, the keyword “Apple” is matched, but the semantics are completely unrelated.
- Insufficient recall: If you search for “benefits of bananas”, and the document says “bananas are rich in potassium”, the keyword matching might not be precise enough, ranking it lower or even failing to retrieve it.
2.2 Advantages of Semantic Vectors
Embedding models map text to a high-dimensional space and measure similarity using “semantic distance.” In other words, they do not look at the literal words, but rather whether the “meanings are close.” To illustrate with an example:
- Text A: “Apple is a fruit”
- Text B: “Banana is a fruit”
- Text C: “Apple Inc. released a new phone”
In the vector space, the vector distance between A and B is very close because they both describe the semantics of fruit; the distance between A and C is relatively far, even though they both contain the keyword “Apple”.
In this way, when searching for “nutritional value of apples”, the system can prioritize retrieving truly relevant fruit documents rather than news about Apple Inc.
Note: The concept of vectors is the universal language of the AI world and is very important. Interested friends can refer to my other article:Home Data Center Series - Vectors: The Universal Language in the AI World。
2.3 The Value of Embedding Models in the RAG Process
In the workflow of RAG (Retrieval-Augmented Generation), the role of the embedding model is crucial: it does not just convert text into vectors, but also determines what information the model can “see” during the retrieval phase, directly affecting the quality of the generated results. Imagine if you ask the model “What is the nutritional value of apples?”, and the knowledge base contains both documents like “apples are rich in vitamin C” and news like “Apple Inc. releases a new phone.” If the embedding model is accurate, the documents retrieved by the model will mainly be the former, and the generated answer will naturally be correct; if the embedding model is not good enough, the model might use irrelevant news as a reference, and the generated answer can easily go off track.
At the same time, embedding models also improve the controllability of RAG: they clearly divide the semantic space, allowing the generation model to only reference relevant content and reducing interference from irrelevant information. For beginners, it can be understood as “helping the model filter out noise,” making the answers more reliable and predictable.
In addition, embedding vectors have strong interpretability: you can quantify the relevance between questions and documents, and view the top-5 most relevant documents along with their similarity scores, so you can clearly know what content the model actually referenced. This is very helpful for experimentation and tuning: if you switch to a different embedding model, the most relevant documents might be completely different, and the generated answers will naturally differ as well, making performance differences observable and analyzable.
Finally, the value of embedding models also lies in their scalability: vectors can be stored in databases and used in various scenarios such as Q&A, recommendation, or similarity search. Once a reliable embedding system is established, subsequently adding new documents or switching generation models can smoothly extend the closed loop of retrieval and generation.
Overall, for beginners, getting the embedding model sorted out first is equivalent to laying a solid foundation for RAG. As long as the foundation is stable, no matter how you adjust chunking strategies, retrieval parameters, or generation models later on, you can achieve more controllable and intuitive results. This is precisely why I chose the embedding model as the first step in RAG practice.
3 Practical Guide to Deploying Embedding Models with Ollama
1 Preparing the Ollama Environment
This part of the operation varies depending on the system platform everyone uses. For example, I am using the M4 Pro Mac mini. The detailed steps for downloading, deploying Ollama, and using large language models can be found in my previous article:Home Data Center Series: Building a Private AI: A Detailed Tutorial on Building Open-Source Large Language Models Locally Based on Ollama, which I will not repeat here.
However, the macOS version of Ollama now allows you to directly set the model download location in the GUI settings and directly enable the ”cross-origin access” feature, which makes it much more convenient:
As for the cross-origin settings of other Ollama versions, you can refer to my other article:Deploying Llama 3.2 on Mac mini (M4 Pro) in the Home Data Center Series: A Complete Guide to Efficient Operation and Cross-Origin Access Optimization via Ollama。
2 Choosing an Embedding Model Based on Your Needs
Among the models supported by Ollama (see:https://ollama.com/library), there are several mainstream embedding models to choose from, each differing in accuracy, scale, and resource consumption. To make it easier to understand, I have classified them into three categories: small, medium, and large:
| Scale Type | Model Name | Model Size | Suitable Scenarios |
|---|---|---|---|
| Small (Small) | nomic-embed-text | 4M | Blog posts, personal notes, small knowledge bases, fast deployment, low resource consumption |
| Small (Small) | snowflake-arctic-embed 22M/33M | 22M / 33M | Small text libraries, rapid experimentation, low hardware requirements |
| Small (Small) | granite-embedding 30M | 30M | Single-machine small-scale text vectorization, can run on CPU |
| Medium (Medium) | snowflake-arctic-embed 110M/137M/335M | 110M / 137M / 335M | Enterprise documents, cross-departmental knowledge bases, hundreds of thousands of documents, requires high semantic precision |
| Medium (Medium) | granite-embedding 278M | 278M | Medium-scale enterprise documents, multi-document semantic retrieval, requires GPU or high CPU memory |
| Large | mxbai-embed-large | 335M | Large-scale enterprise knowledge bases, complex Q&A systems, multilingual scenarios, high-VRAM GPU recommended |
| Large | snowflake-arctic-embed2 | 568M | Multilingual knowledge bases, large-scale vector databases, high precision requirements, requires high-performance servers |
Small Models
They are usually small in size, such as nomic-embed-text (36M), the 22M or 33M versions of snowflake-arctic-embed, and the 30M version of granite-embedding. Their advantages are being lightweight and fast, running on almost any modern personal computer. For blog posts, personal notes, or small-scale knowledge bases, these models are completely sufficient. The deployment cost is low, they can run on CPU, and if a GPU is available, the speed will be even faster. Memory usage is also very low, generally 8–16GB is enough. If you want to vectorize your blog .md files, these small models are the most effortless and practical choice.
Medium Models
They are slightly larger, such as the 110M, 137M, and 335M versions of snowflake-arctic-embed, or the 278M version of granite-embedding. They are suitable for handling enterprise-level documents or cross-departmental knowledge bases, maintaining high semantic precision even when the document volume reaches hundreds of thousands. These models have slightly higher hardware requirements: preferably a GPU with 4–12GB of VRAM; they can run on CPU, but the speed will be slower, and the memory requirement is about 16–32GB. Medium models offer a good balance between precision and resource consumption, making them the preferred choice for formal business scenarios.
Large Models
Aimed at large-scale enterprise applications, such as mxbai-embed-large (335M) or snowflake-arctic-embed2 (568M, with multilingual support). They can handle document libraries of over a million documents, have extremely strong semantic understanding capabilities, and are suitable for complex Q&A systems or multilingual retrieval scenarios. However, the hardware requirements are also much higher: a high-performance GPU with over 16GB of VRAM, at least 32GB of memory, and the vector storage space also needs to be expanded accordingly. For general personal blog posts, using a large model is not only a waste of resources but also comes with high deployment costs and slow running speeds.
For most personal or small-scale text libraries, especially for vectorizing blog post .md files,small models are already sufficientThey are lightweight, fast, low-cost, and allow you to complete the entire RAG workflow experience locally. Only in scenarios involving large-scale enterprise documents or extremely high requirements for semantic precision do you need to consider medium or large models.
Based on the above analysis, my requirement is to vectorize local blog posts in ”.md” format, and the most suitable embedding model is ”nomic-embed-text”.
3 Pulling a Suitable Embedding Model in Ollama
According to the analysis in the previous section, the most suitable embedding model for my scenario is ”nomic-embed-text”, so next I need to pull this model in ollama:
ollama pull nomic-embed-text
If successful, the result is as follows:
You can use the following command to view the model's information:
ollama show nomic-embed-text
4 Testing the Availability of the Embedding Model
Use the following command to test via API whether the ”nomic-embed-text” embedding model can normally vectorize the sentence ”The sky is blue because of Rayleigh scattering”:
curl http://localhost:11434/api/embeddings \
-d '{"model": "nomic-embed-text", "prompt": "The sky is blue because of Rayleigh scattering"}'
If successful, you will get a set of vector outputs:
Note: If you use other embedding models, you need to replace the vector model in the above command with your actual model name.
5 Tips: How Ollama Works
In Docker, the familiar workflow is usually: first use docker pull to pull an image, and then use docker run to start a container. If the run step is not executed, the image just sits quietly on the local disk and cannot be used at all. This logic easily leads to a mindset: only run is the key action that truly makes things “come alive”.
While Ollama's command names look very similar, it has a completely different running mechanism. Although it also provides the “ollama pull” command to download models and the “ollama run” command, which on the surface seems very similar to Docker's run, the essence is completely different: Ollama's core is actually a resident service that runs in the background on the local machine after installation (listening by default on http://127.0.0.1:11434). As long as this service is running, local models can be called via API at any time without being manually started via the command line. Therefore, the ”ollama run” command is more of an experiential, interactive way.
For example, when you execute “ollama run llama3” in the terminal, it essentially just sends a request to the background service for you and prints the result in the command line, making it easy for people to try it out directly. For generative models, this is indeed an intuitive way to experience them. However, for embedding models, they do not support text generation at all, so running the “ollama run nomic-embed-text” command will result in an error:
In practical applications, whether it is a generative model or an embedding model, they are called via APIs in formal environments. Generative models output text via POST /api/generate, and embedding models return vectors via POST /api/embeddings. These APIs are Ollama's delivery interfaces and the way that should be relied upon in production environments. Therefore, unlike Docker, Ollama's run is not a mandatory switch; it is just a small tool for users to experience models in the command line.
With this mechanism, many friends might worry: if Ollama can be called without run, does that mean all models are running in the background all the time? Actually, that is not the case.
Ollama's models are just files stored on the disk locally, and they are only loaded into memory or GPU when an API call is made. After the call ends, the model is temporarily kept in memory as a cache for quick response next time; but if resources are tight or the service restarts, these models will also be unloaded to release system resources. In other words, models do not occupy computing power endlessly, but are started on demand and released on demand. This mechanism ensures both the immediacy of calls and avoids resource waste.
4 Creating a Knowledge Base in Chatbox
1 Adding Ollama's Embedding Model in Chatbox
After completing the deployment of the local Ollama embedding model and testing its availability, the next step is to connect it to Chatbox (friends who are not familiar with Chatbox can refer to my other article:Home Data Center Series - The Most Convenient AI App Frontend: Chatbox Comprehensive Introduction + User Guide), as a callable model provider.
First, open Chatbox's model management interface, find “Add Model Provider” or a similar entry, select Ollama as the provider here, and fill in the local service address, which is usually http://localhost:11434, next, click the ”Get” button to refresh, and select the embedding model you downloaded earlier from the list of available models, such as “nomic-embed-text” to add, as shown below:
Then, select to set the model type to embedding, and click the settings button to the right of ”nomic-embed-text”:
“Select ”Embedding” from the ”Model Type" dropdown menu and save:
Success:
2 Creating a Knowledge Base and Importing Markdown Documents
Knowledge base creation completed:
After successfully importing all .md format articles:
Note: The chunking in the red box in the image above is to cut a long article into small segments, just like tearing a book into individual pages. The reason for doing this is that the embedding model can only process text of a limited length at a time. If the entire article is not cut, the model cannot fit it all and is also prone to missing information. After chunking, each small segment will be converted into a vector and stored in the knowledge base. Later, when a user asks a question, the system can quickly find the segments most relevant to the question, instead of rummaging through the entire article.
5 Using the Knowledge Base in Chatbox
Then, construct a reasonable question, prompting the AI to refer to the content in the knowledge base to make a judgment, for example: ”Refer to my blog posts in the knowledge base, compare my mastery of different technical fields, and point out my strengths and areas for improvement.”, and the results are as follows:
It works extremely well. Previously, my biggest headache was asking ChatGPT to analyze the pros and cons of some of my articles. One article was fine, but analyzing a series of articles was a real hassle. Now, with the knowledge base and all blog posts imported, this requirement has become very simple.
In addition, my question this time only cost a total of 4,345 tokens:
Compared to the hundreds of thousands of tokens that might be required to directly input all articles into the model at once, this embedding-vector-based retrieval-augmented approachsaves hundreds of times more tokens, which not only drastically reduces costs but also ensures that the model's answers focus only on the most relevant content, making the evaluation both efficient and precise.
Why does it have this effect? This is because in the RAG workflow, the model does not stuff the entire knowledge base into the context. Instead, it first retrieves through embedding vectors to quickly lock onto several document segments most relevant to your question, and then appends these segments to the prompt for the model to process. In this way, irrelevant content does not consume tokens at all, the information “seen” by the model is cleaner, the reasoning space is more focused, and the results are naturally both cost-effective and reliable.
In previous articles, I mentioned that the RAG workflow can be divided into five core steps:Text Chunking → Vectorization → Vector Storage → Retrieval → Generate Answers. For the knowledge base feature provided by Chatbox, it actually helps us solve thetext chunking, vector storage, and retrievalsteps, allowing documents to be neatly stored and the most relevant content to be quickly found. Meanwhile,the second step, “vectorization,” still requires us to provide an embedding model ourselvesto complete.
Note: LobeChat's knowledge base and Chatbox's knowledge base are similar in their core functions; both chunk documents, store them as vectors, and retrieve the most relevant content when a user asks a question. They differ slightly in how they handle embedding models: LobeChat has a built-in embedding model by default, which automatically generates vectors after documents are uploaded without user intervention; Chatbox, on the other hand, can use an embedding model provided by the user (such as Ollama's local model), theoretically allowing control over the version and quality of the vectorized model. However, in practice, apart from choosing the model and uploading documents, adjustable options like chunk size, overlap parameters, and vector storage methods remain limited. In other words, LobeChat leans towards out-of-the-box usability, while Chatbox offers the possibility of model selection, but is not much stronger than a self-built LobeChat server in terms of customization and controllability.
It should be noted that LobeChat's knowledge base has a certain barrier to deployment: to fully use its features, you must enable a server-side database and additionally configure an S3-compatible object storage (such as MinIO, COS, OSS), while also handling domain name and cross-origin access settings. This makes LobeChat's knowledge base seem rather cumbersome for personal or lightweight scenarios. Chatbox, on the other hand, is more lightweight and particularly suitable for personal knowledge bases that only need occasional updates, with the only typical hassle being the need to provide your own embedding model.
6 Afterword
After this series of operations, I have finally run through a complete, minimum viable RAG system: from deploying the embedding model with Ollama, to creating a knowledge base in Chatbox, and then to uploading blog posts and conducting retrieval-augmented conversations. Although the entire process seems step-by-step, the core logic running through it is very clear:First make the embedding model stable and controllable, then build the knowledge base, and finally generate through the large language model.This is the first bridge from theory to practice, and also the most “visible” breakthrough point in the RAG workflow.
In practice, I have also discovered some interesting insights. For example, many readers and beginners might think they must first build a vector database to use a knowledge base. In fact, solutions like Chatbox, which come with built-in SQLite, are completely sufficient for data volumes at the personal blog level. As long as the embedding model is working properly, the system can immediately generate vectors after documents are uploaded, and they can be retrieved. This is very friendly for rapid validation and iteration, and greatly lowers the barrier to the entire RAG practice.
Another realization is that many people (including myself), influenced by the “docker run” mindset, easily mistake that the embedding model must be started in the command line to be called. In fact, Ollama's embedding model is essentially a service that can be called across domains by the frontend via an API; ”ollama run” merely provides a local interactive experience and is not required. This design makes actual deployment more flexible and facilitates using the same embedding model across multiple terminals and frontends.
Finally, I want to emphasize one point: although this article has run through the minimum viable workflow, the potential of RAG goes far beyond this. In the future, you can try optimizing chunking strategies, introducing larger-scale vector databases, adjusting retrieval parameters, or even combining multiple models to enhance retrieval capabilities. What is shown today is the first step in bringing theory to fruition—a “minimum viable loop”—and the subsequent iterations and optimizations are entirely in your hands.
Note: Starting from some version of Chatbox, its built-in document parser, when processing longer Chinese articles, may generate text structures that do not meet the input expectations of the embedding model, thereby causing the locally deployed Ollama embedding model (such as nomic-embed-text) to frequently trigger "the input length exceeds the context length" error when adding new knowledge base articles.
It is worth noting that when directly calling the same embedding model via the Ollama API and passing equivalent or even longer text content, the embedding request can be completed normally. This indicates that the problem does not lie with the embedding model itself, nor is it entirely caused by the operating environment (macOS / Linux) or hardware computing power; rather, it is more likely to stem from anomalies in the intermediate results introduced by Chatbox's built-in parser during the document parsing, text splicing, or chunking stages.
Since the specific parsing logic, chunking strategy, and intermediate text of this parser are neither observable nor configurable, under the premise of using the built-in parser, this issue is currently difficult to avoid by changing the embedding model or running platform. Therefore, at least under current conditions, there is a clear compatibility boundary between the Chatbox knowledge base feature and the local Ollama embedding model when processing long Chinese documents.
In plain English: until Chatbox officially fixes or reconstructs its built-in document parser, its knowledge base feature is not suitable for direct use with a local Ollama embedding model to process longer Chinese articles. Under this combination, the problem is not something that can be solved by “tweaking it a bit more,” but is limited by the implementation of the parsing pipeline itself.