1 Introduction

In a previous article (see: Home Data Center Series: Understanding RAG from Scratch (Part 1): Principles and Complete Process Analysis), I introduced the theoretical five-step process of RAG: "Segmentation → Vectorization → Vector Storage → Retrieval → Answer Generation"; subsequently, I based my self-built embedding model on Ollam. nomic-embed-textA knowledge base was implemented on Chatbox, which can be considered the simplest RAG practice (see article:Home Data Center Series: Using Ollama's Self-Built Embedding Model + Chatbox Knowledge Base Practice).

However, the Chatbox knowledge base implementation essentially delegates the three steps of "segmentation, vector storage, and retrieval" to Chatbox, while the answer generation is handled by a user-specified large language model (in this article, I used gpt-5-mini). For everyday personal use, this approach is sufficient, but if you want to create a... Completely free blog chatbotThis method won't work.

To truly achieve Fully local, free blog chatbotThis requires building a complete local RAG workflow from scratch. Theoretically, this can be achieved using existing frameworks, such as... LangChain or LlamaIndex To achieve this. However, in order to familiarize myself with the entire RAG process first, I decided to create a set first. Minimal Demo Based on PythonThis approach uses transformers to call Hugging Face's embedding and LLM models: it doesn't pursue complex functionality or performance optimization, but only aims to complete the entire closed loop from "document → embedding → retrieval → calling the large model to generate answers". Compared to Ollam or other inference platforms, this approach is more lightweight, controllable, and fully localized, making it ideal for beginners to understand each step of the RAG process.

Through this demo, I hope to truly... Understanding the internal logic and data flow of each stepThis lays the foundation for building a formal blog chatbot in PVE LXC later. In other words, this article is a...Step-by-step demonstration of the minimal local RAG processThe learning case studies can help you become familiar with the RAG process before considering using the framework or deploying it to a production environment.

Note: The simplified demo aims to be "lightweight, controllable, and fully local," suitable for learning the RAG workflow; while Ollam or other inference platforms are geared towards...Official deploymentIt supports larger models, higher performance, and online services. During the learning phase, you can start with this minimal demo to understand the process and then gradually upgrade to the production environment.

2. Preparations

2.1 Preparing the Hardware Environment

To run a minimal local RAG workflow, the hardware requirements are actually not high. My practical environment this time is... M4 Pro Mac mini (24GB memory) The configuration is sufficient to run Hugging Face's small models smoothly (e.g., sentence-transformers/all-MiniLM-L6-v2 as an embedding model, and Jackrong/llama-3.2-3B-Chinese-Elite-v2 as a large model).

Of course, you don't necessarily need a Mac mini like mine. As long as your computer meets the following conditions, it should be able to run successfully:

• CPU supports AVX2 instruction set(Most recent Intel/AMD processors work without issue; Apple chips are also natively supported under macOS).
• At least 16GB of RAM(Running a 3B-scale model is more stable; if you want to run an 8B model, 24GB+ is recommended.)
• Disk space of 15GB or more(Primarily used to store Hugging Face model files and vector library data).
• Internet access(The first download of the model requires a network connection. If the download is interrupted, you can also manually download the model file and put it in the local cache directory; subsequent inference will run entirely locally.)

In other words, as long as you can install the Hugging Face dependency library on your device and run a model of an appropriate size, you can complete the minimum RAG Demo shown in this article.

2.2 Preparing the Software Environment

2.2.1 Python environment

To run the minimal RAG Demo, you first need to prepare a suitable Python environment.

The Python that comes with macOS is usually... 3.9:

This version is too old and incompatible with the dependencies we will need later, so we need to install a separate version. Version 3.11 or abovePython. On macOS, there are two main ways:

Method 1: Install via Homebrew (Recommended)

This is the most common and recommended method for macOS users because Homebrew is well isolated from the system, does not affect the built-in Python, and is convenient for future upgrades or uninstallation.

If you only want stability, I suggest you install it directly. Python 3.11:

brew install [email protected]

After successful installation, the result will look something like the following:

You can also confirm using the following command:

python3.11 --version

If you'd like to try it out, you can also directly install the latest release version provided by Homebrew (currently 3.14):

brew install python python3 --version

Note: Although the latest release of Python is 3.14, some dependencies may not be fully compatible. To minimize compatibility issues, installing version 3.11 is recommended as a safer choice.

Method 2: Use the Python official website installation package (pkg)

Python official website (https://www.python.org/downloads/It also provides an installer for macOS (.pkg format), and the installation method is very intuitive: simply download and double-click to complete the installation.

This method installs Python to the system path (/Library/Frameworks), which is less elegant than Homebrew and may conflict with the system's built-in version.

therefore,Homebrew is recommended for macOS users.;and Non-macOS users (such as Windows and Linux)If so, you can download the installation package for the corresponding platform from the official Python website.

2.2.2 pip and dependency installation

2.2.2.1 Confirm the pip version included with Python 3.11.

Once the Python environment is ready, the next step is to check if pip is available, as we need to install the dependencies required for our demo using pip. The older Python 3.9 that comes with macOS usually includes pip3, but it's an older version and not recommended. Newer Python installations via Homebrew or the official website will include a matching pip by default. For example, if you use `brew install [email protected]`, it will automatically install pip3.11, ensuring dependency compatibility and ease of use in virtual environments. You can check this by running the following command:

python3.11 -m pip --version

If the version number is output (e.g., pip 25.x from …), it means that pip is working correctly.

2.2.2.2 Manually install or upgrade pip as needed (optional)

If pip is missing, you can install it manually:

curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11

It is recommended to upgrade pip to the latest version to avoid errors during subsequent dependency installations.

python3.11 -m pip install --upgrade pip

If successful, enter the following:

2.2.2.3 Install Dependencies

To run the minimal RAG Demo, we need some basic packages:

python3.11 -m pip install -U faiss-cpu numpy scipy torch torchvision sentence-transformers transformers tqdm safetensors

illustrate:

• faiss-cpuA high-performance vector search library (CPU version, sufficient for Mac mini).
• numpyIt is a scientific computing library, and libraries such as Faiss depend on it.
• scipy: A common scientific computing library used for similarity calculations (such as cosine similarity).
• sentence-transformers: A common text vectorization toolkit used to convert segmented text blocks into vectors.
• transformers: A model library for Hugging Faces that can call Embedding models or simple LLMs.
• tqdmThe progress bar tool displays the progress in real time when batch processing segmented text blocks, making it more intuitive.

The result after installation is as follows:

This way, the Python environment has the minimum dependencies to run the RAG Demo.

Note: The dependencies installed here are only the most basic packages required to run the minimal RAG Demo. Further dependencies will be added later. LangChain or LlamaIndex Setting up a proper local RAG system will require even more dependencies and configurations.

3. Additional Knowledge: About Transformers

3.1 What are transformers?

In the field of Natural Language Processing (NLP),Transformers It has become one of the most core model architectures. Originally proposed by Vaswani et al. in 2017, its core idea is to...Self-attention mechanismThis allows the model to understand the relationships between different positions in text without relying on traditional recurrent or convolutional structures. This mechanism enables the model to efficiently capture contextual information in long texts, making it suitable for handling various language understanding and generation tasks.

In the Hugging Face ecosystem, transformers refer not only to this model architecture, but also to an open-source... Python librariesThe purpose of this library is:

1. Unified interface loading model:Whether it's GPT, BERT, RoBERTa, T5, or Mistral, you can use the same library to load, infer, and train them.

2. Facilitates the use of various pre-trained models:Hugging Facebook Hub offers thousands of open-source models covering tasks such as text generation, text embedding, classification, question answering, and translation. The model is automatically downloaded and cached locally the first time it's used, and can then be run directly in a Python script.

3. Flexible operation: Local or cloud-based:Inference can be performed directly on local CPU/GPU, or large models can be deployed in conjunction with cloud services or inference platforms. This means that even without Ollam or the OpenAI API, a complete RAG demo can still be run using transformers.

In short, transformers are RAG Demo's core engineIt's responsible for converting text into vectors (Embedding) and can also be used to generate responses (LLM), covering almost all the functionalities that might be used in the entire NLP process. In this article, I will mainly use it for processing... EmbeddingHowever, its capabilities extend far beyond this, paving the way for future expansion into more complex local RAGs or LangChain/LlamaIndex.

3.2 Available Model Types for Hugging Face

Within Hugging Face's transformer ecosystem, you can directly use various model types. Each type has a specific purpose; below is a summary of some of the most common model types in RAG or general NLP scenarios:

Model type	Typical uses	Example Model	illustrate
Embedding (Vectorization)	Text vectorization for similarity retrieval and RAG vector libraries	sentence-transformers/all-MiniLM-L6-v2, all-mpnet-base-v2	Its main function is to convert text or sentences into fixed-dimensional vectors, which facilitates vector retrieval by libraries such as FAISS.
LLM (Large Language Model)	Text generation, dialogue, summarization	gpt2、mistralai/Mistral-7B-v0.1、LLaMA	It can generate text locally for the "Generate Answers" step in RAG, or it can be used for standalone question-and-answer or creative writing.
Text classification	Sentiment analysis, topic classification, spam detection, etc.	distilbert-base-uncased-finetuned-sst-2-english	Output category probabilities to determine text attributes or labels.
Extractive QA	Extract specific answers from the paragraph.	deepset/roberta-base-squad2	Given a question and context, the model returns a text fragment as the answer.
translate	Interlingual translation	Helsinki-NLP/opus-mt-en-zh	It supports translation between multiple languages and can also be used for cross-language retrieval and other scenarios.
Multimodal	Speech recognition, image + text, audio classification	facebook/wav2vec2-base-960h, some CLIP models	Requires additional dependencies (such as torchaudio, datasets), and can handle non-text data.

---

💡 Supplementary Explanation

• The core of RAG DemoIn the minimal RAG Demo of this article, we mainly use Embedding Vectorization can be performed, and a small LLM can be selected for generation. Other types of models can serve as the basis for subsequent extensions or other NLP projects.
• It can run without Ollam.The models shown in this table can all be loaded and inferred locally using Transformers, without relying on Ollam or any online platform.

3.3 Model Loading Methods: Hub and Local

When using transformers or other Hugging Face models, there are two main ways to load the model: directly from... Hugging Face Hub Download, or use the one already saved to Local disk The model. It should be noted that these two methods may load the same model, only the source and loading method are different.

The Hub model is hosted on the official Hugging Face servers and can be downloaded directly using from_pretrained().An internet connection is required for the first load.Once downloaded, the model is automatically cached locally, and subsequent calls typically do not rely on the network. The advantage of Hub models is their ease of acquisition and updating, making them suitable for rapid experimentation, learning, or testing of new models without the need for manual file management; however, their disadvantage is...The first download and update depend on the network.If the Hugging Face service is temporarily unavailable, you may not be able to pull or upgrade the model.

A local model refers to a model file already saved on disk, which can be loaded by pointing to a local path. The advantages of using a local model are complete offline operation, high controllability, and greater stability and reliability for production deployments. The disadvantage is that model files need to be manually managed; if an upgrade is needed, they must be manually replaced or re-downloaded.

In summary, the main difference between the Hub model and the local model lies in... Convenience and controllabilityThe Hub model is convenient and fast, suitable for learning and rapid iteration; the local model is stable and controllable, suitable for production deployment or completely offline scenarios. Understanding the advantages and disadvantages of these two loading methods helps in choosing the most appropriate strategy at different stages and lays the foundation for the subsequent implementation of the RAG process.

3.4 Formal Deployment Recommendations

In a production environment, if the goal is a stable, high-performance blog chatbot, the choice of model loading method and inference platform becomes particularly critical. For devices with sufficient hardware (such as the M4 Pro Mac mini), this is especially important.The Ollama method is usually the optimal choice.Ollama is deeply optimized for Apple M-series GPUs, fully utilizing GPU computing power and supporting low-precision calculations (fp16/bf16) and efficient memory layout, allowing even medium to large models to run efficiently locally. Ollama is also optimized for concurrency performance, making it more suitable for production deployments.

On the other hand, using HF transformers local modeThis is entirely feasible. It relies on PyTorch's Metal API for acceleration on Mac GPUs, which can significantly improve inference speed for small models. However, large models may be limited by GPU memory, requiring batch inference or hybrid CPU-GPU operation. The advantage of the HF approach lies in its complete control and transparency, making it suitable for education, testing, or low-concurrency scenarios, allowing developers to deeply understand the internal logic of each step in the RAG process.

In summary, for Demo or learning stageThis can be achieved by directly using the HF local model, which can quickly run the entire process of "document → embedding → retrieval → large model to generate answer" and facilitates understanding of the operation of each step. Meanwhile... Formal deployment or scenarios requiring high performanceIf hardware allows, the Ollam method is not only more convenient but also offers the best performance, making it the more recommended solution; the HF local method can still be used, but its performance and concurrency support for large models are limited.

4. RAG Core Module Analysis and Sample Code

4.1 Five steps and three key roles in the minimum closed loop of a RAG

In the introduction, I mentioned the five-step process of RAG: "Segmentation → Vectorization → Vector Storage → Retrieval → Answer Generation". These five steps are the specific operational steps, but from a higher level, the entire system can be abstracted into three core roles:Text blocks, vector libraries, large models.

Here, "text chunks" do not refer to the original entire article, but rather to the smallest semantic units obtained after the original document has been segmented and vectorized—that is, the form in which knowledge is actually represented and retrieved in the system. In other words, the original document is transformed into text chunks through "segmentation" and "vectorization," and these text chunks are the objects that the system needs to store and retrieve. The vector library is responsible for storing the vectors of these text chunks and performing similarity retrieval; the large model is responsible for generating the final answer based on the retrieved context.

Therefore, the five steps form an "operational level" pipeline, while the three roles are participants at the "structural level." Once this hierarchical relationship is understood, the subsequent work becomes clear: first, process the original document into a form of appropriate granularity.text block(This is the first core concept), vectorizing text blocks and storing them in a vector library (the second core concept), and then using a large model combined with the search results to generate answers (the third core concept). To make it easier for readers to follow along, I will add category labels (e.g., ...) after the titles of subsequent subsections. 4.2 Preparing the Input Document (Text Block Category),4.3 Document Segmentation (Text Block Category) (etc.), so that the content of each section and its corresponding role will be clear at a glance.

Having understood these three core roles, the next step is to prepare the first core:Text block (text block category)In other words, you need to organize the content you want the RAG system to "remember" and transform it into units that the system can use through segmentation and vectorization. In the following chapters, I will guide you step by step to build this minimum closed loop.

4.2 Text Blocks

4.2.1 Preparing the original document

In the RAG process,text blockIt is the smallest unit of knowledge after it enters the system. But before segmenting, we must first prepare an original document as the starting point for subsequent processing.

For demonstration purposes, I've chosen an existing .md file as the source document. There are two reasons for choosing a Markdown file: firstly, it's a common knowledge management format, and many people use .md for blogging and note-taking; secondly, it's essentially a plain text file, making it very easy to read and process without involving complex format parsing.

It's important to note that the original document's format is not strictly required. The .md file is just one example; it could also be a .txt file, or even plain text scraped from a webpage. RAG's essence is text processing; as long as the content can be converted into a string, it can proceed to the next step.

In this minimal demo, we'll only use the simplest plain text files (such as .md and .txt) because Python can read them directly without relying on additional libraries. To handle complex formats like .docx and .pdf, you'll need additional parsing tools (such as python-docx and pymupdf), which are beyond the scope of this chapter.

Suppose we have a sample.md file in the current directory, the content of which may be an article or a set of notes.In the following chapters, we will first divide this document into text blocks, then vectorize them and store them in a vector library, gradually building a minimal closed-loop RAG system.

4.2.2 Document Segmentation

Once you have the original document, the first step is to... SlicingThe reason is simple: most documents are often very long, sometimes containing tens of thousands of words. If they are directly fed into a vectorized model, not only will the computational efficiency be low, but the semantic representation will also be too vague, making it difficult to hit the precise segment during retrieval.

Segmentation, in essence, is breaking a long document down into several smaller parts. text blockEach block should maintain semantic integrity as much as possible, but it shouldn't be too large; otherwise, it might still exceed the model's processing capacity when performing embeddings later. Typically, the length of a block is... Hundreds to thousands of charactersThe middle range is more suitable.

In the world of RAG, segmentation is a crucial step: too much segmentation can fragment the semantics; too much segmentation hinders retrieval and matching. Here, we won't pursue complex segmentation algorithms, but rather a minimal demo. The simplest approach is to split by paragraphs, for example, by segmenting upon encountering a newline character. This way, the logical structure is largely preserved while the document is divided into smaller chunks.

In the subsequent code implementation, we will write a short function to accomplish this task: read sample.md, segment it by paragraphs, and return a list of text blocks. This list will serve as the input for subsequent vectorization.

in other words,Segmentation is the "entry point of entry".“This determines the granularity at which knowledge enters the RAG system. Only with proper segmentation can the retrieval and answer generation results be satisfactory.

4.2.3 Vectorization

In the core role of "text blocks," segmentation is only the first step; it breaks long documents down into smaller fragments. But text blocks alone are not enough; machines cannot directly understand text. To give it "semantic awareness," we need to further transform these text blocks into... Vector representation.

Vectorization is the process of encoding natural language into a string of numbers—these numbers correspond to the semantic coordinates of the model in a high-dimensional space, which can be used to measure the similarity between different text blocks.

To give a simple example: humans see "apple" and "banana" and know they are both fruits; while "apple" and "laptop" are completely different in meaning, we can still understand that "apple" refers to the Apple Inc. in certain contexts. The purpose of vectorization is to allow machines to perceive this "semantic distance" in the digital world. Two text blocks with similar meanings will have similar vectors; conversely, the distance will be greater. (For a detailed introduction to vectors, please see the article: ...)

In practice, we can leverage the ready-made embedding models provided by Hugging Face, such as... sentence-transformers/all-MiniLM-L6-v2The advantages of this model are: it's compact, fast, and accurate enough, making it ideal for demos. Using the transformers library, we can easily convert text blocks into vectors.

At this point, the role of the "text block" has come to an end. We now have a set of vectorized "semantic numbers," but they are still just data scattered in memory. Next, we will introduce the second core role—Vector libraryOrganizing these vectors makes it easier to retrieve and generate answers.

4.3 Vector Library

4.3.1 The role of vector libraries

In the previous section, we converted text blocks into vectors. At first glance, these vectors are just a bunch of high-dimensional arrays of numbers, which are actually not very useful on their own. Their real value lies in the fact that we can put them into a... Specialized storage and retrieval toolsThat is, the vector library.

Why do we need vector libraries? Imagine this: if we have a few hundred text blocks, iterating through the array in Python to calculate "cosine similarity" is passable; but if the data volume becomes hundreds of thousands or even millions of blocks, comparing them one by one becomes extremely slow, almost unusable. The purpose of vector libraries is to provide... Efficient similarity searchThis allows us to quickly find the few blocks in the huge vector set that are closest to the query statement.

---

The passage above mentions the term "cosine similarity." What exactly does this term mean?

In simple terms, each text block, after being vectorized, becomes a point in a high-dimensional space. If we consider a vector as an arrow extending from the origin, then "cosine similarity" compares the angle between two arrows: the smaller the angle (the closer the directions), the more semantically similar the two text blocks are; the larger the angle (the much different the directions), the greater the semantic difference between them.

Because cosine similarity only considers direction and not vector length, it is particularly suitable for measuring semantic relevance. In other words, a vector library is like a "semantic index." Unlike traditional databases that use keywords for retrieval, it finds relevant content through "semantic distance." For example, if a user asks, "When will Apple release its new products?", the vector library will look for semantically relevant blocks like "Apple event," rather than rigidly matching the word "Apple."

---

In practical applications, there are many "vector libraries" to choose from. However, it should be noted that the "vector libraries" commonly referred to in the industry usually fall into two categories:Vector search library and Vector databaseThe former focuses more on efficient similarity retrieval, while the latter adds data management and scalability capabilities on top of that.

• Vector search library—This refers to the underlying retrieval algorithms and index implementations, typically represented by… FAISS, Annoy, hnswlibThey offer efficient nearest neighbor search (K-NN), various index structures, and compression strategies, making them suitable for embedding in local applications or as core components of vector retrieval. Their advantages include lightweight design, high performance, and zero maintenance costs (single-machine operation). The disadvantage is that they typically lack full database functionality (such as complex metadata filtering, access control, distributed scaling, etc.). If you use FAISS for retrieval but need to store the document ID, original text, or other metadata corresponding to each vector, you usually need to use a small external database (such as SQLite, LevelDB, or simple JSON/CSV) to store this metadata.
• Vector DatabaseThis is a productized wrapper built on top of a search library, encompassing both high-performance retrieval capabilities and database-level functionalities: metadata indexing/filtering, persistence, sharding/replication, online index management, query interfaces, cloud hosting services, etc. Typical examples include... Milvus, Qdrant, Weaviate, PineconeThe advantages are comprehensive functionality, ease of production-level deployment and scalability; the disadvantages are heavyweight nature, higher maintenance or operating costs (especially for cloud services). Milvus, Qdrant More commonly used for self-hosted/open source deployment;Pinecone, Weaviate (managed version) It's a cloud service, which is convenient but requires payment.

Examples and comparisons (when to choose which):

• For example, if you're doing introductory experiments or small-scale local RAG in your own environment and want "zero maintenance and rapid verification".FAISS (Vector Search Library) It is the preferred choice because it does not require data management and scalability.
• If you are developing an online service that requires metadata filtering (e.g., filtering candidate snippets by date, author, or category), multi-node scaling, or a stable SLA, we recommend considering [this option]. Vector Database(Milvus, Qdrant, Pinecone, etc.).

In my minimized RAG Demo, I will choose FAISS (i.e., the previously installed FAISS-CPU)It has few dependencies, is easy to configure, and can intuitively display semantic search results, making it very suitable for local experimentation on a Mac mini. If the demo needs to be upgraded to a production-grade system in the future, then consider replacing FAISS or connecting it to a vector database to gain more management and scalability capabilities.

4.3.2 Constructing Vector Indexes

After the text blocks are converted into vector representations, I need to organize them into a data structure that can be retrieved quickly. This is... Vector index The role of vector indexes can be understood as the core component of the vector library—they are responsible for efficiently storing vectors and supporting similarity queries.

---

When building vector indexes, we can typically choose between in-memory indexes, disk indexes, or a combination of both. The following are...Common types and storage methods of vector indices:

Usage scenarios	Storage type	Common Indexes	Features and applicable scenarios
Small-scale experiment/debugging	Memory Index	IndexFlatIP / IndexFlatL2	It offers precise searching, a simple structure, and fast query speeds; however, memory consumption increases rapidly with the amount of data, making it suitable for demos and validation.
Million-level search	Memory + Approximate Index	IVFFlat / IVFPQ	By using clustering and quantization to perform approximate search, sacrificing some accuracy for speed and storage, it is suitable for large-scale retrieval.
Online high-performance service	Memory Graph Index	HNSW	Graph-based approximate search offers low latency and high recall, making it suitable for applications with high real-time requirements.
Massive data persistence	Disk index or hybrid index	DiskANN / Faiss on-disk IVF	Supports persistence, remaining available after a restart, suitable for terabyte-scale vector storage and offline/nearline queries.

For the local RAG Demo, if the ultimate goal is to write vectors into a unified vector library, then using in-memory indexes directly during index building and debugging is perfectly fine.

---

Next, I will demonstrate a minimal demo using a Python script. FAISSThis demonstrates how to implement vector indexing using Facebook's open-source vector search library, which is high-performance and easy to run locally. It shows how to add vectors to the index and perform similarity queries. For ease of understanding, I've broken down the key steps of the script into steps 1, 2, and 3; in actual use, you can directly execute the complete script in step 4.

1. Initialize vector indexes

import faiss import numpy as np dimension = 384 # (consistent with text vector dimension) index = faiss.IndexFlatIP(dimension) # Uses inner product, which can be used for cosine similarity

Here we use IndexFlatIP because cosine similarity can be converted into inner product calculation after vector normalization, so that the vector of each text block can be directly added to the index.

2. Add index to vector

# Assuming we already have a list of vectors vec1, vec2, vec3, `vectors = np.array([vec1, vec2, vec3], dtype='float32')` `index.add(vectors)` # This adds vectors to the index. `print(f"There are {index.ntotal} vectors in the vector index")`

Once added, the vector index is complete and can be used immediately for similarity retrieval.

3. Persistence (optional)

# Save the index to disk: `faiss.write_index(index, "vector_index.faiss")` # Next time load: `index = faiss.read_index("vector_index.faiss")`

This way, even if the program ends, the existing vector index can be used directly on the next startup, achieving persistence.

At this point, we have completed the construction of the minimal vector library.It provides a reliable storage foundation for subsequent text retrieval and answer generation.In the next section, we can discuss how to incrementally manage, save, and load vector libraries to handle the ever-growing blocks of text in real-world use.

4.3.3 Vector Library Management

Once the vector library is built, it's not simply a matter of "leaving it there and calling it a day." In practical use, the vector library requires dynamic management to ensure retrieval efficiency and data integrity. Management mainly includes... Incremental addition, update, save and load Four aspects:

1. Incremental addition

New text blocks continuously generate new vectors, and the vector library should support adding them at any time. FAISS supports incrementally adding vectors by directly calling index.add().

new_vectors = np.array([vec_new1, vec_new2], dtype='float32') index.add(new_vectors) print(f"The total number of vectors has been updated to {index.ntotal}")

This approach eliminates the need to rebuild the index, making it ideal for scenarios where knowledge is dynamically expanded in RAG.

2. Vector update

If a text block is modified, the corresponding vector must be deleted and then re-added. FAISS's native indexes (such as IndexFlatIP) do not support single-row deletion; this can be achieved using IndexIDMap combined with a custom ID.

In #, create an index with an ID: `index_id = index.IndexIDMap(index)` and `index_id.add_with_ids(vectors, ids)`. When updating a vector in #, remove the element before adding it.

3. Save and Load

To prevent data loss due to program exit, the vector library should be saved periodically:

faiss.write_index(index_id, "vector_index.faiss") # Next time load index_id = faiss.read_index("vector_index.faiss")

For large indexes, efficiency can be improved by combining segmented or incremental saving strategies.

4. Index optimization (optional)

For large-scale vector libraries, FAISS's IVF, PQ, and other compressed or clustered indexes can be used to improve retrieval speed while saving memory.

nlist = 100 # clustering number quantizer = faiss.IndexFlatL2(dimension) index_ivf = faiss.IndexIVFFlat(quantizer, dimension, nlist, faiss.METRIC_L2) index_ivf.train(vectors) index_ivf.add(vectors)

Although complex, these management methods are essential when the number of vectors grows to hundreds of thousands or millions. Through these management techniques, the vector library can... Continuously and stably serve RAG retrievalEven as the text blocks continue to increase, it will not affect the retrieval efficiency and accuracy.

At this point, the basic structure of the vector library has been completed. We can not only store the vectors of text blocks in an orderly manner, but also retrieve them quickly when needed, and persist them so that they remain available after the program restarts.With such a stable "semantic repository", RAG's infrastructure gradually became complete.Next, we move on to the final part of Chapter Four—how to use vector libraries to retrieve information and ultimately generate the answers we want.

4.4 Large Model

4.4.1 The Responsibilities of the Large Model in RAG

In the previous two sections, we completed the vectorization of the "text blocks" and organized these vectors into a "vector library." This way, when a user's question is retrieved, a set of semantically related text fragments can be found. Next, it's time to... Large Model (LLM) They're on stage.

In the RAG architecture, the main responsibilities of the large model can be summarized in three points:

1. Understanding User Problems

The first step for a large model is to perform semantic understanding of the user's input question and identify the underlying information needs.

2. Based on the search results

Relying solely on the large model itself may result in outdated or limited knowledge. Therefore, we need to provide "relevant text chunks" retrieved from vector libraries as supplementary information for the large model's reference. In this way, the model's answers can be based on the latest, domain-specific data, rather than solely relying on the training corpus.

3. Generate the final answer

After obtaining the question and supplementary materials, the large model is responsible for combining the two to generate a natural language response. This step ensures both factual accuracy and fluent language, which reflects the value of RAG.

The role of the large model in RAG can be likened to the "final interpreter": the vector library provides "reference materials," while the large model decides how to organize these materials to ultimately provide the most useful answers for the user.

In order for the large model to successfully fulfill these responsibilities, we must design a reasonable... Input structure (prompt structure)That is, how to combine "user questions + retrieved text blocks" into the input of the large model.

4.4.2 Input Structure

In the previous section, we clarified the three responsibilities of the large model within the RAG architecture: understanding the question, integrating search results, and generating answers. The key to enabling the large model to effectively perform these tasks lies in… Input structure (prompt structure) The design.

In other words, the answers that a large model can output largely depend on us. How to combine user questions and retrieved text blocks?If the input structure is disorganized, the model may ignore the search results; if the input is too long, the model may lose sight of the key points.

A common input structure generally consists of three parts:

1. Instruction section

Clearly tell the large model what its task is, for example: "Please answer the user's questions based on the following information. Do not make up answers." This part sets the tone and prevents the model from going off-topic or creating illusions.

2. Context section

This section contains several relevant text blocks retrieved from the vector library. These serve as the "factual basis" for the large model to generate its answers, essentially acting as temporary add-on knowledge.

3. Questions section

Finally, there's the user's original question. Placing the question towards the end helps the model understand the context before focusing on the question itself.

A typical input structure can look like this:

You are an intelligent assistant. Please answer users' questions strictly according to the context. If the answer cannot be found in the context, please explicitly answer "I don't know," and do not make it up. [Context] {text_chunks} [Question] {user_question}

The advantages of this structure are: clear task instructions prevent the model from acting arbitrarily; clear division of context and problem makes the model easy to parse; and it can maintain the relevance of the generated content while reducing the risk of hallucinations.

Of course, in actual projects, the input structure can be continuously optimized according to requirements, such as adding "response format requirements" or controlling "response length". But in any case, the core idea remains the same:Within a clearly defined task framework, the large model combines retrieved data and user questions to generate answers..

4.4.3 Enhanced Retrieval Call Flow

Having understood the responsibilities of large models and the basic structure of input data, we can now connect them with vector libraries to form a complete system. Enhanced search call processThe core of this process is:User asks a question → Search relevant documents → Construct input → Generate answer using a large model.

The entire process can be divided into four steps:

1. User Questions

Users can input natural language questions, such as: "What is the difference between a vector database and a vector search library?"“

2. Vector retrieval

The system first transforms the question into vectors, then retrieves the most similar text blocks from the vector library. These text blocks are the "candidate knowledge," which will be provided as context to the larger model.

3. Construct the input structure

Following the rules of the previous section, the retrieved text block (Context) and user question (Question) are concatenated into a unified input, and a task instruction (Instruction) is added.

4. Large-scale model generates answers

Finally, the assembled input is fed into a large model, which generates an answer based on the context. If the context does not cover the information required for the question, the model will answer "I don't know" as instructed.

A simplified pseudocode flow would look something like this:

# 1. User input question = "What is the difference between a vector database and a vector search library?" # 2. Vector retrieval q_vector = embed(question) # Vectorize the question D, I = index.search(q_vector, k=3) # Retrieve the top-3 in the vector library retrieved_chunks = [chunks[i] for i in I[0]] # 3. Construct input prompt = f"""You are an intelligent assistant. Please answer the question based on the following "context". If the answer cannot be found in the context, please answer "I don't know". 【Context】 {retrieved_chunks} 【Question】 {question} """ # 4. Large model generates answer answer = llm.generate(prompt) print(answer)

Thus, a complete RAG call loopAnd so it was established:

• Vector libraries provide external knowledge;
• The input structure ensures that knowledge is correctly transferred;
• The large model is responsible for understanding and generating natural language answers.

This is why we say that RAG is not about "making large models know everything," but rather about "making large models learn to use external data."

4.4.4 Additional Knowledge: Selection of Large Models

In the previous sections, we clarified the role of the large model in RAG, its input structure, and the call flow for retrieval enhancements. The next step is to consider... Which major model to use?(For the sake of simplicity, this demo uses the LLM model provided by Hugging Face; if you are looking for performance or want to deploy locally, you can also run LLaMA and other models through the Ollam platform.) This choice will directly affect the model generation quality, speed, and hardware resource requirements.

1. Model Selection Principles

When selecting a large model, the following factors generally need to be considered:

Generation capability and accuracy

The larger the model, the stronger its ability to understand questions and generate high-quality answers, but it also consumes more resources.

Hardware resource limitations

This includes GPU/CPU memory, number of threads, and whether quantization is supported.

Latency and response speed

Real-time interactive scenarios have certain requirements for generation speed; excessively large models may lead to response delays.

Usability and compatibility

Is there a quantifiable version (GGUF, GGML, bitsandbytes, etc.)? Is it easy to deploy locally or load via Hugging Face?

2. Comparison of Common Optional Models

Model	Parameters	Advantage	Hardware Recommendations
GPT-2 / GPT-Neo 125M–1.3B	0.1–1.3B	Compact, fast reasoning, easy to deploy	It can run on Mac/PC CPUs or small GPUs.
Mistral 7B	7B	High performance, fast inference speed	16GB+ GPU or 24GB Mac RAM
LLaMA 3 8B (Quantization GGUF)	8B	Balancing performance and resource consumption, suitable for local deployment	The Mac M series with 24GB of RAM can run
LLaMA 3 13B	13B	More powerful generation capabilities, suitable for complex problems	40GB+ GPU or large server
LLaMA 3 70B	70B	Extremely high generation capacity, but high resource consumption.	Multi-GPU or large cloud servers are not suitable for local Macs.

Note: The larger the model, the more GPU memory and system memory are required. When deploying locally, choosing a quantized version (such as Q4_K_M GGUF) can significantly reduce resource consumption while maintaining reasonable generation results.

3. Choosing a strategy

• Personal laptop or local Mac

I recommend LLaMA 3 8B (quantized version of GGUF), which balances performance and speed.

• Mid-range GPU servers or cloud environments

You can choose Mistral 7B or LLaMA 3 13B for stronger generation capabilities.

• High-end multi-GPU cloud environment

Consider using LLaMA 3 70B or other ultra-large models to achieve the best generation results.

4. Practical Recommendations

• Choose a model tailored to your hardware resources to avoid running out of memory and causing failure.
• For interactive demonstrations like RAG Demo, the model doesn't need to be too large; maintaining responsiveness and runnability is sufficient.
• If we need to handle larger-scale knowledge or more complex problems in the future, we can then consider upgrading to a larger model.

5. Script Structure and Data Flow

5.1 Script Structure Description

In the previous chapters, we respectively started from...Document blocks, vector libraries, and large modelsFrom the perspectives of three roles, the key aspects of the minimum RAG Demo are broken down step by step. To truly connect these disparate steps, we need to organize them into several independent Python scripts. The advantages of doing so are: clear logic, well-defined responsibilities, and ease of future expansion or replacement of any part.

In this demo, we will use three scripts:

1. document_process.py

• Responsibilities: Document segmentation and vectorization (corresponding to "Text Blocks" in Chapter 4.2).
• Input: Original document (txt/markdown, etc.).
• Output: A list of vectorized text blocks, stored in memory or an intermediate file.

2. vector_store.py

• Responsibilities: Building and managing vector indexes (corresponding to "Vector Library" in Chapter 4.3).
• Input: Vectors generated by document_process.py.
• Output: A vector library with a completed index (supports retrieval and can be optionally persisted to a local file).

3. rag_pipeline.py

• Responsibilities: Organizing the invocation of the large model and generating answers by combining the search results (corresponding to "Large Model" in Chapter 4.4).
• Input: The user's query question.
• Output: The final generated result of RAG.

These three scripts can run independently (for easy testing) or form a complete pipeline through data transfer. In other words:They correspond exactly to the "three core roles," only the tasks are slightly broken down in implementation.

5.2 document_process.py

"""" document_process.py Function: Completes the processing of "document blocks" (optimized version) Includes: 1. Document splitting (corresponding to 4.2 Document Blocks - Document Splitting) 2. Text vectorization (corresponding to 4.2 Document Blocks - Vectorization) 3. Output process files: processed_docs.json + embeddings.npy """ import os import json import numpy as np from sentence_transformers import SentenceTransformer # ====================== # 1. Document Splitting # ====================== def load_and_split_md(file_path): """Load the Markdown file and split it by paragraph""" blocks = [] with open(file_path, "r", encoding="utf-8") as f: paragraph = [] for line in f: line = line.strip() if line == "": if paragraph: blocks.append(" ".join(paragraph)) paragraph = [] else: paragraph.append(line) if paragraph: blocks.append(" ".join(paragraph)) return blocks def load_and_split_dir(dir_path): """Traverse all .md files in the directory and its subdirectories and split them by paragraph""" all_blocks = [] for root, dirs, files in os.walk(dir_path): for file in files: if file.lower().endswith(".md"): file_path = os.path.join(root, file) blocks = load_and_split_md(file_path) all_blocks.extend(blocks) return all_blocks # ====================== # 2. Text Vectorization # ====================== def embed_texts(texts, model_name="sentence-transformers/all-MiniLM-L6-v2"): """Use a pre-trained model to convert text blocks into vectors""" model = SentenceTransformer(model_name) embeddings = model.encode(texts, convert_to_numpy=True, normalize_embeddings=True) return embeddings # ====================== # Main Flow # ====================== if __name__ == "__main__": dir_path = "document" print(f"Processing directory: {dir_path}") # Step 1: Traverse the directory and generate text blocks text_blocks = `load_and_split_dir(dir_path)` prints `f"A total of {len(text_blocks)} text blocks were generated."` # Save the text blocks as JSON: `with open("processed_docs.json", "w", encoding="utf-8") as f: json.dump(text_blocks, f, ensure_ascii=False, indent=2)` prints `"Text blocks have been saved to processed_docs.json")` # Step 2: Text vectorization: `vectors = embed_texts(text_blocks)` `np.save("embeddings.npy", vectors)` prints `"Vectorization has been saved to embeddings.npy")` prints `"Vectorization complete, vector matrix dimensions:", vectors.shape)`

document_process.py script description:

This script does two things:

1. Document Segmentation (corresponding to 4.2.1 Document Blocks – Document Segmentation)It iterates through the Markdown files in the document/ directory, breaks each document into smaller blocks by paragraph, and generates a processed_docs.json file that records all the split text blocks.

2. Text vectorization (corresponding to 4.2.2 Document Blocks – Vectorization)The sentence-transformers model is called to convert text blocks into vectors and store the vectors as NumPy arrays in the embeddings.npy file for use in the next step of building vector indices.

The generated intermediate files:

• processed_docs.json: A list of split document fragments used to build a vector index.
• embeddings.npy: The vector matrix corresponding to the text blocks, used to build vector indices.

Operating method:

• Save as document_process.py
• Run the following in the terminal:

python3.11 document_process.py

5.3 vector_index.py

"""" vector_index.py Function: Builds and manages vector indexes, including: 1. Loading text blocks and vector files 2. Building the FAISS vector index 3. Saving the index and mapping relationships""" import faiss import numpy as np import pickle import json # ===================== # Parameter Settings # ====================== dimension = 384 index_file = "vector_index.faiss" mapping_file = "vector_index.pkl" text_blocks_file = "processed_docs.json" vectors_file = "embeddings.npy" top_k_demo = 3 # Demo Retrieving the Top k Results # ====================== # Step 1: Loading Text Blocks and Vectors # ====================== with open(text_blocks_file, "r", encoding="utf-8") as f: text_blocks = json.load(f) vectors = np.load(vectors_file) print(f"Loading {len(text_blocks)} text blocks and a vector matrix, dimension {vectors.shape}") # ====================== # Step 2: Building the FAISS Index # ====================== index = faiss.IndexFlatIP(dimension) # Inner product as cosine similarity index.add(vectors) print(f"Vector index construction complete, total number of vectors: {index.ntotal}") # ====================== # Step 3: Save the index and mapping relationship # ====================== faiss.write_index(index, index_file) print(f"Index has been saved to {index_file}") with open(mapping_file, "wb") as f: pickle.dump(text_blocks, f) print(f"Vector mapping relationship has been saved to {mapping_file}") # ====================== # Step 4: Simple search demonstration # ====================== query_vec = vectors[0].reshape(1, -1) distances, indices = index.search(query_vec, top_k_demo) print("\nSearch demonstration:") print(f"Query text block: {text_blocks[0]}\n") for rank, idx in enumerate(indices[0], 1): print(f"Rank {rank}: Similarity {distances[0][rank-1]:.4f}") print(f"Text block content: {text_blocks[idx]}\n")

Description of the vector_index.py script:

This script does two things:

1. Build a vector index (corresponding to 4.3 Vector Library – Building an Index):

• Load text blocks from processed_docs.json
• Load text block vectors from embeddings.npy
• Use FAISS to add vectors to the vector library.
• Establish a mapping relationship between vectors and text blocks.

2. Vector library persistence (corresponding to 4.3 Vector Library – Persistence Management):

• Write the constructed vector index to disk, generating vector_index.faiss.
• The mapping relationship between vectors and text blocks is saved as vector_index.pkl for easy reference when searching for corresponding text blocks later.

The generated intermediate files:

• vector_index.faiss: Vector index file used for fast retrieval.
• vector_index.pkl: The mapping relationship between vectors and text blocks, used to restore the search results to the original text blocks.

Operating method:

• Save as vector_index.py
• Ensure that document_process.py has generated processed_docs.json and embeddings.npy.
• Run the following in the terminal:

Python 3.11 vector_index.py

5.4 rag_query.py

"`rag_query.py` Function: Completes the calling of the "large model" in RAG, including: 1. Loading vector indices and mapping relationships 2. Receiving user queries 3. Retrieving relevant text blocks based on similarity 4. Calling the large model to generate answers and displaying reference fragments (console beautified output) ``` import faiss import pickle from sentence_transformers import SentenceTransformer from transformers import AutoTokenizer, AutoModelForCausalLM import torch # Attempts to import colorama; if unavailable, it falls back to an empty string (compatible with environments without colorama) try: from colorama import Fore, Style, init init(autoreset=True) except Exception: class _Dummy: def __getattr__(self, name): return "" Fore = _Dummy() Style = _Dummy() # ====================== # Parameter settings # ====================== index_file = "vector_index.faiss" mapping_file = "vector_index.pkl" embedding_model_name = "sentence-transformers/all-MiniLM-L6-v2" # Chinese optimized public model lm_model_name = "Jackrong/llama-3.2-3B-Chinese-Elite-v2" top_k = 3 # ====================== # Step 1: Loading the index and mapping # ====================== index = faiss.read_index(index_file) with open(mapping_file, "rb") as f: text_blocks = pickle.load(f) print(f"{Fore.CYAN} vector index loading complete, total vectors: {index.ntotal}{Style.RESET_ALL}") print(f"{Fore.CYAN} mapped text blocks loading complete, total: {len(text_blocks)}{Style.RESET_ALL}") # ====================== # Step 2: Initialize the query vector model # ===================== embed_model = SentenceTransformer(embedding_model_name) # ====================== # Step 3: Initialize the LLM (adapted for macOS / CPU) # ====================== tokenizer = AutoTokenizer.from_pretrained(lm_model_name) lm_model = AutoModelForCausalLM.from_pretrained( lm_model_name, device_map="auto", torch_dtype=torch.float16 ) # ======================== # Step 4: Query and retrieval # ====================== def retrieve_relevant_blocks(query, top_k=3): query_vec = embed_model.encode([query], convert_to_numpy=True, normalize_embeddings=True) distances, indices = index.search(query_vec, top_k) results = [(text_blocks[i], float(distances[0][rank])) for rank, i in enumerate(indices[0])] return Results # ====================== # Step 5: Generate Answer (Optimize Prompt) # ====================== def generate_answer(query): retrieved = retrieve_relevant_blocks(query, top_k) context = "\n".join([block for block, _ in retrieved]) prompt = f""" You are a smart assistant. Your task is to answer user questions based on the given "context". Please note: - Strictly quote key sentences from the context to answer questions; - Combine multiple pieces of information into one or two sentences, keeping it as concise as possible and avoiding repetition; - Do not restate the context or question; - If the answer is not found in the context, output "I don't know"; - Answers are in the format of "\n". End. Context: {context} Question: {query} Answer: """ inputs = tokenizer(prompt, return_tensors="pt") inputs = {k: v.to(lm_model.device) for k, v in inputs.items()} output_ids = lm_model.generate( **inputs, max_new_tokens=200, do_sample=False, repetition_penalty=1.2, eos_token_id=tokenizer.eos_token_id ) full_output = tokenizer.decode(output_ids[0], skip_special_tokens=True) answer = full_output.split(" ")[-1].split(" ")[0].strip() return answer, retrieved # ====================== # Step 6: Interactive Demonstration # ====================== if __name__ == "__main__": while True: user_query = input(Fore.YELLOW + "Please enter your question (press Enter to exit):" + Style.RESET_ALL) if not user_query.strip(): break answer, refs = generate_answer(user_query) print(f"\n{Fore.GREEN}RAG Generated Answer:{Style.RESET_ALL}\n{Fore.GREEN}{answer}{Style.RESET_ALL}") print(f"\n{Fore.MAGENTA}Reference Fragment:{Style.RESET_ALL}") for rank, (block, score) in enumerate(refs, 1): print(f"{Fore.YELLOW}[Rank {rank}] Similarity: {score:.4f}{Style.RESET_ALL}") print(f"{Fore.LIGHTBLACK_EX}{block}{Style.RESET_ALL}") print(f"{Fore.CYAN}{'-' * 40}{Style.RESET_ALL}") print(f"{Fore.CYAN}{'=' * 80}{Style.RESET_ALL}")

This rag_query.py script implements a simple RAG workflow, with the following main functions:

1. Load vector indices and mapping relationships

• Load the constructed vector index from vector_index.faiss
• Load text block mappings from vector_index.pkl
• Restore the correspondence between the processed document vector library and text blocks.

2. Query text blocks

• Problem of receiving user input
• Vectorize the problem using SentenceTransformer
• Retrieves the text block most similar to the question from the vector index (defaults to top_k=3).

3. Use a large model to generate answers.

• Concatenate the retrieved text blocks into context.
• Constructing a Prompt: Only allowing answers based on context.
• Call the large model to generate the final answer and return the fragment referenced when generating the answer.
• If the answer cannot be found in the context, return 0. “"I have no idea"”

4. Interactive query demonstration (with console enhancement)

• Support for loop input problem
• The output is color-coded (if the Colorama dependency is installed):green → The generated answer;Purple/Yellow → Refer to the segment title and similarity score;grey → Refer to the main text of the text block;blue → Separator.
• Make search results more intuitive

5. Prompt Design Principles

• Answers must be strictly based on the “context” text block.
• If there is no answer in the context, return 0. “"I have no idea"”
• Avoid generating or expanding content out of thin air in the model.

Dependent intermediate files

• vector_index.faiss: Vector index file (generated by vector_index.py)
• vector_index.pkl: Vector mapping relationships (generated by vector_index.py)

Operating method:

python3.11 rag_query.py

---

Precautions and Frequently Asked Questions:

1. Model quantization problem
   • In early scripts load_in_8bit=True or load_in_4bit=True Quantization loading may result in errors on Mac CPUs/M series because it requires dependencies. bits and bytes or accelerate.
   • The latest version of Transformers is recommended to use quantization_config The configuration object replaces the old parameters.
   • In this script, the bitsandbytes dependency has been removed, and the model is loaded directly using the CPU to avoid quantization errors.
2. runtime error problem
   • If you encounter ImportError: CUDA not available or bits and bytes The errors are mostly due to the fact that older quantization methods require CUDA support.
   • This script does not require a GPU and is executed directly using the CPU, and is compatible with Mac M series CPUs.
   • Some transformer versions may indicate that certain parameters are invalid (such as temperature, top_p), but this does not affect normal operation and can be ignored.
3. Model download interrupted or slow download speed
   • Large models (especially 3B+ models) may experience interruptions or network issues when downloading via Hugging Face, prompting you to retry or indicating that the download has failed.
   • When encountering such problems, you can manually resolve them in your browser or by using [another tool/system]. git lfs Download the model file to your local machine, and... from_pretrained() Specify the local path. Example: lm_model_name = "/path/to/local/model/directory"“
4. suggestion
   • For Mac CPU/M series users, it is recommended to use the 1B or 3B Chinese optimized public model to avoid excessive memory pressure caused by the 7B or higher model.
   • If only the RAG process is being tested, a smaller model is sufficient for functional verification.

---

5.5 Summary

Based on the content of the previous sections, the final project directory structure is as follows:

`project_root/` ├── `document/` # Original document directory (containing .md files), subdirectories are also allowed │ ├── `doc1.md` │ ├── `doc2.md` │ └── … ├── `processed_docs.json` # JSON block text (generated by `document_process.py`) ├── `embeddings.npy` # Text vector matrix (generated by `document_process.py`) ├── `vector_index.faiss` # Faiss vector index file (generated by `vector_index.py`) ├── `vector_index.pkl` # Vector mapping relationship (generated by `vector_index.py`) ├── `document_process.py` # Document splitting and vectorization script ├── `vector_index.py` # Vector index building script └── `rag_query.py` # Interactive Question and Answer Script

The actual data flow process is as follows:

1. **Document Segmentation and Vectorization (document_process.py)** - Input: All `.md` files under `document/` - Output: - `processed_docs.json` (segmented text blocks) - `embeddings.npy` (vector matrix corresponding to the text blocks) 2. **Building a Vector Index (vector_index.py)** - Input: `processed_docs.json` + `embeddings.npy` - Output: - `vector_index.faiss` (FAISS vector index) - `vector_index.pkl` (text block to vector mapping relationship) 3. **Interactive Question Answering (rag_query.py)** - Input: User question + vector index file - Process: Retrieve relevant text blocks → Generate answer using LLM - Output: Interactively display the final answer in the command line

6. Hands-on Practice – Getting RAG Running

In the previous chapter, we prepared three core scripts (document_process.py, vector_index.py, and rag_query.py), as well as a Markdown document for testing. The following steps will demonstrate how to actually run the entire RAG Demo:

1. Create a new project directory and document directory

mkdir ~/Projects mkdir ~/Projects/document

2. Copy the three prepared core scripts to the project directory (or create them directly in the directory).

cp /xx/document_process.py ~/Projects cp /xx/vector_index.py ~/Projects cp /xx/rag_query.py ~/Projects

3. Place the test document in the document directory.

Place several .md files in the ~/Projects/document/ directory (which may contain subdirectories). These files will be the source of knowledge for the subsequent Q&A. In this practical exercise, I placed the .md files corresponding to 3 articles:

4. Segmentation and Vectorization

Run the following commands sequentially in the terminal:

python3.11 document_process.py

The terminal output after completion is as follows:

Run again:

Python 3.11 vector_index.py

The terminal output after completion is as follows:

After both scripts have finished running, they will generate processed_docs.json and embeddings.npy (generated by the document_process.py script), vector_index.faiss and vector_index.pkl (generated by the vector_index.py script) in the project root directory.

5. Interactive Question and Answer

implement:

python3.11 rag_query.py

After entering the interactive interface, input your question to get answers based on local document searches. Below are some test questions I prepared for different article content and their corresponding outputs:

Regarding Article 1 (The Value of Blogs in the AI Era)

• Why is writing a personal blog still valuable in the age of AI?

RAG's response is as follows:

• What is a "trustworthy knowledge anchor"? What is its significance?

RAG's response is as follows:

Regarding article 2 (Cloudflare Tunnel)

• What SEO risks might exist when using Cloudflare Tunnel to build a website?

The RAG response is as follows:

• Why do non-standard ports affect search engine indexing?

The RAG response is as follows:

Regarding article 3 (WordPress Multi-Active Architecture)

• Why should personal blogs consider a WordPress multi-active architecture?

The RAG response is as follows:

• What are the key technical aspects of a WordPress multi-active architecture?

The RAG response is as follows:

Based on the test results, I am satisfied with the performance of the RAG demo.

---

RAG's strength lies in providing answers "for a single question, combined with relevant fragments," which means performing "point-to-point" knowledge retrieval and question answering.

If you try to make it aggregate and summarize across multiple articles or even across topics at once, the results are often unstable—because the model needs to "stitch together logic" between multiple contexts, which is beyond RAG's comfort zone.

Therefore, in practical applications, it is recommended to focus on a specific knowledge point or article, gradually obtain the answer, and then integrate it yourself. This will maximize the value of RAG and avoid the gap between expectations and actual results.

---

7 Conclusion

I finally got this minimal local RAG demo running! I never imagined it would take me more than two weeks from initially outlining the process and organizing the technical details to actually completing the practical implementation. I encountered quite a few problems along the way, such as: the safetensors file getting stuck when downloading the HF model due to a fast internet connection; segmentation faults caused by incompatibility between GPU memory/memory or underlying libraries; and errors in bitsandbytes 8-bit quantization due to the lack of CUDA on macOS… These problems piled up, forcing me to modify rag_query.py dozens of times. Fortunately, I eventually solved them one by one.

However, every coin has two sides. Although the process of tinkering was painful, it also gave me a more intuitive understanding of many underlying details that I wouldn't normally pay attention to: such as the mechanism of model downloading, the coupling relationship between Python libraries and hardware acceleration, the boundary between memory and video memory, and so on. These things that were originally "invisible" were forced to be dissected and examined in the process of error reporting and debugging, which gave me unexpected insights.

Through this demo, I have gained a more solid grasp of the overall RAG workflow: from document slicing and vectorized storage to retrieval and reorganization, and finally to generating answers, each step is interconnected and indispensable. Just building a minimal, functional prototype was enough for me to appreciate the power and limitations of RAG within the "retrieval + generation" framework. It's not a panacea, but it is a highly efficient and practical paradigm.

Next, I can consider using LangChain or LlamaIndex(I feel a dedicated article is needed to compare these two approaches to clarify things.) Modularizing these processes and truly moving towards "production-grade" applications, the goal is clear—to integrate with my blog and create a chatbot that provides readers with instant Q&A and in-depth interaction. At that point, the challenges will likely increase: for example, how to optimize search efficiency, how to make answers more context-aware, how to balance accuracy and fluency… but it is precisely these challenges that make the whole process more interesting.

Looking back, this demo was actually like a "testing ground": it forced me to find ways to solve problems with limited resources, and it also showed me the boundaries and potential of technology. It's foreseeable that in the future, RAG will not be an isolated tool in my blog ecosystem, but will become a piece of the puzzle, combined with knowledge graphs, intelligent recommendations, automatic summarization, and other capabilities to form a more comprehensive knowledge system.

Note 1: I feel that I have only truly completed the first step of "RAG introduction" by writing this far. I don't know how many more steps there are, or how far I can go. I can only take it one step at a time.

Note 2: This article was actually written at the end of September last year, but it was delayed for almost half a year because it was given way to the "Awakening of the Voice" series of articles.

Note 3: Article formatting has always been a major headache for me. I now directly paste Markdown formatted articles from Obsidian into my own articles, and my Markdown plugin, "WP Editor.md," hasn't been updated in a long time. I also use CSS with automatic first-paragraph indentation and some rendering from the Argon theme. So, when there are formatting issues, I don't even know where to begin troubleshooting. After all, I'm not good at this, and I have little interest in researching it. This also makes me dread complex articles—once there are many different types of content, the formatting becomes a mess, just like this article.