Introduction to RAG

What is RAG?

RAG (Retrieval-Augmented Generation) is a technical paradigm designed to solve the problem of "knowing what is happening but not why" in large language models (LLM). Its core is to incorporate what is learned inside the model "Parametric knowledge” (the solidified and fuzzy “memory” in the model weights), and the “memory” from the external knowledge basenon-parametric knowledge"(external data that can be updated at any time). Its operating logic is to dynamically obtain relevant information from the external knowledge base through the retrieval mechanism before LLM generates text, and integrate these "reference materials" into the generation process, thereby improving the accuracy and timeliness of the output.

Technical principles

Retrieval stage: looking for “non-parametric knowledge”

  • Knowledge vectorization: Embedding Model Acts as a "connector". It encodes the external knowledge base into a vector index (Index) and stores it invector database.
  • semantic recall: When the user initiates a query, the retrieval module uses the same embedding model to vectorize the question and passSimilarity Search , accurately locking the document fragments most relevant to the problem from massive data.

Generation phase: merging two kinds of knowledge

  • contextual integration: Generate moduleReceive relevant document fragments sent during the retrieval phase as well as the user's original questions.
  • instruction boot generation: This module will follow the default Prompt Instructions, effectively integrate context and questions, and guide LLM (such as DeepSeek) to perform controllable and well-founded text generation.

technological evolution

Why use RAG?

Technology selection: RAG vs. fine-tuning

When choosing a specific technology path, an important consideration is the balance between costs and benefits. Usually, we should give priority to the solution with the smallest changes to the model and the lowest cost, so the technology selection path often follows the order:Prompt Engineering -> Retrieval Enhancement Generation -> Fine-tuning .

We can understand the difference between these technologies from two dimensions. As shown in Figure 1-3,horizontal axis represent" LLM optimization" , that is, to what extent the model itself is modified. From left to right, the degree of optimization increases, with hint engineering and RAG not changing the model weights at all, and fine-tuning directly modifying the model parameters.The vertical axis represents "contextual optimization" , is the extent to which the information input to the model is enhanced. From bottom to top, the degree of enhancement is getting higher and higher, where prompt engineering only optimizes the way of asking questions, while RAG greatly enriches contextual information by introducing external knowledge bases.

Based on this, our choice path is clear:

  • Try the prompt project first: Guide the model through carefully designed prompt words, which is suitable for scenarios where the task is simple and the model already has relevant knowledge.
  • Then select RAG: If the model lacks specific or real-time knowledge to answer, use RAG to provide it with contextual information through a plug-in knowledge base.
  • Finally consider fine-tuning: Fine-tuning is the final and most appropriate option when the goal is to change "how" the model does (behavior/style/format) rather than "what" it knows (knowledge). For example, let the model learn to strictly follow a unique output format, imitate the conversational style of a specific person, or "distill" extremely complex instructions into the model weights

Key advantages

(1)Double improvement of accuracy and credibility

The core value of RAG is to break through the limitations of model pre-training knowledge. It can not onlySupplement knowledge blind spots in professional fields, and also by providing specific reference materials, effectivelySuppress the illusion of "serious nonsense". The paper research also shows that the content generated by RAGconcretenessandDiversityIt is also significantly better than pure LLM. More importantly, RAG hasTraceability——The source of the corresponding original document can be found for each answer. This "well-documented" feature greatly improves the credibility of the content in serious scenarios such as legal and medical.

(2)Timeliness guarantee

In terms of knowledge updating, RAG solves the inherent problems of LLMknowledge lag problem(i.e. the model does not know what happened after the training deadline). RAG allows the knowledge base to be model independentDynamic updates——Once new policies or new data are stored in the database, they can be retrieved immediately. This ability is called "Index Hot-swapping" in the paper - just like changing a memory card to a robot, it can instantly switch its world knowledge base without retraining the model, realizing real-time online knowledge.

(3)Significant overall cost effectiveness

From an economic perspective, RAG is a cost-effective solution. First of all, itHigh frequency fine-tuning is avoidedThe huge computing power cost brought by it; secondly, due to the powerful assistance of external knowledge, when we deal with problems in specific fields, we can often useBasic model with smaller number of parametersto achieve similar effects, thereby directly reducing the reasoning cost. This architecture also reduces the consumption of computational resources required to try to "cram" massive amounts of knowledge into model weights.

(4)Flexible modular scalability

RAG's architecture is extremely inclusive and supportsMulti-source integration, whether it is PDF, Word or web page data, it can be unified into the knowledge base. At the same time, itsModular designThe decoupling of retrieval and generation is achieved, which means that we can independently optimize the retrieval component (such as replacing a better Embedding model) without affecting the stability of the generation component, which facilitates long-term iteration of the system.

Applicable scenario risk classification

How to get started with RAG quickly

(1) Data preparation and cleaning: This is the foundation of the system. We need to standardize multi-source heterogeneous data such as PDF and Word, and adopt reasonable chunking strategies (such as splitting by semantic paragraphs instead of fixed number of characters) to avoid information fragmentation during cutting.

(2) Index construction: Convert the segmented text into vectors through the embedding model and store them in the database. Metadata (e.g. source, page numbers) can be associated at this stage, which can be helpful for precise citations later on.

(3) Search strategy optimization: Don’t rely on a single vector search. Mixed retrieval (vector + keyword) and other methods can be used to improve the recall rate, and a re-ranking model can be introduced to re-select the retrieval results to ensure that LLM sees only the best.

(4) Generation and prompt engineering: Finally, design a clear set of Prompt templates to guide LLM to answer user questions based on the retrieved context, and clearly require the model to "say you don't know if you don't know" to prevent hallucinations.

Data preparation

Data loading

Document loader

In the RAG system,Data loadingIt is the first step in the entire assembly line and an indispensable step. The document loader is responsible for converting unstructured documents in various formats (such as PDF, Word, Markdown, HTML, etc.) into structured data that can be processed by the program. The quality of data loading will directly affect subsequent index construction, retrieval results and final generation quality.

The document loader generally needs to complete three core tasks in RAG's data pipeline. One is to parse original documents in different formats and convert PDF, Word, Markdown The second is to extract key information such as document source, page number, author, etc. as metadata during the parsing process. The third is to organize the text and metadata into a unified data structure to facilitate subsequent segmentation, vectorization, and storage. The overall process is similar to the extraction, conversion, and loading in traditional data engineering. The goal is to clean and align messy original documents into standardized corpus suitable for retrieval and modeling.

Current mainstream RAG document loaders

Unstructured document processing library

Unstructured’s core strengths

Unstructured (1) is a professional document processing library specifically designed for unstructured data preprocessing in RAG and AI fine-tuning scenarios. It provides a unified interface to handle multiple document formats and is one of the most widely used document loading solutions. Unstructured has obvious advantages in format support and content parsing. On the one hand, it supports multiple document formats such as PDF, Word, Excel, HTML, Markdown, etc., and avoids writing separate codes for different formats through a unified API interface. On the other hand, it can automatically identify document structures such as titles, paragraphs, tables, and lists, while retaining corresponding metadata information.

Supported document element types

text chunking

Understanding text chunking

Text Chunking is a key step in building a RAG process. Its principle is to divide the loaded long document into smaller and easier-to-process units. These segmented text blocks are used for subsequent vector retrieval and model processing.basic unit.

Importance of text chunking

Meet model context constraints

The primary reason for chunking text is to accommodate the hard constraints of two core components in the RAG system:

  • Embedding Model : Responsible for converting text blocks into vectors. This type of model has a strict upper limit on input length. For example, many commonly used embedding models such as bge-base-zh-v1.5) has a context window of 512 tokens. Any text block that exceeds this limit will be truncated when input, resulting in information loss, and the generated vector will not fully represent the semantics of the original text. Therefore, the size of the text blockmustLess than or equal to the context window of the embedded model.
  • Large language model ( LLM ) : Responsible for generating answers based on retrieved context. LLMs also have context window constraints (although typically much larger than embedding models, ranging from a few thousand to millions of tokens). All retrieved blocks of text, along with user questions and prompt words, must fit into this window. If a single block is too large, it may only accommodate a few related blocks, limiting the breadth of information that LLM can refer to when answering questions.
Why isn’t bigger the “block” better?

Assuming that the embedding model can handle up to 8192 tokens, should the chunks be cut as large as possible (e.g. 8000 tokens)? The answer is no.Block size is not always better, too large blocks will seriously affect the performance of the RAG system.

2.2.1 Information loss during embedding process

Most embedding models are based on Transformer encoders. The workflow is roughly as follows:

  • Tokenization : Decompose the input text block into tokens.
  • Vectorization : Transformer iseach token Generate a high-dimensional vector representation.
  • Pooling ( Pooling ) : by some method (such as taking [CLS] Bit vector, average all token vectors mean pooling etc.), convert the vectors of all tokenscompressioninto onesingle vector, this vector represents the semantics of the entire text block.

[CLS] It is a special mark added at the beginning of the input text by Transformer models such as BERT. It dynamically aggregates the contextual information of the entire sequence through a self-attention mechanism, and its final vector is trained to be used as an embedding representing global semantics.

in thiscompressionDuring the process, information loss is inevitable. A 768-dimensional vector needs to summarize all the information of the entire text block.The longer the text block and the more semantic points it contains, the more dilute the information carried by this single vector, causing its representation to become general and key details to be blurred, thus reducing the accuracy of retrieval.

2.2.2 “Lost in the Middle” of the generation process

Even if multiple retrieved large blocks of text are stuffed into the long context window of LLM, there will still be a problem of key information being "drowned" in a large amount of irrelevant content. Studies have shown (1) that when LLM processes a very long context filled with a large amount of information, it tends to remember the information at the beginning and end better, while ignoring the content in the middle.

If the context block provided to LLM is large, complex, and full of noise irrelevant to the question, it will be difficult for the model to extract the most critical information from it to form an answer, resulting in a decrease in answer quality or hallucinations.

2.2.3 Topic dilution leads to retrieval failure

A good text block should focus on a clear, single topic. If a block contains too many irrelevant topics, its semantics will be diluted and it will not be accurately matched during retrieval.

Give me a chestnut🌰:

Suppose there is a strategy document about the hero Luban No. 7 of "Glory of Kings".

  • Bad chunking strategy: Put three completely different themes of "skill introduction", "recommended equipment" and "backstory" into one huge text block.

    • When a player queries "How to equip Luban No. 7?", although this large block contains outfit information, due to being severely diluted by irrelevant topics such as skill descriptions and hero stories, its overall retrieval relevance score may be very low, resulting in the inability to be recalled.

  • Excellent chunking strategy: Divide "skills", "outfits" and "story" into three independent, theme-focused blocks.

    • When the player queries again, the "Recommended Outfits" block will receive a very high score because it is highly relevant to the query, and will be accurately retrieved.

Through reasonable blocking, the signal-to-noise ratio of retrieval can be effectively improved, ensuring that the subsequent generation process can obtain the highest quality and most relevant context.

Basic chunking strategy

Fixed size chunking

This is the simplest and most straightforward method of chunking. According to the LangChain source code, this method works in two main stages:

(1)Split by paragraph: CharacterTextSplitter Use default delimiter "\n\n", use regular expressions to split the text into paragraphs, by _split_text_with_regex Function processing.

(2)Smart merge: Call the function inherited from the parent class _merge_splits Method to merge the divided paragraphs one by one. This method will monitor the cumulative length and when it exceeds chunk_size new blocks are formed and passed through the overlapping mechanism (chunk_overlap) maintains context continuity while warning of overlong blocks when necessary.

Need to pay attention to,CharacterTextSplitter What is actually implemented is not strictly fixed-size chunking. according to _merge_splits Source code logic, this method will:

  • Prioritize paragraph integrity: Only when adding new paragraphs would cause the total length to exceed chunk_size The current block will end when
  • Handle very long paragraphs: If a single paragraph exceeds chunk_size, the system will issue a warning but still retain it as a complete block
  • Apply overlapping mechanism:pass chunk_overlap Parameters keep content overlapping between blocks, ensuring contextual continuity

Therefore, LangChain's implementation should more accurately be called "paragraph-aware adaptive chunking", where the chunk size is dynamically adjusted based on paragraph boundaries.

from langchain.text_splitter import CharacterTextSplitter
from langchain_community.document_loaders import TextLoader

loader = TextLoader("../../data/C2/txt/bee doctor.txt")
docs = loader.load()

text_splitter = CharacterTextSplitter(
    chunk_size=200,    # The target size of each block is100characters
    chunk_overlap=10   # Overlap between each block10characters to alleviate semantic fragmentation
)

chunks = text_splitter.split_documents(docs)

print(f"The text is divided into {len(chunks)} blocks.\n")
print("--- forward5Block content example ---")
for i, chunk in enumerate(chunks[:5]):
    print("=" * 60)
    # chunk is a Document object, need to access its .page_content property to get the text
    print(f'piece {i+1} (length: {len(chunk.page_content)}): "{chunk.page_content}"')

The main advantages of this approach are simple implementation, fast processing and low computational overhead. The disadvantage is that the text may be cut off at semantic boundaries, affecting the integrity and coherence of the content. Practical fixed-size chunking implementation (like LangChain’s CharacterTextSplitter) will usually be combined with delimiters to reduce this problem, preferentially splitting at paragraph boundaries, and only forcing severing when necessary. Therefore, this method still has application value in scenarios such as log analysis and data preprocessing.

recursion character chunking

In the previous chapters, we have tried using RecursiveCharacterTextSplitter The default configuration to handle document chunking. Now let’s take a closer look RecursiveCharacterTextSplitter realization. This chunker improves the processing of very long text by recursively processing delimiter levels, relative to fixed-size chunks.

Algorithm process: (1)Find valid delimiters: Traverse from front to back in the delimiter list and find the first one in the current textexistseparator. If neither exists, the last delimiter is used (usually the empty string "").

(2)Segmentation and classification processing: Split the text using the selected delimiter, then iterate through all segments:

  • If the fragment does not exceed the block size: temporarily save to _good_splits in, ready to merge

  • If the fragment exceeds the block size:

    • First, pass the temporary qualified fragment through _merge_splits Merge into blocks

    • Then, check if there are any remaining delimiters:

      • There are remaining separators: recursive call _split_text Continue to divide
      • No remaining delimiters: Directly reserved as super long block

(3)final processing: Merge the remaining staging fragments into the final chunk

Implementation details:

  • Batch processing mechanism: First collect all qualified fragments (_good_splits), the merge operation is triggered only when an extremely long segment is encountered.
  • recursion Termination condition: The key is if not new_separators judge. When delimiters are exhausted (new_separators is empty), stop the recursion and directly retain the super long fragment. Make sure the algorithm does not recur infinitely.

Key differences from fixed-size chunking:

  • Fixed-size chunking can only issue a warning when encountering an overly long paragraph and retain it.
  • Recursive chunking continues with finer-grained delimiters (sentence → word → character) until the size requirement is met.
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import TextLoader

loader = TextLoader("../../data/C2/txt/bee doctor.txt")
docs = loader.load()

text_splitter = RecursiveCharacterTextSplitter(
    separators=["\n\n", "\n", ".", ", ", " ", ""],  # Delimiter precedence
    chunk_size=200,
    chunk_overlap=10,
)

chunks = text_splitter.split_text(docs)

The principle of recursive character blocking is to use a set of hierarchical delimiters (such as paragraphs, sentences, words) for recursive segmentation, aiming to effectively balance semantic integrity and block size control. exist RecursiveCharacterTextSplitter In its implementation, the chunker first attempts to split the text using the highest priority delimiter (such as a paragraph mark). If the split chunk is still too large, the next priority delimiter (such as a period) is applied to the chunk, and so on until the chunk meets the size limit. This hierarchical processing mechanism can effectively control the block size while maintaining the integrity of the high-level semantic structure as much as possible.

semantic chunking

Semantic chunking (Semantic Chunking) is a more intelligent method that does not rely on a fixed number of characters or preset delimiters, but instead attempts to segment the text based on its semantic connotation. Its core is:Segment where the semantic topic changes significantly. This makes each chunk highly internally semantically consistent. LangChain provides langchain_experimental.text_splitter.SemanticChunker to implement this function.

Implementation principle

SemanticChunker The workflow can be summarized into the following steps:

(1)Sentence Splitting : First, the input text is split into a list of sentences using standard sentence segmentation rules (e.g., based on periods, question marks, exclamation points).

(2)Context-Aware Embedding :This is SemanticChunker a key design. Instead of embedding each sentence independently, this chunker buffer_size parameter (default 1) to capture contextual information. For each sentence in the list, this method associates it with the sentences before and after it. buffer_size sentences, and then embed this temporary, longer combined text. In this way, the final embedding vector for each sentence incorporates the semantics of its context.

(3)Calculate semantic distance (Distance Calculation) : Calculate each pairadjacentCosine distance between embedding vectors of sentences. This distance value quantifies the semantic difference between two sentences - the greater the distance, the weaker the semantic connection and the more obvious the jump.

(4)identify breakpoint ( Breakpoint Identification) : SemanticChunker All calculated distance values ​​are analyzed and based on a statistical method (default is percentile) to determine a dynamic threshold. For example, it might use the 95th percentile value among all distances as the segmentation threshold. All points whose distance is greater than this threshold are identified as semantic "breakpoints".

(5)Merging into Chunks : Finally, the original sentence sequence is segmented according to all identified breakpoint positions, and all sentences within each segmented part are combined to form a final, semantically coherent text block.

breakpoint identification method ( breakpoint_threshold_type )

How to define "significant semantic jump" is the key to semantic chunking.SemanticChunker Several statistics-based methods are provided to identify breakpoints:

  • percentile (Percentile method - Default method):

    • logic: Calculate the semantic difference values ​​of all adjacent sentences and sort these difference values. When a difference value exceeds a certain percentile threshold, the difference value is considered a breakpoint.
    • parameter: breakpoint_threshold_amount (Default is 95), indicating using the 95th percentile as the threshold. This means that only the most significant 5% of semantic difference points will be selected as segmentation points.

  • standard_deviation (standard deviation method):

    • logic: Calculate the mean and standard deviation of all difference values. When a difference value exceeds "mean + N * standard deviation", it is considered an abnormally high jump, that is, a breakpoint.
    • parameter: breakpoint_threshold_amount (Default is 3), indicating using 3 times the standard deviation as the threshold.

  • interquartile (Interquartile range method):

    • logic: Use the interquartile range (IQR) in statistics to identify outliers. When a difference value exceeds Q3 + N * IQR is considered a breakpoint.
    • parameter: breakpoint_threshold_amount (Default is 1.5), indicating using 1.5 times the IQR.

  • gradient (Gradient method):

    • logic: This is a more complex approach. It first calculates the rate of change (gradient) of the difference value and then applies the percentile method to the gradient. It is particularly effective for texts with close semantic connections between sentences and generally low difference values ​​(such as legal and medical documents), because this method can better capture the "inflection point" of semantic changes.
    • parameter: breakpoint_threshold_amount (Default is 95).
Chunking based on document structure

For document formats with clear structural tags (such as Markdown, HTML, LaTex), these tags can be used to achieve smarter and more logical segmentation.

by Markdown Structural chunking as an example

For a clearly structured Markdown document, using its heading hierarchy for chunking is an efficient method that retains rich semantics. LangChain provides MarkdownHeaderTextSplitter to handle.

  • Implementation principle: The main logic of this chunker is "group by title first, then segment as needed".

    • Define splitting rules: The user first needs to provide a title-level mapping relationship, for example [ ("#", "Header 1"), ("##", "Header 2") ], telling the chunker # It is a first-level title,## It is a secondary title.
    • Content aggregation: The chunker goes through the entire document, grouping together all the content under each heading until the next heading of the same or higher level appears. Each aggregated piece of content is given a metadata containing its full title path.

  • Advantages of Metadata Injection: This is the main feature of this method. For example, for an article about machine learning, a certain paragraph might be in "Section 3.2: Evaluation Metrics" under "Chapter 3: Model Evaluation". After segmentation, the metadata of the text block formed by this paragraph will be {"Header 1": "Chapter 3: Model Evaluation", "Header 2": "3.2Section: Evaluation Metrics"}. This metadata provides a precise “address” for each chunk, greatly enhancing contextual accuracy and allowing large models to better understand the origin and context of pieces of information.

  • Limitations and combinations: Splitting simply by title may cause a problem: the content under a certain chapter may be very long, far exceeding the context window that the model can handle. To solve this problem,MarkdownHeaderTextSplitter Can be combined with other chunkers such as RecursiveCharacterTextSplitter)Use in combination. The specific process is:

    • The first step is to use MarkdownHeaderTextSplitter Split the document by title into several large, logical chunks with metadata.
    • In the second step, apply these logical blocks again RecursiveCharacterTextSplitter, further divide it into chunk_size Small pieces as requested. Since this process takes place after the first step, all the resulting small chunks willinheritTitle metadata from step one.

  • RAG application advantages: This two-stage chunking method not only preserves the macroscopic logical structure of the document (through metadata), but also ensures that each chunk is appropriately sized, making it an ideal solution for processing structured documents for RAG.

Unstructured: Intelligent chunking based on document elements

UnstructuredIt is a powerful document processing tool that also provides practicalChunking function.

(1)Partitioning : This is an important function, responsible for parsing original documents (such as PDF, HTML) into a series of structured "elements" (Elements). Each element is tagged with a semantic tag like Title (title),NarrativeText (narrative text),ListItem (list item) etc. This process itself accomplishes a deep understanding and structuring of the document.

(2)Chunking : This function is built onPartitionon top of the results. The chunking function does not operate on plain text, but takes the list of "elements" generated by partitioning as input for intelligent combination. Unstructured provides two main methods of chunking:

  • basic: This is the default method. This method continuously combines document elements (such as paragraphs, list items) until max_characters cap, filling each block as much as possible. If a single element exceeds the upper limit, it will be text-split.
  • by_title: This method is basic On the basis of the method, the perception of "chapters" is added. This method will Title The element is treated as the beginning of a new chapter and forces a new block to start there, ensuring that the same block does not contain content from different chapters. This is very useful when processing structured documents such as reports and books. The effect is similar to LangChain's MarkdownHeaderTextSplitter, but has a wider scope of application.

Unstructured allows chunking to be done in a single call as a parameter to partitioning, or as a separate step after partitioning. This "understand first, segment later" strategy allows Unstructured to retain the original semantic structure of the document to the greatest extent, especially when processing documents with complex layouts. The advantage is particularly obvious.

LlamaIndex: node-oriented parsing and conversion

LlamaIndex Abstract the data processing process into "Node " operation. After the document is loaded, it will first be parsed into a series of "nodes", and chunking is only a part of the node transformation (Transformation).

LlamaIndex's blocking system has the following characteristics:

(1)Rich nodes parser (Node Parser ) : LlamaIndex provides a large number of node parsers for specific data formats and methods, which can be roughly divided into several categories:

  • structure aware: like MarkdownNodeParser, JSONNodeParser, CodeSplitter etc., can understand and segment the source file according to its structure (such as Markdown titles, code functions).

  • Semantic aware:

    • SemanticSplitterNodeParser: with LangChain SemanticChunker Similarly, this parser uses an embedding model to detect semantic "breakpoints" between sentences, cutting where semantic continuity is significantly weakened, so as to make each chunk as internally coherent as possible.
    • SentenceWindowNodeParser: This is a clever approach. This method divides the document into individual sentences, but in the metadata of each sentence node (Node), N adjacent sentences (i.e., "window") are stored. This makes it possible to first use the embedding of a single sentence for exact matching during retrieval, and then send the complete text containing the context "window" to the LLM, which greatly improves the quality of the context.

  • Conventional type: like TokenTextSplitter, SentenceSplitter etc., providing conventional segmentation methods based on the number of Tokens or sentence boundaries.

(2)Flexible conversion pipeline: Users can build a flexible pipeline, such as using MarkdownNodeParser Split the document by chapters and apply it to each chapter node SentenceSplitter Perform more fine-grained sentence-level segmentation. Each node carries rich metadata, recording its origin and context.

(3)good interoperability: LlamaIndex provides LangchainNodeParser, you can easily convert any LangChain TextSplitter The node parser encapsulated as LlamaIndex is seamlessly integrated into its processing flow.

vector embedding

Vector embedding basics

basic concepts

What is Embedding

Vector embedding (Embedding) is a technology that converts complex, high-dimensional data objects in the real world (such as text, images, audio, videos, etc.) into mathematically easy-to-process, low-dimensional, dense, continuous numerical vectors.

Imagine that we place every word, every paragraph, and every picture in a huge multi-dimensional space and give it a unique coordinate. This coordinate is a vector that "embeds" all the key information of the original data. This process is Embedding.

  • data object: Any message, such as the text "Hello World", or a picture of a cat.
  • Embedding model: A deep learning model responsible for receiving data objects and transforming them.
  • Output vector: A fixed-length one-dimensional array, for example [0.16, 0.29, -0.88, ...]. The dimensions (length) of this vector typically range from a few hundred to a few thousand.

vector space semantic representation of

The real significance of Embedding is that the vector it generates is not a pile of random values, but an analysis of the data.Semanticsmathematical coding.

  • core principles: In the vector space constructed by Embedding, the corresponding vectors of semantically similar objects will be closer in the space; while the vectors of semantically unrelated objects will be further apart.

  • key metrics: We usually use the following mathematical methods to measure the "distance" or "similarity" between vectors:

    • cosine similarity ( Cosine Similarity ) : Calculate the cosine of the angle between two vectors. The closer the value is to 1, the more consistent the directions and the more similar the semantics. This is the most commonly used measurement.
    • Dot Product : Calculates the sum of the products of two vectors. After vector normalization, the dot product is equivalent to cosine similarity.
    • Euclidean distance ( Euclidean Distance ) : Calculate the straight-line distance between two vectors in space. The smaller the distance, the more similar the semantics are.

The role of Embedding in RAG

In the RAG process, Embedding plays an irreplaceable and important role.

1.2.1 Basics of semantic retrieval

The "retrieval" link of RAG usually centers on semantic search based on Embedding. The general process is as follows: (1)Offline index building: After segmenting the documents in the knowledge base, use the Embedding model to convert each document chunk (Chunk) into a vector and store it in a special vector database.

(2)Online query and search: used when a user asks a questionsame one The Embedding model also converts the user's question into a vector.

(3)Similarity calculation: In the vector database, calculate the similarity between the "question vector" and all "document block vectors".

(4)recall context: Select the Top-K document blocks with the highest similarity as supplementary contextual information, and send them to the large language model (LLM) together with the original question to generate the final answer.

1.2.2 Key factors that determine search quality

The quality of Embedding directly determines the accuracy and relevance of RAG retrieval and recall content. An excellent embedding model can accurately capture the deep semantic connection between the question and the document, even if the user's question is not completely consistent with the original text. Conversely, a poor embedding model may “contaminate” the context provided to the LLM by recalling irrelevant or erroneous information due to its inability to understand the semantics, resulting in low-quality answers that are ultimately generated.

Embedding technology development

Static word embeddings: context-free representations

  • representative model: Word2Vec (2013), GloVe (2014)
  • Main principles: Generate a fixed, context-free vector for each word in the vocabulary. For example,Word2Vec Through the Skip-gram and CBOW architecture, the local context window is used to learn word vectors, and the semantic capabilities of vector operations are verified (such as king - man + woman ≈ queen).GloVe It integrates the statistical information of the global word-word co-occurrence matrix.
  • limitation: Unable to handle polysemy. In "Apple released a new mobile phone" and "I ate an apple", the word vectors of "apple" are exactly the same, which limits its semantic expression ability in complex contexts.

Dynamic contextual embedding

2017,Transformer The birth of the architecture brings the self-attention mechanism (Self-Attention), which allows the model to dynamically consider the influence of all other words in the sentence when generating a word vector. Based on this, in 2018 BERT Model utilization Transformer The encoder, pre-trained through self-supervised tasks such as masked language models (MLM), generates deep context-sensitive embeddings. The same word will generate different vectors in different contexts, which effectively solves the polysemy problem of static embedding.

RAG’s new requirements for embedded technology

At the beginning, we mentioned that the RAG framework (1) was proposed to solve the problem of large language models. Knowledge solidification(Internal knowledge is difficult to update) and hallucination(The generated content may not be factually correct and cannot be traced back to its source). It dynamically injects external knowledge into LLM through the “retrieval-generation” paradigm. The core of this process is Semantic retrieval, relies heavily on high-quality vector embeddings.

The subsequent rise of RAG has put forward higher and more specific requirements for embedding technology:

  • domain adaptability: General embedding models often perform poorly in professional fields (such as law, medical), which requires the embedding model to have domain adaptation capabilities and be able to adapt to the terminology and semantics of specific fields through fine-tuning or using instructions (such as the INSTRUCTOR model).
  • Multi-granularity and multi-modality support: A RAG system needs to process not just short sentences, but may also include long documents, code, and even images and tables. This requires the embedding model to be able to handle different lengths and types of input data.
  • Search efficiency and hybrid search: The dimensions of the embedding vector and the model size directly affect the storage cost and retrieval speed. At the same time, in order to combine the advantages of semantic similarity (dense retrieval) and keyword matching (sparse retrieval), embedding models that support hybrid retrieval (such as BGE-M3) emerged as the times require, becoming the key to improving recall in some tasks.

Embedded Model Selection Guide

  • horizontal axis -Number of Parameters : represents the size of the model. Generally, a model with a larger number of parameters (further to the right) has stronger potential capabilities, but also has higher requirements on computing resources.
  • Vertical axis - Mean Task Score : Represents the comprehensive performance of the model. This score is the average performance of the model on a series of standard NLP tasks such as classification, clustering, retrieval, etc. The higher the score (the higher it is), the stronger the model’s general semantic understanding ability is.
  • Bubble Size - Embedding Size : Represents the dimension of the model output vector. The larger the bubble, the higher the dimension, which can theoretically encode richer semantic details, but it will also take up more storage and computing resources.
  • Bubble Color - Maximum Processing Length (Max Tokens) : Represents the upper limit of text length that the model can process. The darker the color, the more tokens the model can handle and the better its adaptability to long text.

Index optimization

context extension

In RAG systems, we often face a trade-off problem: using small blocks of text for retrieval can achieve higher accuracy, but small blocks of text lack sufficient context, which may cause the large language model (LLM) to be unable to generate high-quality answers; while using large blocks of text, although rich in context, can easily introduce noise and reduce the relevance of retrieval. In order to solve this contradiction, LlamaIndex proposes a practical indexing strategy——Sentence Window Retrieval (2). This technology cleverly combines the advantages of both methods: it focuses on a highly precise single sentence during retrieval, and intelligently expands the context back to a wider "window" before sending it to LLM to generate answers, thus ensuring both the accuracy of retrieval and the quality of generation.

Main idea

The idea of ​​sentence window retrieval can be summarized as:Index small chunks for retrieval precision and large chunks for contextual richness.

The workflow is as follows:

(1)indexing phase: When building the index, the document is split intosingle sentence. Each sentence is stored in the vector database as an independent "Node". At the same time, each sentence node stores itscontext window, that is, the first N and last N sentences in the original text of the sentence. The text within this window is not indexed and is simply stored as metadata.

(2)retrieval stage: When a user initiates a query, the system willsingle sentence nodePerform a similarity search on. Because a sentence is the smallest unit that expresses complete semantics, this method can very accurately locate the core information most relevant to the user's problem.

(3)Post-processing stage: After retrieving the most relevant sentence nodes, the system will use a file called MetadataReplacementPostProcessor post-processing module. This module will read the metadata of the retrieved sentence node and use thefull context windowto replace the original single sentence content in the node.

(4)generation phase: Finally, these nodes with replaced content and rich context are passed to LLM for generating the final answer.

Hybrid retrieval-dense+sparse retrieval fusion

What is hybrid search

Hybrid Search is a combination of sparse vector ( Sparse Vectors) and Dense Vectors Advantages of advanced search technology. It aims to simultaneously utilize the precise keyword matching capabilities of sparse vectors and the semantic understanding capabilities of dense vectors to overcome the limitations of single vector retrieval, thereby providing more accurate and robust retrieval results in various search scenarios.

sparse vectors vs dense vectors

sparse vector

dense vector

Dense vectors, also often called "semantic vectors", are low-dimensional, dense floating-point representations of data (such as text, images) learned by deep learning models. These vectors aim to capture "semantics" or "concepts" by mapping raw data into a continuous, meaning-filled "semantic space". In an ideal semantic space, the distance and direction between vectors represent the relationship between the concepts they represent. A classic example is vector('king') - vector('man') + vector('woman') The calculation results of are very close in vector space vector('Queen'), which shows that the model has learned the abstract concepts of the two dimensions of "gender" and "royalty". Its representatives include Word2Vec, GloVe, and all Embeddings generated by Transformer-based models (such as BERT, GPT).

Its main advantages are its ability to understand synonyms, synonyms and contextual relationships, its strong generalization ability, and its excellent performance in semantic search tasks. But the shortcomings are equally obvious: poor interpretability (each dimension in the vector usually has no specific physical meaning), a large amount of data and computing power are required for model training, and the processing of out-of-service words (OOV) [^1] is relatively difficult.

OOV ( Out-of-Vocabulary ) unregistered words: Refers to new words that do not appear in the vocabulary during model training but are encountered in actual use. For example, if the word "ChatGPT" is not in the vocabulary when the model is trained, it will be OOV when it is encountered in actual applications. Traditional sparse vector methods (such as BM25) will completely ignore OOV words, while modern dense vector methods can better handle OOV problems through sub-word segmentation (such as BPE, WordPiece).

Hybrid search

From the above, we can see that sparse vectors and dense vectors have their own advantages, so it is a good choice to combine them to achieve complementary advantages. Hybrid retrieval is based on this idea, improving the relevance and recall of search results by combining multiple search algorithms (the most common ones are sparse and dense retrieval).

  • main goal: Solve the limitations of single search technology. For example, keyword retrieval cannot understand semantics, while vector retrieval may ignore keywords that must be accurately matched (such as product model, function name, etc.). Hybrid retrieval aims to exploit both sparse vectorsAccuracyand dense vectorGeneralizability, to cope with complex and changing search needs.

2.1 Technical principles and integration methods

Hybrid retrieval typically executes two retrieval algorithms in parallel and then fuses the two sets of heterogeneous result sets into a unified sorted list. The following are two mainstream integration strategies:

by adjusting α The value can flexibly control the contribution proportion of semantic similarity and keyword matching in the final ranking. For example, in e-commerce search, you can increase the weight of keywords; in intelligent question and answer, you can focus on semantics.

2.2 Advantages and limitations

Advantageslimitations
High recall and accuracy: it can capture keywords and semantics at the same time, which is significantly better than single retrieval.High computing resource consumption: two sets of indexes need to be maintained and queried at the same time.
Strong flexibility: It can be adapted to different business scenarios through integration strategies and weight adjustments.Parameter debugging is complicated: Hyperparameters such as fusion weights require repeated experiments and tuning.
Good fault tolerance: Keyword retrieval can partially compensate for the sensitivity of vector models to spelling errors or rare words.Interpretability remains a challenge: the reasons for ranking the fused results are difficult to analyze intuitively.