LlamaIndex.rst

LlamaIndex
==========

.. attention:: 
    To be updated for Qwen3.

To connect Qwen2.5 with external data, such as documents, web pages, etc., we offer a tutorial on `LlamaIndex <https://www.llamaindex.ai/>`__.
This guide helps you quickly implement retrieval-augmented generation (RAG) using LlamaIndex with Qwen2.5.

Preparation
--------------------------------------

To implement RAG, 
we advise you to install the LlamaIndex-related packages first. 

The following is a simple code snippet showing how to do this:

.. code:: bash

   pip install llama-index
   pip install llama-index-llms-huggingface
   pip install llama-index-readers-web

Set Parameters
--------------------------------------

Now we can set up LLM, embedding model, and the related configurations.  
Qwen2.5-Instruct supports conversations in multiple languages, including English and Chinese.
You can use the ``bge-base-en-v1.5`` model to retrieve from English documents, and you can download the ``bge-base-zh-v1.5`` model to retrieve from Chinese documents. 
You can also choose ``bge-large`` or ``bge-small`` as the embedding model or modify the context window size or text chunk size depending on your computing resources.
Qwen2.5 model families support a maximum of 32K context window size (up to 128K for 7B, 14B, 32B, and 72B, requiring extra configuration)

.. code:: python
  
    import torch
    from llama_index.core import Settings
    from llama_index.core.node_parser import SentenceSplitter
    from llama_index.llms.huggingface import HuggingFaceLLM
    from llama_index.embeddings.huggingface import HuggingFaceEmbedding

    # Set prompt template for generation (optional)
    from llama_index.core import PromptTemplate
  
    def completion_to_prompt(completion):
       return f"<|im_start|>system\n<|im_end|>\n<|im_start|>user\n{completion}<|im_end|>\n<|im_start|>assistant\n"
    
    def messages_to_prompt(messages):
        prompt = ""
        for message in messages:
            if message.role == "system":
                prompt += f"<|im_start|>system\n{message.content}<|im_end|>\n"
            elif message.role == "user":
                prompt += f"<|im_start|>user\n{message.content}<|im_end|>\n"
            elif message.role == "assistant":
                prompt += f"<|im_start|>assistant\n{message.content}<|im_end|>\n"
    
        if not prompt.startswith("<|im_start|>system"):
            prompt = "<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n" + prompt
    
        prompt = prompt + "<|im_start|>assistant\n"
    
        return prompt
    
    # Set Qwen2.5 as the language model and set generation config
    Settings.llm = HuggingFaceLLM(
        model_name="Qwen/Qwen2.5-7B-Instruct",
        tokenizer_name="Qwen/Qwen2.5-7B-Instruct",
        context_window=30000,
        max_new_tokens=2000,
        generate_kwargs={"temperature": 0.7, "top_k": 50, "top_p": 0.95},
        messages_to_prompt=messages_to_prompt,
        completion_to_prompt=completion_to_prompt,
        device_map="auto",
    )

    # Set embedding model                       
    Settings.embed_model = HuggingFaceEmbedding(
        model_name = "BAAI/bge-base-en-v1.5"
    )

    # Set the size of the text chunk for retrieval
    Settings.transformations = [SentenceSplitter(chunk_size=1024)]

Build Index
--------------------------------------

Now we can build index from documents or websites.

The following code snippet demonstrates how to build an index for files (regardless of whether they are in PDF or TXT format) in a local folder named 'document'.                               

.. code:: python
    
    from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
    
    documents = SimpleDirectoryReader("./document").load_data()
    index = VectorStoreIndex.from_documents(
        documents,
        embed_model=Settings.embed_model,
        transformations=Settings.transformations
    )

The following code snippet demonstrates how to build an index for the content in a list of websites.                               
                               
.. code:: python
                               
    from llama_index.readers.web import SimpleWebPageReader
    from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
    
    documents = SimpleWebPageReader(html_to_text=True).load_data(
        ["web_address_1","web_address_2",...]
    )
    index = VectorStoreIndex.from_documents(
        documents,
        embed_model=Settings.embed_model, 
        transformations=Settings.transformations
    )

To save and load the index, you can use the following code snippet.                              

.. code:: python

    from llama_index.core import StorageContext, load_index_from_storage

    # save index
    storage_context = StorageContext.from_defaults(persist_dir="save")
    
    # load index
    index = load_index_from_storage(storage_context)
                            
                               
RAG
-------------------

Now you can perform queries, and Qwen2.5 will answer based on the content of the indexed documents.                               
                               
.. code:: python

  query_engine = index.as_query_engine()
  your_query = "<your query here>"                             
  print(query_engine.query(your_query).response)