Unverified Commit 85df8afd authored by Michael Yao's avatar Michael Yao Committed by GitHub
Browse files

[Docs] Revise frameworks/anything-llm.md (#24489)


Signed-off-by: default avatarwindsonsea <haifeng.yao@daocloud.io>
parent 6aeb1dab
# Anything LLM # AnythingLLM
[Anything LLM](https://github.com/Mintplex-Labs/anything-llm) is a full-stack application that enables you to turn any document, resource, or piece of content into context that any LLM can use as references during chatting. [AnythingLLM](https://github.com/Mintplex-Labs/anything-llm) is a full-stack application that enables you to turn any document, resource, or piece of content into context that any LLM can use as references during chatting.
It allows you to deploy a large language model (LLM) server with vLLM as the backend, which exposes OpenAI-compatible endpoints. It allows you to deploy a large language model (LLM) server with vLLM as the backend, which exposes OpenAI-compatible endpoints.
## Prerequisites ## Prerequisites
- Setup vLLM environment Set up the vLLM environment:
```bash
pip install vllm
```
## Deploy ## Deploy
- Start the vLLM server with the supported chat completion model, e.g. 1. Start the vLLM server with a supported chat-completion model, for example:
```bash ```bash
vllm serve Qwen/Qwen1.5-32B-Chat-AWQ --max-model-len 4096 vllm serve Qwen/Qwen1.5-32B-Chat-AWQ --max-model-len 4096
``` ```
1. Download and install [AnythingLLM Desktop](https://anythingllm.com/desktop).
1. Configure the AI provider:
- At the bottom, click the 🔧 wrench icon -> **Open settings** -> **AI Providers** -> **LLM**.
- Enter the following values:
- LLM Provider: Generic OpenAI
- Base URL: `http://{vllm server host}:{vllm server port}/v1`
- Chat Model Name: `Qwen/Qwen1.5-32B-Chat-AWQ`
- Download and install [Anything LLM desktop](https://anythingllm.com/desktop). ![set AI providers](../../assets/deployment/anything-llm-provider.png)
- On the bottom left of open settings, AI Providers --> LLM: 1. Create a workspace:
- LLM Provider: Generic OpenAI
- Base URL: http://{vllm server host}:{vllm server port}/v1
- Chat Model Name: `Qwen/Qwen1.5-32B-Chat-AWQ`
![](../../assets/deployment/anything-llm-provider.png) 1. At the bottom, click the ↺ back icon and back to workspaces.
1. Create a workspace (e.g., `vllm`) and start chatting.
- Back to home page, New Workspace --> create `vllm` workspace, and start to chat: ![create a workspace](../../assets/deployment/anything-llm-chat-without-doc.png)
![](../../assets/deployment/anything-llm-chat-without-doc.png) 1. Add a document.
- Click the upload button: 1. Click the 📎 attachment icon.
- upload the doc 1. Upload a document.
- select the doc and move to the workspace 1. Select and move the document into your workspace.
- save and embed 1. Save and embed it.
![](../../assets/deployment/anything-llm-upload-doc.png) ![add a document](../../assets/deployment/anything-llm-upload-doc.png)
- Chat again: 1. Chat using your document as context.
![](../../assets/deployment/anything-llm-chat-with-doc.png) ![chat with your context](../../assets/deployment/anything-llm-chat-with-doc.png)
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment