Unverified Commit 85df8afd authored by Michael Yao's avatar Michael Yao Committed by GitHub
Browse files

[Docs] Revise frameworks/anything-llm.md (#24489)


Signed-off-by: default avatarwindsonsea <haifeng.yao@daocloud.io>
parent 6aeb1dab
# Anything LLM
# AnythingLLM
[Anything LLM](https://github.com/Mintplex-Labs/anything-llm) is a full-stack application that enables you to turn any document, resource, or piece of content into context that any LLM can use as references during chatting.
[AnythingLLM](https://github.com/Mintplex-Labs/anything-llm) is a full-stack application that enables you to turn any document, resource, or piece of content into context that any LLM can use as references during chatting.
It allows you to deploy a large language model (LLM) server with vLLM as the backend, which exposes OpenAI-compatible endpoints.
## Prerequisites
- Setup vLLM environment
Set up the vLLM environment:
```bash
pip install vllm
```
## Deploy
- Start the vLLM server with the supported chat completion model, e.g.
1. Start the vLLM server with a supported chat-completion model, for example:
```bash
vllm serve Qwen/Qwen1.5-32B-Chat-AWQ --max-model-len 4096
```
```bash
vllm serve Qwen/Qwen1.5-32B-Chat-AWQ --max-model-len 4096
```
1. Download and install [AnythingLLM Desktop](https://anythingllm.com/desktop).
1. Configure the AI provider:
- At the bottom, click the 🔧 wrench icon -> **Open settings** -> **AI Providers** -> **LLM**.
- Enter the following values:
- LLM Provider: Generic OpenAI
- Base URL: `http://{vllm server host}:{vllm server port}/v1`
- Chat Model Name: `Qwen/Qwen1.5-32B-Chat-AWQ`
- Download and install [Anything LLM desktop](https://anythingllm.com/desktop).
![set AI providers](../../assets/deployment/anything-llm-provider.png)
- On the bottom left of open settings, AI Providers --> LLM:
- LLM Provider: Generic OpenAI
- Base URL: http://{vllm server host}:{vllm server port}/v1
- Chat Model Name: `Qwen/Qwen1.5-32B-Chat-AWQ`
1. Create a workspace:
![](../../assets/deployment/anything-llm-provider.png)
1. At the bottom, click the ↺ back icon and back to workspaces.
1. Create a workspace (e.g., `vllm`) and start chatting.
- Back to home page, New Workspace --> create `vllm` workspace, and start to chat:
![create a workspace](../../assets/deployment/anything-llm-chat-without-doc.png)
![](../../assets/deployment/anything-llm-chat-without-doc.png)
1. Add a document.
- Click the upload button:
- upload the doc
- select the doc and move to the workspace
- save and embed
1. Click the 📎 attachment icon.
1. Upload a document.
1. Select and move the document into your workspace.
1. Save and embed it.
![](../../assets/deployment/anything-llm-upload-doc.png)
![add a document](../../assets/deployment/anything-llm-upload-doc.png)
- Chat again:
1. Chat using your document as context.
![](../../assets/deployment/anything-llm-chat-with-doc.png)
![chat with your context](../../assets/deployment/anything-llm-chat-with-doc.png)
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment