Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
OpenDAS
ollama
Commits
39f2bc6b
Unverified
Commit
39f2bc6b
authored
Aug 05, 2024
by
Michael Yang
Committed by
GitHub
Aug 05, 2024
Browse files
Merge pull request #6167 from ollama/mxyng/line-feed
line feed
parents
b73b0940
6a073447
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
83 additions
and
83 deletions
+83
-83
docs/docker.md
docs/docker.md
+71
-71
llm/ext_server/CMakeLists.txt
llm/ext_server/CMakeLists.txt
+12
-12
No files found.
docs/docker.md
View file @
39f2bc6b
# Ollama Docker image
# Ollama Docker image
### CPU only
### CPU only
```
bash
```
bash
docker run
-d
-v
ollama:/root/.ollama
-p
11434:11434
--name
ollama ollama/ollama
docker run
-d
-v
ollama:/root/.ollama
-p
11434:11434
--name
ollama ollama/ollama
```
```
### Nvidia GPU
### Nvidia GPU
Install the
[
NVIDIA Container Toolkit
](
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installation
)
.
Install the
[
NVIDIA Container Toolkit
](
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installation
)
.
#### Install with Apt
#### Install with Apt
1.
Configure the repository
1.
Configure the repository
```
bash
```
bash
curl
-fsSL
https://nvidia.github.io/libnvidia-container/gpgkey
\
curl
-fsSL
https://nvidia.github.io/libnvidia-container/gpgkey
\
|
sudo
gpg
--dearmor
-o
/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
|
sudo
gpg
--dearmor
-o
/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl
-s
-L
https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list
\
curl
-s
-L
https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list
\
|
sed
's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g'
\
|
sed
's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g'
\
|
sudo tee
/etc/apt/sources.list.d/nvidia-container-toolkit.list
|
sudo tee
/etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo
apt-get update
sudo
apt-get update
```
```
2.
Install the NVIDIA Container Toolkit packages
2.
Install the NVIDIA Container Toolkit packages
```
bash
```
bash
sudo
apt-get
install
-y
nvidia-container-toolkit
sudo
apt-get
install
-y
nvidia-container-toolkit
```
```
#### Install with Yum or Dnf
#### Install with Yum or Dnf
1.
Configure the repository
1.
Configure the repository
```
bash
```
bash
curl
-s
-L
https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
\
curl
-s
-L
https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
\
|
sudo tee
/etc/yum.repos.d/nvidia-container-toolkit.repo
|
sudo tee
/etc/yum.repos.d/nvidia-container-toolkit.repo
```
```
2.
Install the NVIDIA Container Toolkit packages
2.
Install the NVIDIA Container Toolkit packages
```
bash
```
bash
sudo
yum
install
-y
nvidia-container-toolkit
sudo
yum
install
-y
nvidia-container-toolkit
```
```
#### Configure Docker to use Nvidia driver
#### Configure Docker to use Nvidia driver
```
```
sudo nvidia-ctk runtime configure --runtime=docker
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
sudo systemctl restart docker
```
```
#### Start the container
#### Start the container
```
bash
```
bash
docker run
-d
--gpus
=
all
-v
ollama:/root/.ollama
-p
11434:11434
--name
ollama ollama/ollama
docker run
-d
--gpus
=
all
-v
ollama:/root/.ollama
-p
11434:11434
--name
ollama ollama/ollama
```
```
### AMD GPU
### AMD GPU
To run Ollama using Docker with AMD GPUs, use the
`rocm`
tag and the following command:
To run Ollama using Docker with AMD GPUs, use the
`rocm`
tag and the following command:
```
```
docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:rocm
docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:rocm
```
```
### Run model locally
### Run model locally
Now you can run a model:
Now you can run a model:
```
```
docker exec -it ollama ollama run llama3.1
docker exec -it ollama ollama run llama3.1
```
```
### Try different models
### Try different models
More models can be found on the
[
Ollama library
](
https://ollama.com/library
)
.
More models can be found on the
[
Ollama library
](
https://ollama.com/library
)
.
llm/ext_server/CMakeLists.txt
View file @
39f2bc6b
set
(
TARGET ollama_llama_server
)
set
(
TARGET ollama_llama_server
)
option
(
LLAMA_SERVER_VERBOSE
"Build verbose logging option for Server"
ON
)
option
(
LLAMA_SERVER_VERBOSE
"Build verbose logging option for Server"
ON
)
include_directories
(
${
CMAKE_CURRENT_SOURCE_DIR
}
)
include_directories
(
${
CMAKE_CURRENT_SOURCE_DIR
}
)
add_executable
(
${
TARGET
}
server.cpp utils.hpp json.hpp httplib.h
)
add_executable
(
${
TARGET
}
server.cpp utils.hpp json.hpp httplib.h
)
install
(
TARGETS
${
TARGET
}
RUNTIME
)
install
(
TARGETS
${
TARGET
}
RUNTIME
)
target_compile_definitions
(
${
TARGET
}
PRIVATE
target_compile_definitions
(
${
TARGET
}
PRIVATE
SERVER_VERBOSE=$<BOOL:
${
LLAMA_SERVER_VERBOSE
}
>
SERVER_VERBOSE=$<BOOL:
${
LLAMA_SERVER_VERBOSE
}
>
)
)
target_link_libraries
(
${
TARGET
}
PRIVATE ggml llama common llava
${
CMAKE_THREAD_LIBS_INIT
}
)
target_link_libraries
(
${
TARGET
}
PRIVATE ggml llama common llava
${
CMAKE_THREAD_LIBS_INIT
}
)
if
(
WIN32
)
if
(
WIN32
)
TARGET_LINK_LIBRARIES
(
${
TARGET
}
PRIVATE ws2_32
)
TARGET_LINK_LIBRARIES
(
${
TARGET
}
PRIVATE ws2_32
)
endif
()
endif
()
target_compile_features
(
${
TARGET
}
PRIVATE cxx_std_11
)
target_compile_features
(
${
TARGET
}
PRIVATE cxx_std_11
)
\ No newline at end of file
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment