Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
OpenDAS
ollama
Commits
39f2bc6b
Unverified
Commit
39f2bc6b
authored
Aug 05, 2024
by
Michael Yang
Committed by
GitHub
Aug 05, 2024
Browse files
Merge pull request #6167 from ollama/mxyng/line-feed
line feed
parents
b73b0940
6a073447
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
83 additions
and
83 deletions
+83
-83
docs/docker.md
docs/docker.md
+71
-71
llm/ext_server/CMakeLists.txt
llm/ext_server/CMakeLists.txt
+12
-12
No files found.
docs/docker.md
View file @
39f2bc6b
# Ollama Docker image
### CPU only
```
bash
docker run
-d
-v
ollama:/root/.ollama
-p
11434:11434
--name
ollama ollama/ollama
```
### Nvidia GPU
Install the
[
NVIDIA Container Toolkit
](
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installation
)
.
#### Install with Apt
1.
Configure the repository
```
bash
curl
-fsSL
https://nvidia.github.io/libnvidia-container/gpgkey
\
|
sudo
gpg
--dearmor
-o
/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl
-s
-L
https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list
\
|
sed
's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g'
\
|
sudo tee
/etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo
apt-get update
```
2.
Install the NVIDIA Container Toolkit packages
```
bash
sudo
apt-get
install
-y
nvidia-container-toolkit
```
#### Install with Yum or Dnf
1.
Configure the repository
```
bash
curl
-s
-L
https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
\
|
sudo tee
/etc/yum.repos.d/nvidia-container-toolkit.repo
```
2.
Install the NVIDIA Container Toolkit packages
```
bash
sudo
yum
install
-y
nvidia-container-toolkit
```
#### Configure Docker to use Nvidia driver
```
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
```
#### Start the container
```
bash
docker run
-d
--gpus
=
all
-v
ollama:/root/.ollama
-p
11434:11434
--name
ollama ollama/ollama
```
### AMD GPU
To run Ollama using Docker with AMD GPUs, use the
`rocm`
tag and the following command:
```
docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:rocm
```
### Run model locally
Now you can run a model:
```
docker exec -it ollama ollama run llama3.1
```
### Try different models
More models can be found on the
[
Ollama library
](
https://ollama.com/library
)
.
# Ollama Docker image
### CPU only
```
bash
docker run
-d
-v
ollama:/root/.ollama
-p
11434:11434
--name
ollama ollama/ollama
```
### Nvidia GPU
Install the
[
NVIDIA Container Toolkit
](
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installation
)
.
#### Install with Apt
1.
Configure the repository
```
bash
curl
-fsSL
https://nvidia.github.io/libnvidia-container/gpgkey
\
|
sudo
gpg
--dearmor
-o
/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl
-s
-L
https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list
\
|
sed
's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g'
\
|
sudo tee
/etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo
apt-get update
```
2.
Install the NVIDIA Container Toolkit packages
```
bash
sudo
apt-get
install
-y
nvidia-container-toolkit
```
#### Install with Yum or Dnf
1.
Configure the repository
```
bash
curl
-s
-L
https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
\
|
sudo tee
/etc/yum.repos.d/nvidia-container-toolkit.repo
```
2.
Install the NVIDIA Container Toolkit packages
```
bash
sudo
yum
install
-y
nvidia-container-toolkit
```
#### Configure Docker to use Nvidia driver
```
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
```
#### Start the container
```
bash
docker run
-d
--gpus
=
all
-v
ollama:/root/.ollama
-p
11434:11434
--name
ollama ollama/ollama
```
### AMD GPU
To run Ollama using Docker with AMD GPUs, use the
`rocm`
tag and the following command:
```
docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:rocm
```
### Run model locally
Now you can run a model:
```
docker exec -it ollama ollama run llama3.1
```
### Try different models
More models can be found on the
[
Ollama library
](
https://ollama.com/library
)
.
llm/ext_server/CMakeLists.txt
View file @
39f2bc6b
set
(
TARGET ollama_llama_server
)
option
(
LLAMA_SERVER_VERBOSE
"Build verbose logging option for Server"
ON
)
include_directories
(
${
CMAKE_CURRENT_SOURCE_DIR
}
)
add_executable
(
${
TARGET
}
server.cpp utils.hpp json.hpp httplib.h
)
install
(
TARGETS
${
TARGET
}
RUNTIME
)
target_compile_definitions
(
${
TARGET
}
PRIVATE
SERVER_VERBOSE=$<BOOL:
${
LLAMA_SERVER_VERBOSE
}
>
)
target_link_libraries
(
${
TARGET
}
PRIVATE ggml llama common llava
${
CMAKE_THREAD_LIBS_INIT
}
)
if
(
WIN32
)
TARGET_LINK_LIBRARIES
(
${
TARGET
}
PRIVATE ws2_32
)
endif
()
set
(
TARGET ollama_llama_server
)
option
(
LLAMA_SERVER_VERBOSE
"Build verbose logging option for Server"
ON
)
include_directories
(
${
CMAKE_CURRENT_SOURCE_DIR
}
)
add_executable
(
${
TARGET
}
server.cpp utils.hpp json.hpp httplib.h
)
install
(
TARGETS
${
TARGET
}
RUNTIME
)
target_compile_definitions
(
${
TARGET
}
PRIVATE
SERVER_VERBOSE=$<BOOL:
${
LLAMA_SERVER_VERBOSE
}
>
)
target_link_libraries
(
${
TARGET
}
PRIVATE ggml llama common llava
${
CMAKE_THREAD_LIBS_INIT
}
)
if
(
WIN32
)
TARGET_LINK_LIBRARIES
(
${
TARGET
}
PRIVATE ws2_32
)
endif
()
target_compile_features
(
${
TARGET
}
PRIVATE cxx_std_11
)
\ No newline at end of file
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment