modelfile.md 9.98 KB
Newer Older
Jeffrey Morgan's avatar
Jeffrey Morgan committed
1
2
3
4
5
6
# Ollama Model File

> Note: this model file syntax is in development

A model file is the blueprint to create and share models with Ollama.

Bruce MacDonald's avatar
Bruce MacDonald committed
7
8
9
10
11
12
13
14
## Table of Contents

- [Format](#format)
- [Examples](#examples)
- [Instructions](#instructions)
  - [FROM (Required)](#from-required)
    - [Build from llama2](#build-from-llama2)
    - [Build from a bin file](#build-from-a-bin-file)
Bruce MacDonald's avatar
Bruce MacDonald committed
15
  - [EMBED](#embed)
Bruce MacDonald's avatar
Bruce MacDonald committed
16
17
18
19
20
  - [PARAMETER](#parameter)
    - [Valid Parameters and Values](#valid-parameters-and-values)
  - [TEMPLATE](#template)
    - [Template Variables](#template-variables)
  - [SYSTEM](#system)
Bruce MacDonald's avatar
Bruce MacDonald committed
21
  - [ADAPTER](#adapter)
Bruce MacDonald's avatar
Bruce MacDonald committed
22
23
24
  - [LICENSE](#license)
- [Notes](#notes)

Jeffrey Morgan's avatar
Jeffrey Morgan committed
25
26
27
28
29
30
31
32
33
## Format

The format of the Modelfile:

```modelfile
# comment
INSTRUCTION arguments
```

Bruce MacDonald's avatar
Bruce MacDonald committed
34
35
36
37
38
39
| Instruction                         | Description                                                   |
| ----------------------------------- | ------------------------------------------------------------- |
| [`FROM`](#from-required) (required) | Defines the base model to use.                                |
| [`PARAMETER`](#parameter)           | Sets the parameters for how Ollama will run the model.        |
| [`TEMPLATE`](#template)             | The full prompt template to be sent to the model.             |
| [`SYSTEM`](#system)                 | Specifies the system prompt that will be set in the template. |
Michael Yang's avatar
Michael Yang committed
40
| [`ADAPTER`](#adapter)               | Defines the (Q)LoRA adapters to apply to the model.           |
Bruce MacDonald's avatar
Bruce MacDonald committed
41
| [`LICENSE`](#license)               | Specifies the legal license.                                  |
Jeffrey Morgan's avatar
Jeffrey Morgan committed
42
43
44
45
46
47
48
49
50

## Examples

An example of a model file creating a mario blueprint:

```
FROM llama2
# sets the temperature to 1 [higher is more creative, lower is more coherent]
PARAMETER temperature 1
Bruce MacDonald's avatar
Bruce MacDonald committed
51
# sets the context window size to 4096, this controls how many tokens the LLM can use as context to generate the next token
Jeffrey Morgan's avatar
Jeffrey Morgan committed
52
53
PARAMETER num_ctx 4096

Bruce MacDonald's avatar
Bruce MacDonald committed
54
# sets a custom system prompt to specify the behavior of the chat assistant
Jeffrey Morgan's avatar
Jeffrey Morgan committed
55
56
57
58
59
SYSTEM You are Mario from super mario bros, acting as an assistant.
```

To use this:

60
1. Save it as a file (eg. `Modelfile`)
Jeffrey Morgan's avatar
Jeffrey Morgan committed
61
62
63
64
2. `ollama create NAME -f <location of the file eg. ./Modelfile>'`
3. `ollama run NAME`
4. Start using the model!

Bruce MacDonald's avatar
Bruce MacDonald committed
65
66
67
68
69
More examples are available in the [examples directory](../examples).

## Instructions

### FROM (Required)
Jeffrey Morgan's avatar
Jeffrey Morgan committed
70
71
72
73
74
75
76

The FROM instruction defines the base model to use when creating a model.

```
FROM <model name>:<tag>
```

Bruce MacDonald's avatar
Bruce MacDonald committed
77
#### Build from llama2
Jeffrey Morgan's avatar
Jeffrey Morgan committed
78
79
80
81
82
83
84
85

```
FROM llama2
```

A list of available base models:
<https://github.com/jmorganca/ollama#model-library>

Bruce MacDonald's avatar
Bruce MacDonald committed
86
#### Build from a bin file
Jeffrey Morgan's avatar
Jeffrey Morgan committed
87
88
89
90
91

```
FROM ./ollama-model.bin
```

Bruce MacDonald's avatar
Bruce MacDonald committed
92
93
This bin file location should be specified as an absolute path or relative to the Modelfile location.

Bruce MacDonald's avatar
Bruce MacDonald committed
94
95
### EMBED

Bruce MacDonald's avatar
Bruce MacDonald committed
96
The EMBED instruction is used to add embeddings of files to a model. This is useful for adding custom data that the model can reference when generating an answer. Note that currently only text files are supported, formatted with each line as one embedding.
Bruce MacDonald's avatar
Bruce MacDonald committed
97
98
```
FROM <model name>:<tag>
Bruce MacDonald's avatar
Bruce MacDonald committed
99
100
101
EMBED <file path>.txt
EMBED <different file path>.txt
EMBED <path to directory>/*.txt
Bruce MacDonald's avatar
Bruce MacDonald committed
102
103
```

Bruce MacDonald's avatar
Bruce MacDonald committed
104
### PARAMETER
Jeffrey Morgan's avatar
Jeffrey Morgan committed
105
106
107
108
109
110
111
112
113

The `PARAMETER` instruction defines a parameter that can be set when the model is run.

```
PARAMETER <parameter> <parametervalue>
```

### Valid Parameters and Values

Bruce MacDonald's avatar
Bruce MacDonald committed
114
115
116
117
118
119
| Parameter      | Description                                                                                                                                                                                                                                             | Value Type | Example Usage        |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | -------------------- |
| mirostat       | Enable Mirostat sampling for controlling perplexity. (default: 0, 0 = disabled, 1 = Mirostat, 2 = Mirostat 2.0)                                                                                                                                         | int        | mirostat 0           |
| mirostat_eta   | Influences how quickly the algorithm responds to feedback from the generated text. A lower learning rate will result in slower adjustments, while a higher learning rate will make the algorithm more responsive. (Default: 0.1)                        | float      | mirostat_eta 0.1     |
| mirostat_tau   | Controls the balance between coherence and diversity of the output. A lower value will result in more focused and coherent text. (Default: 5.0)                                                                                                         | float      | mirostat_tau 5.0     |
| num_ctx        | Sets the size of the context window used to generate the next token. (Default: 2048)                                                                                                                                                                    | int        | num_ctx 4096         |
120
| num_gqa        | The number of GQA groups in the transformer layer. Required for some models, for example it is 8 for llama2:70b                                                                                                                                         | int        | num_gqa 1            |
Bruce MacDonald's avatar
Bruce MacDonald committed
121
122
123
124
125
| num_gpu        | The number of GPUs to use. On macOS it defaults to 1 to enable metal support, 0 to disable.                                                                                                                                                             | int        | num_gpu 1            |
| num_thread     | Sets the number of threads to use during computation. By default, Ollama will detect this for optimal performance. It is recommended to set this value to the number of physical CPU cores your system has (as opposed to the logical number of cores). | int        | num_thread 8         |
| repeat_last_n  | Sets how far back for the model to look back to prevent repetition. (Default: 64, 0 = disabled, -1 = num_ctx)                                                                                                                                           | int        | repeat_last_n 64     |
| repeat_penalty | Sets how strongly to penalize repetitions. A higher value (e.g., 1.5) will penalize repetitions more strongly, while a lower value (e.g., 0.9) will be more lenient. (Default: 1.1)                                                                     | float      | repeat_penalty 1.1   |
| temperature    | The temperature of the model. Increasing the temperature will make the model answer more creatively. (Default: 0.8)                                                                                                                                     | float      | temperature 0.7      |
126
| stop           | Sets the stop sequences to use.                                                                                                                                                                                                                         | string     | stop "AI assistant:" |
Bruce MacDonald's avatar
Bruce MacDonald committed
127
128
129
| tfs_z          | Tail free sampling is used to reduce the impact of less probable tokens from the output. A higher value (e.g., 2.0) will reduce the impact more, while a value of 1.0 disables this setting. (default: 1)                                               | float      | tfs_z 1              |
| top_k          | Reduces the probability of generating nonsense. A higher value (e.g. 100) will give more diverse answers, while a lower value (e.g. 10) will be more conservative. (Default: 40)                                                                        | int        | top_k 40             |
| top_p          | Works together with top-k. A higher value (e.g., 0.95) will lead to more diverse text, while a lower value (e.g., 0.5) will generate more focused and conservative text. (Default: 0.9)                                                                 | float      | top_p 0.9            |
Jeffrey Morgan's avatar
Jeffrey Morgan committed
130

Bruce MacDonald's avatar
Bruce MacDonald committed
131
132
### TEMPLATE

Bruce MacDonald's avatar
Bruce MacDonald committed
133
`TEMPLATE` of the full prompt template to be passed into the model. It may include (optionally) a system prompt and a user's prompt. This is used to create a full custom prompt, and syntax may be model specific.
Bruce MacDonald's avatar
Bruce MacDonald committed
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151

#### Template Variables

| Variable        | Description                                                                                                  |
| --------------- | ------------------------------------------------------------------------------------------------------------ |
| `{{ .System }}` | The system prompt used to specify custom behavior, this must also be set in the Modelfile as an instruction. |
| `{{ .Prompt }}` | The incoming prompt, this is not specified in the model file and will be set based on input.                 |
| `{{ .First }}`  | A boolean value used to render specific template information for the first generation of a session.          |

```
TEMPLATE """
{{- if .First }}
### System:
{{ .System }}
{{- end }}

### User:
{{ .Prompt }}
Jeffrey Morgan's avatar
Jeffrey Morgan committed
152

Bruce MacDonald's avatar
Bruce MacDonald committed
153
154
### Response:
"""
Jeffrey Morgan's avatar
Jeffrey Morgan committed
155

Bruce MacDonald's avatar
Bruce MacDonald committed
156
157
158
159
160
161
162
163
164
165
166
SYSTEM """<system message>"""
```

### SYSTEM

The `SYSTEM` instruction specifies the system prompt to be used in the template, if applicable.

```
SYSTEM """<system message>"""
```

Michael Yang's avatar
Michael Yang committed
167
168
169
170
171
172
173
174
### ADAPTER

The `ADAPTER` instruction specifies the LoRA adapter to apply to the base model. The value of this instruction should be an absolute path or a path relative to the Modelfile and the file must be in a GGML file format. The adapter should be tuned from the base model otherwise the behaviour is undefined.

```
ADAPTER ./ollama-lora.bin
```

Bruce MacDonald's avatar
Bruce MacDonald committed
175
176
### LICENSE

Michael Yang's avatar
Michael Yang committed
177
The `LICENSE` instruction allows you to specify the legal license under which the model used with this Modelfile is shared or distributed.
Jeffrey Morgan's avatar
Jeffrey Morgan committed
178

Bruce MacDonald's avatar
Bruce MacDonald committed
179
180
181
182
183
```
LICENSE """
<license text>
"""
```
Jeffrey Morgan's avatar
Jeffrey Morgan committed
184
185
186
187

## Notes

- the **modelfile is not case sensitive**. In the examples, we use uppercase for instructions to make it easier to distinguish it from arguments.
Michael Yang's avatar
Michael Yang committed
188
- Instructions can be in any order. In the examples, we start with FROM instruction to keep it easily readable.