api.md 4.31 KB
Newer Older
1
2
# API

3
4
5
6
7
8
9
10
## Endpoints

- [Generate a completion](#generate-a-completion)
- [Create a model](#create-a-model)
- [List local models](#list-local-models)
- [Copy a model](#copy-a-model)
- [Delete a model](#delete-a-model)
- [Pull a model](#pull-a-model)
Matt Williams's avatar
Matt Williams committed
11

12
## Conventions
Matt Williams's avatar
Matt Williams committed
13

14
### Model names
Matt Williams's avatar
Matt Williams committed
15

16
Model names follow a `model:tag` format. Some examples are `orca:3b-q4_1` and `llama2:70b`. The tag is optional and if not provided will default to `latest`. The tag is used to identify a specific version.
Matt Williams's avatar
Matt Williams committed
17
18
19

### Durations

20
All durations are returned in nanoseconds.
Matt Williams's avatar
Matt Williams committed
21

22
## Generate a completion
Matt Williams's avatar
Matt Williams committed
23

24
25
26
```
POST /api/generate
```
27

28
Generate a response for a given prompt with a provided model. This is a streaming endpoint, so will be a series of responses. The final response object will include statistics and additional data from the request.
Matt Williams's avatar
Matt Williams committed
29

30
### Parameters
Matt Williams's avatar
Matt Williams committed
31

32
33
- `model`: (required) the [model name](#model-names)
- `prompt`: the prompt to generate a response for
Matt Williams's avatar
Matt Williams committed
34

35
Advanced parameters:
Matt Williams's avatar
Matt Williams committed
36

37
38
39
40
41
- `options`: additional model parameters listed in the documentation for the [Modelfile](./modelfile.md#valid-parameters-and-values) such as `temperature`
- `system`: system prompt to (overrides what is defined in the `Modelfile`)
- `template`: the full prompt or prompt template (overrides what is defined in the `Modelfile`)

### Request
Matt Williams's avatar
Matt Williams committed
42

43
44
45
46
47
48
```
curl -X POST http://localhost:11434/api/generate -d '{
  "model": "llama2:7b",
  "prompt": "Why is the sky blue?"
}'
```
Matt Williams's avatar
Matt Williams committed
49

Matt Williams's avatar
Matt Williams committed
50
51
### Response

52
A stream of JSON objects:
Matt Williams's avatar
Matt Williams committed
53

54
```json
Matt Williams's avatar
Matt Williams committed
55
{
56
57
58
  "model": "llama2:7b",
  "created_at": "2023-08-04T08:52:19.385406455-07:00",
  "response": "The",
Matt Williams's avatar
Matt Williams committed
59
60
61
62
  "done": false
}
```

63
The final response in the stream also includes additional data about the generation:
Matt Williams's avatar
Matt Williams committed
64

65
66
67
68
69
70
71
72
73
74
75
- `total_duration`: time spent generating the response
- `load_duration`: time spent in nanoseconds loading the model
- `sample_count`: number of samples generated
- `sample_duration`: time spent generating samples
- `prompt_eval_count`: number of tokens in the prompt
- `prompt_eval_duration`: time spent in nanoseconds evaluating the prompt
- `eval_count`: number of tokens the response
- `eval_duration`: time in nanoseconds spent generating the response

To calculate how fast the response is generated in tokens per second (token/s), divide `eval_count` / `eval_duration`.

76
```json
Matt Williams's avatar
Matt Williams committed
77
{
78
79
80
81
82
83
84
85
86
87
88
89
  "model": "llama2:7b",
  "created_at": "2023-08-04T19:22:45.499127Z",
  "done": true,
  "total_duration": 5589157167,
  "load_duration": 3013701500,
  "sample_count": 114,
  "sample_duration": 81442000,
  "prompt_eval_count": 46,
  "prompt_eval_duration": 1160282000,
  "eval_count": 113,
  "eval_duration": 1325948000
}
Matt Williams's avatar
Matt Williams committed
90
91
```

92
## Create a Model
Matt Williams's avatar
Matt Williams committed
93

94
95
96
97
98
```
POST /api/create
```

Create a model from a [`Modelfile`](./modelfile.md)
99

100
### Parameters
Matt Williams's avatar
Matt Williams committed
101

102
103
- `name`: name of the model to create
- `path`: path to the Modelfile
Matt Williams's avatar
Matt Williams committed
104
105
106

### Request

107
108
109
110
111
```
curl -X POST http://localhost:11434/api/create -d '{
  "name": "mario",
  "path": "~/Modelfile"
}'
Matt Williams's avatar
Matt Williams committed
112
113
114
115
```

### Response

116
A stream of JSON objects. When finished, `status` is `success`
Matt Williams's avatar
Matt Williams committed
117

118
```json
Matt Williams's avatar
Matt Williams committed
119
120
121
122
123
{
  "status": "parsing modelfile"
}
```

124
## List Local Models
Matt Williams's avatar
Matt Williams committed
125
126

```
127
GET /api/tags
Matt Williams's avatar
Matt Williams committed
128
129
```

130
List models that are available locally.
Matt Williams's avatar
Matt Williams committed
131
132
133

### Request

134
135
136
```
curl http://localhost:11434/api/tags
```
Matt Williams's avatar
Matt Williams committed
137
138
139

### Response

140
```json
Matt Williams's avatar
Matt Williams committed
141
142
143
{
  "models": [
    {
144
145
146
147
148
149
150
      "name": "llama2:7b",
      "modified_at": "2023-08-02T17:02:23.713454393-07:00",
      "size": 3791730596
    },
    {
      "name": "llama2:13b",
      "modified_at": "2023-08-08T12:08:38.093596297-07:00",
Matt Williams's avatar
Matt Williams committed
151
      "size": 7323310500
Matt Williams's avatar
Matt Williams committed
152
153
    }
  ]
Matt Williams's avatar
Matt Williams committed
154
155
156
}
```

157
## Copy a Model
Matt Williams's avatar
Matt Williams committed
158
159

```
160
POST /api/copy
Matt Williams's avatar
Matt Williams committed
161
```
162

163
Copy a model. Creates a model with another name from an existing model.
Matt Williams's avatar
Matt Williams committed
164
165
166
167

### Request

```
168
169
170
curl http://localhost:11434/api/copy -d '{
  "source": "llama2:7b",
  "destination": "llama2-backup"
Matt Williams's avatar
Matt Williams committed
171
172
173
}'
```

Matt Williams's avatar
Matt Williams committed
174
## Delete a Model
Matt Williams's avatar
Matt Williams committed
175

176
177
```
DELETE /api/delete
Matt Williams's avatar
Matt Williams committed
178
179
```

180
Delete a model and its data.
Matt Williams's avatar
Matt Williams committed
181

182
### Parameters
Matt Williams's avatar
Matt Williams committed
183

184
- `model`: model name to delete
Matt Williams's avatar
Matt Williams committed
185

186
### Request
Matt Williams's avatar
Matt Williams committed
187

188
189
190
```
curl -X DELETE http://localhost:11434/api/delete -d '{
  "name": "llama2:13b"
Matt Williams's avatar
Matt Williams committed
191
192
193
}'
```

194
## Pull a Model
Matt Williams's avatar
Matt Williams committed
195

196
197
198
199
200
```
POST /api/pull
```

Download a model from a the model registry. Cancelled pulls are resumed from where they left off, and multiple calls to will share the same download progress.
Matt Williams's avatar
Matt Williams committed
201

202
### Parameters
Matt Williams's avatar
Matt Williams committed
203

204
- `name`: name of the model to pull
Matt Williams's avatar
Matt Williams committed
205
206
207

### Request

208
209
210
211
```
curl -X POST http://localhost:11434/api/pull -d '{
  "name": "llama2:7b"
}'
Matt Williams's avatar
Matt Williams committed
212
213
214
```

### Response
215

216
```json
Matt Williams's avatar
Matt Williams committed
217
{
218
219
220
  "status": "downloading digestname",
  "digest": "digestname",
  "total": 2142590208
Matt Williams's avatar
Matt Williams committed
221
222
}
```