api.md 13.1 KB
Newer Older
1
2
# API

3
4
5
## Endpoints

- [Generate a completion](#generate-a-completion)
Matt Williams's avatar
Matt Williams committed
6
7
8
9
10
11
12
13
14
- [Create a Model](#create-a-model)
- [List Local Models](#list-local-models)
- [Show Model Information](#show-model-information)
- [Copy a Model](#copy-a-model)
- [Delete a Model](#delete-a-model)
- [Pull a Model](#pull-a-model)
- [Push a Model](#push-a-model)
- [Generate Embeddings](#generate-embeddings)

15
## Conventions
Matt Williams's avatar
Matt Williams committed
16

17
### Model names
Matt Williams's avatar
Matt Williams committed
18

Matt Williams's avatar
Matt Williams committed
19
Model names follow a `model:tag` format. Some examples are `orca-mini:3b-q4_1` and `llama2:70b`. The tag is optional and, if not provided, will default to `latest`. The tag is used to identify a specific version.
Matt Williams's avatar
Matt Williams committed
20
21
22

### Durations

23
All durations are returned in nanoseconds.
Matt Williams's avatar
Matt Williams committed
24

25
26
27
28
### Streaming responses

Certain endpoints stream responses as JSON objects delineated with the newline (`\n`) character.

29
## Generate a completion
Matt Williams's avatar
Matt Williams committed
30

Matt Williams's avatar
Matt Williams committed
31
```shell
32
33
POST /api/generate
```
34

35
Generate a response for a given prompt with a provided model. This is a streaming endpoint, so will be a series of responses. The final response object will include statistics and additional data from the request.
Matt Williams's avatar
Matt Williams committed
36

37
### Parameters
Matt Williams's avatar
Matt Williams committed
38

39
40
- `model`: (required) the [model name](#model-names)
- `prompt`: the prompt to generate a response for
Matt Williams's avatar
Matt Williams committed
41

42
Advanced parameters (optional):
Matt Williams's avatar
Matt Williams committed
43

44
45
46
- `options`: additional model parameters listed in the documentation for the [Modelfile](./modelfile.md#valid-parameters-and-values) such as `temperature`
- `system`: system prompt to (overrides what is defined in the `Modelfile`)
- `template`: the full prompt or prompt template (overrides what is defined in the `Modelfile`)
Bruce MacDonald's avatar
Bruce MacDonald committed
47
- `context`: the context parameter returned from a previous request to `/generate`, this can be used to keep a short conversational memory
48
- `stream`: if `false` the response will be returned as a single response object, rather than a stream of objects
49
- `raw`: if `true` no formatting will be applied to the prompt and no context will be returned. You may choose to use the `raw` parameter if you are specifying a full templated prompt in your request to the API, and are managing history yourself.
50

51
52
53
### Examples

#### Request
Matt Williams's avatar
Matt Williams committed
54

Matt Williams's avatar
Matt Williams committed
55
```shell
56
57
58
59
60
curl -X POST http://localhost:11434/api/generate -d '{
  "model": "llama2:7b",
  "prompt": "Why is the sky blue?"
}'
```
Matt Williams's avatar
Matt Williams committed
61

62
#### Response
Matt Williams's avatar
Matt Williams committed
63

64
A stream of JSON objects is returned:
Matt Williams's avatar
Matt Williams committed
65

66
```json
Matt Williams's avatar
Matt Williams committed
67
{
68
69
70
  "model": "llama2:7b",
  "created_at": "2023-08-04T08:52:19.385406455-07:00",
  "response": "The",
Matt Williams's avatar
Matt Williams committed
71
72
73
74
  "done": false
}
```

75
The final response in the stream also includes additional data about the generation:
Matt Williams's avatar
Matt Williams committed
76

77
78
79
80
81
82
83
84
- `total_duration`: time spent generating the response
- `load_duration`: time spent in nanoseconds loading the model
- `sample_count`: number of samples generated
- `sample_duration`: time spent generating samples
- `prompt_eval_count`: number of tokens in the prompt
- `prompt_eval_duration`: time spent in nanoseconds evaluating the prompt
- `eval_count`: number of tokens the response
- `eval_duration`: time in nanoseconds spent generating the response
Bruce MacDonald's avatar
Bruce MacDonald committed
85
- `context`: an encoding of the conversation used in this response, this can be sent in the next request to keep a conversational memory
86
- `response`: empty if the response was streamed, if not streamed, this will contain the full response
87
88
89

To calculate how fast the response is generated in tokens per second (token/s), divide `eval_count` / `eval_duration`.

90
```json
Matt Williams's avatar
Matt Williams committed
91
{
92
93
  "model": "llama2:7b",
  "created_at": "2023-08-04T19:22:45.499127Z",
94
  "response": "",
Bruce MacDonald's avatar
Bruce MacDonald committed
95
  "context": [1, 2, 3],
96
97
98
99
100
101
102
103
104
105
  "done": true,
  "total_duration": 5589157167,
  "load_duration": 3013701500,
  "sample_count": 114,
  "sample_duration": 81442000,
  "prompt_eval_count": 46,
  "prompt_eval_duration": 1160282000,
  "eval_count": 113,
  "eval_duration": 1325948000
}
Matt Williams's avatar
Matt Williams committed
106
107
```

108
109
110
111
112
113
114
115
116
117
118
119
#### Request

```shell
curl -X POST http://localhost:11434/api/generate -d '{
  "model": "llama2:7b",
  "prompt": "Why is the sky blue?",
  "stream": false
}'
```

#### Response

120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
If `stream` is set to `false`, the response will be a single JSON object:

```json
{
  "model": "llama2:7b",
  "created_at": "2023-08-04T19:22:45.499127Z",
  "response": "The sky is blue because it is the color of the sky.",
  "context": [1, 2, 3],
  "done": true,
  "total_duration": 5589157167,
  "load_duration": 3013701500,
  "sample_count": 114,
  "sample_duration": 81442000,
  "prompt_eval_count": 46,
  "prompt_eval_duration": 1160282000,
  "eval_count": 13,
  "eval_duration": 1325948000
}
```

140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
#### Request

In some cases you may wish to bypass the templating system and provide a full prompt. In this case, you can use the `raw` parameter to disable formatting and context.

```shell
curl -X POST http://localhost:11434/api/generate -d '{
  "model": "mistral",
  "prompt": "[INST] why is the sky blue? [/INST]",
  "raw": true,
  "stream": false
}'
```

#### Response

```json
{
  "model": "mistral",
  "created_at": "2023-11-03T15:36:02.583064Z",
  "response": " The sky appears blue because of a phenomenon called Rayleigh scattering.",
  "done": true,
  "total_duration": 14648695333,
  "load_duration": 3302671417,
  "prompt_eval_count": 14,
  "prompt_eval_duration": 286243000,
  "eval_count": 129,
  "eval_duration": 10931424000
}
```

170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
#### Request

If you want to set custom options for the model at runtime rather than in the Modelfile, you can do so with the `options` parameter. This example sets every available option, but you can set any of them individually and omit the ones you do not want to override.

```shell
curl -X POST http://localhost:11434/api/generate -d '{
  "model": "llama2:7b",
  "prompt": "Why is the sky blue?",
  "stream": false,
  "options": {
    "num_keep": 5,
    "seed": 42,
    "num_predict": 100,
    "top_k": 20,
    "top_p": 0.9,
    "tfs_z": 0.5,
    "typical_p": 0.7,
    "repeat_last_n": 33,
    "temperature": 0.8,
    "repeat_penalty": 1.2,
    "presence_penalty": 1.5,
    "frequency_penalty": 1.0,
    "mirostat": 1,
    "mirostat_tau": 0.8,
    "mirostat_eta": 0.6,
    "penalize_newline": true,
    "stop": ["\n", "user:"],
    "numa": false,
    "num_ctx": 4,
    "num_batch": 2,
    "num_gqa": 1,
    "num_gpu": 1,
    "main_gpu": 0,
    "low_vram": false,
    "f16_kv": true,
    "logits_all": false,
    "vocab_only": false,
    "use_mmap": true,
    "use_mlock": false,
    "embedding_only": false,
    "rope_frequency_base": 1.1,
    "rope_frequency_scale": 0.8,
    "num_thread": 8
    }
}'
```

#### Response

```json
{
  "model": "llama2:7b",
  "created_at": "2023-08-04T19:22:45.499127Z",
  "response": "The sky is blue because it is the color of the sky.",
  "context": [1, 2, 3],
  "done": true,
  "total_duration": 5589157167,
  "load_duration": 3013701500,
  "sample_count": 114,
  "sample_duration": 81442000,
  "prompt_eval_count": 46,
  "prompt_eval_duration": 1160282000,
  "eval_count": 13,
  "eval_duration": 1325948000
}
```

237
## Create a Model
Matt Williams's avatar
Matt Williams committed
238

Matt Williams's avatar
Matt Williams committed
239
```shell
240
241
242
243
POST /api/create
```

Create a model from a [`Modelfile`](./modelfile.md)
244

245
### Parameters
Matt Williams's avatar
Matt Williams committed
246

247
248
- `name`: name of the model to create
- `path`: path to the Modelfile
249
- `stream`: (optional) if `false` the response will be returned as a single response object, rather than a stream of objects
Matt Williams's avatar
Matt Williams committed
250

251
252
253
### Examples

#### Request
Matt Williams's avatar
Matt Williams committed
254

Matt Williams's avatar
Matt Williams committed
255
```shell
256
257
258
259
curl -X POST http://localhost:11434/api/create -d '{
  "name": "mario",
  "path": "~/Modelfile"
}'
Matt Williams's avatar
Matt Williams committed
260
261
```

262
#### Response
Matt Williams's avatar
Matt Williams committed
263

Matt Williams's avatar
Matt Williams committed
264
A stream of JSON objects. When finished, `status` is `success`.
Matt Williams's avatar
Matt Williams committed
265

266
```json
Matt Williams's avatar
Matt Williams committed
267
268
269
270
271
{
  "status": "parsing modelfile"
}
```

272
## List Local Models
Matt Williams's avatar
Matt Williams committed
273

Matt Williams's avatar
Matt Williams committed
274
```shell
275
GET /api/tags
Matt Williams's avatar
Matt Williams committed
276
277
```

278
List models that are available locally.
Matt Williams's avatar
Matt Williams committed
279

280
281
282
### Examples

#### Request
Matt Williams's avatar
Matt Williams committed
283

Matt Williams's avatar
Matt Williams committed
284
```shell
285
286
curl http://localhost:11434/api/tags
```
Matt Williams's avatar
Matt Williams committed
287

288
#### Response
Matt Williams's avatar
Matt Williams committed
289

290
291
A single JSON object will be returned.

292
```json
Matt Williams's avatar
Matt Williams committed
293
294
295
{
  "models": [
    {
296
297
298
299
300
301
302
      "name": "llama2:7b",
      "modified_at": "2023-08-02T17:02:23.713454393-07:00",
      "size": 3791730596
    },
    {
      "name": "llama2:13b",
      "modified_at": "2023-08-08T12:08:38.093596297-07:00",
Matt Williams's avatar
Matt Williams committed
303
      "size": 7323310500
Matt Williams's avatar
Matt Williams committed
304
305
    }
  ]
Matt Williams's avatar
Matt Williams committed
306
307
308
}
```

Matt Williams's avatar
Matt Williams committed
309
310
311
312
313
314
315
316
317
318
319
## Show Model Information

```shell
POST /api/show
```

Show details about a model including modelfile, template, parameters, license, and system prompt.

### Parameters

- `name`: name of the model to show
Matt Williams's avatar
Matt Williams committed
320

321
322
323
### Examples

#### Request
Matt Williams's avatar
Matt Williams committed
324

325
```shell
Matt Williams's avatar
Matt Williams committed
326
327
328
curl http://localhost:11434/api/show -d '{
  "name": "llama2:7b"
}'
Matt Williams's avatar
Matt Williams committed
329
```
Matt Williams's avatar
Matt Williams committed
330

331
#### Response
Matt Williams's avatar
Matt Williams committed
332
333
334

```json
{
335
336
337
338
  "license": "<contents of license block>",
  "modelfile": "# Modelfile generated by \"ollama show\"\n# To build a new Modelfile based on this one, replace the FROM line with:\n# FROM llama2:latest\n\nFROM /Users/username/.ollama/models/blobs/sha256:8daa9615cce30c259a9555b1cc250d461d1bc69980a274b44d7eda0be78076d8\nTEMPLATE \"\"\"[INST] {{ if and .First .System }}<<SYS>>{{ .System }}<</SYS>>\n\n{{ end }}{{ .Prompt }} [/INST] \"\"\"\nSYSTEM \"\"\"\"\"\"\nPARAMETER stop [INST]\nPARAMETER stop [/INST]\nPARAMETER stop <<SYS>>\nPARAMETER stop <</SYS>>\n",
  "parameters": "stop                           [INST]\nstop                           [/INST]\nstop                           <<SYS>>\nstop                           <</SYS>>",
  "template": "[INST] {{ if and .First .System }}<<SYS>>{{ .System }}<</SYS>>\n\n{{ end }}{{ .Prompt }} [/INST] "
Matt Williams's avatar
Matt Williams committed
339
340
341
342
343
344
}
```

## Copy a Model

```shell
345
POST /api/copy
Matt Williams's avatar
Matt Williams committed
346
```
347

348
Copy a model. Creates a model with another name from an existing model.
Matt Williams's avatar
Matt Williams committed
349

350
351
352
### Examples

#### Request
Matt Williams's avatar
Matt Williams committed
353

Matt Williams's avatar
Matt Williams committed
354
```shell
355
356
357
curl http://localhost:11434/api/copy -d '{
  "source": "llama2:7b",
  "destination": "llama2-backup"
Matt Williams's avatar
Matt Williams committed
358
359
360
}'
```

361
#### Response
362
363
364

The only response is a 200 OK if successful.

Matt Williams's avatar
Matt Williams committed
365
## Delete a Model
Matt Williams's avatar
Matt Williams committed
366

Matt Williams's avatar
Matt Williams committed
367
```shell
368
DELETE /api/delete
Matt Williams's avatar
Matt Williams committed
369
370
```

371
Delete a model and its data.
Matt Williams's avatar
Matt Williams committed
372

373
### Parameters
Matt Williams's avatar
Matt Williams committed
374

375
- `name`: model name to delete
Matt Williams's avatar
Matt Williams committed
376

377
378
379
### Examples

#### Request
Matt Williams's avatar
Matt Williams committed
380

Matt Williams's avatar
Matt Williams committed
381
```shell
382
383
curl -X DELETE http://localhost:11434/api/delete -d '{
  "name": "llama2:13b"
Matt Williams's avatar
Matt Williams committed
384
385
386
}'
```

387
#### Response
388
389
390

If successful, the only response is a 200 OK.

391
## Pull a Model
Matt Williams's avatar
Matt Williams committed
392

Matt Williams's avatar
Matt Williams committed
393
```shell
394
395
396
POST /api/pull
```

Matt Williams's avatar
Matt Williams committed
397
Download a model from the ollama library. Cancelled pulls are resumed from where they left off, and multiple calls will share the same download progress.
Matt Williams's avatar
Matt Williams committed
398

399
### Parameters
Matt Williams's avatar
Matt Williams committed
400

401
- `name`: name of the model to pull
Matt Williams's avatar
Matt Williams committed
402
- `insecure`: (optional) allow insecure connections to the library. Only use this if you are pulling from your own library during development.
403
- `stream`: (optional) if `false` the response will be returned as a single response object, rather than a stream of objects
Matt Williams's avatar
Matt Williams committed
404

405
406
407
### Examples

#### Request
Matt Williams's avatar
Matt Williams committed
408

Matt Williams's avatar
Matt Williams committed
409
```shell
410
411
412
curl -X POST http://localhost:11434/api/pull -d '{
  "name": "llama2:7b"
}'
Matt Williams's avatar
Matt Williams committed
413
414
```

415
#### Response
416

417
418
419
420
421
422
423
424
425
426
427
428
If `stream` is not specified, or set to `true`, a stream of JSON objects is returned:

The first object is the manifest:

```json
{
  "status": "pulling manifest"
}
```

Then there is a series of downloading responses. Until any of the download is completed, the `completed` key may not be included. The number of files to be downloaded depends on the number of layers specified in the manifest.

429
```json
Matt Williams's avatar
Matt Williams committed
430
{
431
432
  "status": "downloading digestname",
  "digest": "digestname",
433
  "total": 2142590208,
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
  "completed": 241970
}
```

After all the files are downloaded, the final responses are:

```json
{
    "status": "verifying sha256 digest"
}
{
    "status": "writing manifest"
}
{
    "status": "removing any unused layers"
}
{
    "status": "success"
}
```

if `stream` is set to false, then the response is a single JSON object:

```json
{
  "status": "success"
Matt Williams's avatar
Matt Williams committed
460
461
}
```
462

Matt Williams's avatar
Matt Williams committed
463
464
465
466
467
468
469
470
471
472
473
## Push a Model

```shell
POST /api/push
```

Upload a model to a model library. Requires registering for ollama.ai and adding a public key first.

### Parameters

- `name`: name of the model to push in the form of `<namespace>/<model>:<tag>`
474
- `insecure`: (optional) allow insecure connections to the library. Only use this if you are pushing to your library during development.
475
- `stream`: (optional) if `false` the response will be returned as a single response object, rather than a stream of objects
Matt Williams's avatar
Matt Williams committed
476

477
478
479
### Examples

#### Request
Matt Williams's avatar
Matt Williams committed
480
481
482
483
484
485
486

```shell
curl -X POST http://localhost:11434/api/push -d '{
  "name": "mattw/pygmalion:latest"
}'
```

487
#### Response
488

489
If `stream` is not specified, or set to `true`, a stream of JSON objects is returned:
Matt Williams's avatar
Matt Williams committed
490
491

```json
492
{ "status": "retrieving manifest" }
493
```
Matt Williams's avatar
Matt Williams committed
494
495
496
497
498

and then:

```json
{
499
500
501
  "status": "starting upload",
  "digest": "sha256:bc07c81de745696fdf5afca05e065818a8149fb0c77266fb584d9b2cba3711ab",
  "total": 1928429856
Matt Williams's avatar
Matt Williams committed
502
503
504
505
506
507
508
}
```

Then there is a series of uploading responses:

```json
{
509
510
511
512
  "status": "starting upload",
  "digest": "sha256:bc07c81de745696fdf5afca05e065818a8149fb0c77266fb584d9b2cba3711ab",
  "total": 1928429856
}
Matt Williams's avatar
Matt Williams committed
513
514
515
516
517
518
519
520
521
```

Finally, when the upload is complete:

```json
{"status":"pushing manifest"}
{"status":"success"}
```

522
523
524
If `stream` is set to `false`, then the response is a single JSON object:

```json
525
{ "status": "success" }
526
527
```

Matt Williams's avatar
Matt Williams committed
528
529
530
## Generate Embeddings

```shell
531
532
533
534
535
536
537
538
539
540
POST /api/embeddings
```

Generate embeddings from a model

### Parameters

- `model`: name of model to generate embeddings from
- `prompt`: text to generate embeddings for

541
542
543
544
Advanced parameters:

- `options`: additional model parameters listed in the documentation for the [Modelfile](./modelfile.md#valid-parameters-and-values) such as `temperature`

545
546
547
### Examples

#### Request
548

Matt Williams's avatar
Matt Williams committed
549
```shell
550
551
552
553
554
555
curl -X POST http://localhost:11434/api/embeddings -d '{
  "model": "llama2:7b",
  "prompt": "Here is an article about llamas..."
}'
```

556
#### Response
557
558
559

```json
{
Alexander F. Rødseth's avatar
Alexander F. Rødseth committed
560
  "embedding": [
561
562
563
    0.5670403838157654, 0.009260174818336964, 0.23178744316101074, -0.2916173040866852, -0.8924556970596313,
    0.8785552978515625, -0.34576427936553955, 0.5742510557174683, -0.04222835972905159, -0.137906014919281
  ]
Costa Alexoglou's avatar
Costa Alexoglou committed
564
565
}
```