Skip to content

GitLab

Unverified Commit 934dd9e1 authored by Parth Sareen's avatar Parth Sareen Committed by GitHub
Browse files

docs: add reference to docs.ollama.com (#12800)

parent 1188f408
Hide whitespace changes
Inline Side-by-side
Showing with 133 additions and 0 deletions
docs/favicon-dark.svg 0 → 100644
View file @ 934dd9e1
<svg width="28" height="28" viewBox="0 0 28 28" fill="none" xmlns="http://www.w3.org/2000/svg">
<path fill-rule="evenodd" clip-rule="evenodd" d="M5.92343 3.10209C6.16484 3.19868 6.38278 3.35777 6.58061 3.56799C6.91031 3.91571 7.18861 4.41342 7.40096 5.00318C7.61443 5.59635 7.75302 6.25316 7.80555 6.91224C8.50922 6.5075 9.29019 6.26103 10.0956 6.18953L10.1526 6.18498C11.125 6.10544 12.0862 6.28384 12.9244 6.72361C13.0373 6.78384 13.1479 6.84861 13.2563 6.91679C13.3122 6.27021 13.4486 5.62817 13.6587 5.04863C13.871 4.45773 14.1493 3.96117 14.4779 3.61231C14.6616 3.41034 14.8862 3.25137 15.1362 3.14641C15.4235 3.03277 15.7286 3.01232 16.0259 3.09868C16.474 3.22822 16.8585 3.51685 17.1614 3.93617C17.4386 4.3191 17.6465 4.81 17.7884 5.39863C18.0455 6.45998 18.0902 7.85655 17.9169 9.54061L17.9762 9.58607L18.0052 9.60766C18.8513 10.2622 19.4403 11.1951 19.7522 12.2781C20.2383 13.9678 19.9936 15.8632 19.1553 16.9235L19.1352 16.9473L19.1374 16.9507C19.6035 17.8166 19.8863 18.7314 19.9466 19.678L19.9489 19.7121C20.0204 20.9223 19.7253 22.1404 19.0391 23.337L19.0312 23.3484L19.0424 23.3756C19.57 24.6904 19.7354 26.0142 19.532 27.3369L19.5253 27.3813C19.4938 27.5744 19.3881 27.7472 19.2316 27.8612C19.0751 27.9753 18.8805 28.0219 18.6903 27.9903C18.5962 27.9753 18.5059 27.9417 18.4246 27.8912C18.3433 27.8408 18.2725 27.7744 18.2165 27.696C18.1604 27.6178 18.1201 27.529 18.0979 27.4347C18.0756 27.3406 18.0718 27.2428 18.0868 27.1472C18.2735 25.9733 18.098 24.7961 17.5503 23.5984C17.4992 23.4871 17.4766 23.3645 17.4844 23.242C17.4922 23.1194 17.5302 23.0009 17.5951 22.8972L17.5995 22.8904C18.2746 21.8404 18.554 20.8109 18.4936 19.7996C18.4422 18.9143 18.1304 18.045 17.5995 17.2166C17.4963 17.0556 17.4595 16.8597 17.4971 16.6713C17.5348 16.4829 17.6438 16.3171 17.8007 16.2098L17.8108 16.203C18.0823 16.0223 18.3327 15.561 18.459 14.9303C18.5983 14.1846 18.562 13.4159 18.3528 12.6872C18.1237 11.8917 17.7046 11.2281 17.1178 10.7747C16.4528 10.2588 15.5721 10.0099 14.4578 10.0815C14.3121 10.0911 14.1669 10.0561 14.041 9.98097C13.9151 9.90581 13.8142 9.79399 13.7515 9.65993C13.4005 8.90426 12.8886 8.36336 12.2504 8.02814C11.6378 7.71723 10.9518 7.58657 10.27 7.65087C8.8785 7.76337 7.65132 8.56108 7.28584 9.56675C7.23414 9.70829 7.14113 9.8304 7.01928 9.91672C6.89744 10.003 6.75257 10.0494 6.60408 10.0497C5.41154 10.052 4.48836 10.3361 3.8133 10.8485C3.22989 11.2917 2.83201 11.911 2.62189 12.6531C2.43175 13.3515 2.40571 14.0857 2.54589 14.7962C2.67107 15.4303 2.91583 15.9553 3.19636 16.2382L3.2053 16.2462C3.44224 16.4814 3.49254 16.8485 3.32713 17.1382C2.92477 17.845 2.62412 18.8984 2.57495 19.9109C2.51906 21.0677 2.78283 22.0722 3.37854 22.7927L3.39642 22.8143C3.48632 22.9208 3.54413 23.0514 3.56297 23.1905C3.58179 23.3295 3.56083 23.4711 3.5026 23.5984C2.85883 25.0029 2.66101 26.1574 2.87448 27.0665C2.91284 27.2556 2.87714 27.4524 2.77496 27.6149C2.67279 27.7776 2.5122 27.8933 2.32739 27.9374C2.14257 27.9817 1.94812 27.9506 1.78542 27.8512C1.62269 27.7517 1.50458 27.5917 1.45618 27.4051C1.18459 26.2483 1.36901 24.9233 1.98483 23.4302L2.00048 23.3904L1.99153 23.3768C1.68885 22.9222 1.46298 22.4195 1.32318 21.8893L1.31759 21.8677C1.14793 21.2062 1.08119 20.5218 1.11977 19.8393C1.16895 18.8052 1.43048 17.7462 1.81495 16.8962L1.82836 16.8666L1.82612 16.8644C1.49865 16.3894 1.25612 15.7814 1.122 15.1087L1.11642 15.0814C0.931624 14.1431 0.967232 13.1737 1.22036 12.2519C1.51318 11.2122 2.08877 10.319 2.93707 9.67356C3.00413 9.62243 3.07454 9.57129 3.14495 9.52357C2.96724 7.827 3.01195 6.42134 3.27013 5.35317C3.41207 4.76455 3.62107 4.27365 3.89825 3.89071C4.20001 3.47254 4.58448 3.18391 5.03266 3.05323C5.32996 2.96686 5.63619 2.98732 5.92343 3.10209ZM10.5237 13.4315C11.5698 13.4315 12.5354 13.7871 13.2575 14.403C13.9616 15.0019 14.3807 15.8064 14.3807 16.6076C14.3807 17.6166 13.9269 18.403 13.1144 18.9052C12.4214 19.3314 11.4927 19.5382 10.4287 19.5382C9.30097 19.5382 8.33755 19.2439 7.64237 18.7041C6.95278 18.17 6.56608 17.42 6.56608 16.6076C6.56608 15.8042 7.0109 14.9973 7.74632 14.3962C8.49291 13.786 9.47867 13.4315 10.5237 13.4315ZM10.5237 14.4496C9.7483 14.4428 8.99346 14.7031 8.38226 15.1883C7.86702 15.6087 7.57532 16.1371 7.57532 16.6087C7.57532 17.095 7.81002 17.5507 8.25708 17.8973C8.76561 18.2916 9.51332 18.52 10.4287 18.52C11.3217 18.52 12.075 18.353 12.588 18.0359C13.1054 17.7178 13.3703 17.2564 13.3703 16.6076C13.3703 16.1269 13.0954 15.5962 12.607 15.1803C12.066 14.7201 11.3329 14.4496 10.5237 14.4496ZM11.2636 15.8246L11.268 15.8292C11.4021 16.0007 11.3742 16.2496 11.2054 16.386L10.8791 16.6473V17.1541C10.8785 17.267 10.8339 17.375 10.7551 17.4544C10.6764 17.5339 10.5698 17.5783 10.4589 17.578C10.3479 17.5783 10.2413 17.5339 10.1626 17.4544C10.0838 17.375 10.0392 17.267 10.0386 17.1541V16.6314L9.73573 16.3837C9.69577 16.3511 9.66255 16.3109 9.63798 16.2652C9.61342 16.2195 9.59799 16.1694 9.59258 16.1176C9.58718 16.0658 9.59191 16.0135 9.60651 15.9636C9.62111 15.9137 9.64527 15.8672 9.67762 15.8269C9.74358 15.7452 9.83858 15.6934 9.94196 15.6825C10.0453 15.6717 10.1487 15.7027 10.2297 15.7689L10.47 15.9644L10.7159 15.7667C10.7966 15.7018 10.899 15.6716 11.0013 15.6825C11.1036 15.6933 11.1977 15.7443 11.2636 15.8246ZM5.6306 13.644C6.16484 13.644 6.59961 14.0871 6.59961 14.6337C6.5999 14.8957 6.4979 15.1471 6.31599 15.3327C6.13408 15.5183 5.88717 15.6229 5.62949 15.6235C5.37218 15.6226 5.12572 15.518 4.9441 15.3327C4.76246 15.1474 4.66048 14.8965 4.66048 14.6349C4.65988 14.3729 4.76161 14.1213 4.94329 13.9355C5.125 13.7497 5.37292 13.6449 5.6306 13.644ZM15.3609 13.644C15.8973 13.644 16.331 14.0871 16.331 14.6337C16.3313 14.8957 16.2293 15.1471 16.0474 15.3327C15.8655 15.5183 15.6186 15.6229 15.3609 15.6235C15.1036 15.6226 14.8571 15.518 14.6755 15.3327C14.4938 15.1474 14.3919 14.8965 14.3919 14.6349C14.3913 14.3729 14.493 14.1213 14.6747 13.9355C14.8564 13.7497 15.1032 13.6449 15.3609 13.644ZM5.40372 4.47705L5.40037 4.47932C5.27088 4.53657 5.16032 4.63046 5.08184 4.74977L5.07625 4.75659C4.92201 4.97136 4.7879 5.28727 4.68731 5.70203C4.49731 6.48839 4.44589 7.55541 4.54872 8.86335C5.02931 8.7179 5.55349 8.62699 6.1179 8.59404L6.12908 8.5929L6.15031 8.55426C6.20172 8.46108 6.25649 8.37131 6.31572 8.28268C6.45319 7.40655 6.34031 6.35998 6.03296 5.50545C5.88319 5.09182 5.70102 4.76682 5.52666 4.5816C5.49067 4.5431 5.45055 4.5088 5.40707 4.47932L5.40372 4.47705ZM15.657 4.52251L15.6548 4.52364C15.6113 4.55312 15.5712 4.58741 15.5352 4.62591C15.3609 4.81114 15.1776 5.13727 15.0289 5.5509C14.7048 6.45316 14.5964 7.56905 14.7719 8.47358L14.8367 8.58381L14.8456 8.59972H14.8792C15.4339 8.59988 15.9856 8.68101 16.5176 8.84062C16.6138 7.56337 16.5601 6.51907 16.3746 5.74749C16.274 5.33272 16.1399 5.01682 15.9845 4.80205L15.98 4.79523C15.9017 4.67548 15.7911 4.58121 15.6615 4.52364L15.657 4.52251Z" fill="black"/>
</svg>
docs/favicon.svg 0 → 100644
View file @ 934dd9e1
<svg width="10" height="12" viewBox="0 0 10 12" fill="none" xmlns="http://www.w3.org/2000/svg">
<path fill-rule="evenodd" clip-rule="evenodd" d="M2.95258 0.545107C3.06058 0.587607 3.15808 0.657607 3.24658 0.750107C3.39408 0.903107 3.51858 1.1221 3.61358 1.3816C3.70908 1.6426 3.77108 1.9316 3.79458 2.2216C4.10938 2.04351 4.45876 1.93506 4.81908 1.9036L4.84458 1.9016C5.27958 1.8666 5.70958 1.9451 6.08458 2.1386C6.13508 2.1651 6.18458 2.1936 6.23308 2.2236C6.25808 1.9391 6.31908 1.6566 6.41308 1.4016C6.50808 1.1416 6.63258 0.923107 6.77958 0.769607C6.86173 0.680739 6.96222 0.610791 7.07408 0.564607C7.20258 0.514607 7.33908 0.505607 7.47208 0.543607C7.67258 0.600607 7.84458 0.727607 7.98008 0.912107C8.10408 1.0806 8.19708 1.2966 8.26058 1.5556C8.37558 2.0226 8.39558 2.6371 8.31808 3.3781L8.34458 3.3981L8.35758 3.4076C8.73608 3.6956 8.99958 4.1061 9.1391 4.5826C9.3566 5.3261 9.2471 6.1601 8.87208 6.6266L8.86308 6.6371L8.86408 6.6386C9.0726 7.0196 9.1991 7.4221 9.2261 7.8386L9.2271 7.8536C9.2591 8.3861 9.1271 8.9221 8.82008 9.4486L8.81658 9.4536L8.82158 9.4656C9.0576 10.0441 9.1316 10.6266 9.0406 11.2086L9.0376 11.2281C9.0235 11.3131 8.97624 11.3891 8.90623 11.4393C8.83621 11.4895 8.74913 11.51 8.66408 11.4961C8.62196 11.4895 8.58156 11.4747 8.54518 11.4525C8.50881 11.4303 8.47717 11.4011 8.45209 11.3666C8.42701 11.3322 8.40897 11.2931 8.39902 11.2516C8.38906 11.2102 8.38738 11.1672 8.39408 11.1251C8.47758 10.6086 8.39908 10.0906 8.15408 9.56361C8.13122 9.51465 8.12107 9.46071 8.12456 9.40679C8.12806 9.35287 8.14509 9.3007 8.17408 9.2551L8.17608 9.2521C8.47808 8.7901 8.60308 8.3371 8.57608 7.8921C8.55308 7.5026 8.41358 7.1201 8.17608 6.7556C8.12989 6.68474 8.11343 6.59855 8.13027 6.51566C8.14711 6.43276 8.1959 6.35982 8.26608 6.3126L8.27058 6.3096C8.39208 6.2301 8.50408 6.0271 8.56058 5.7496C8.62292 5.42147 8.60664 5.08324 8.51308 4.7626C8.41058 4.4126 8.22308 4.1206 7.96058 3.9211C7.66308 3.6941 7.26908 3.5846 6.77058 3.6161C6.70539 3.62034 6.64044 3.60493 6.5841 3.57186C6.52777 3.53879 6.48265 3.48959 6.45458 3.4306C6.29758 3.0981 6.06858 2.8601 5.78308 2.7126C5.50898 2.5758 5.20211 2.51831 4.89708 2.5466C4.27458 2.5961 3.72558 2.9471 3.56208 3.3896C3.53895 3.45188 3.49734 3.50561 3.44283 3.54359C3.38832 3.58156 3.32351 3.60198 3.25708 3.6021C2.72358 3.6031 2.31058 3.7281 2.00858 3.9536C1.74758 4.1486 1.56958 4.4211 1.47558 4.7476C1.39052 5.05493 1.37887 5.37795 1.44158 5.6906C1.49758 5.9696 1.60708 6.2006 1.73258 6.3251L1.73658 6.3286C1.84258 6.4321 1.86508 6.5936 1.79108 6.7211C1.61108 7.0321 1.47658 7.4956 1.45458 7.9411C1.42958 8.4501 1.54758 8.8921 1.81408 9.2091L1.82208 9.2186C1.8623 9.26547 1.88816 9.32293 1.89659 9.38412C1.90501 9.4453 1.89563 9.50761 1.86958 9.56361C1.58158 10.1816 1.49308 10.6896 1.58858 11.0896C1.60574 11.1728 1.58977 11.2594 1.54406 11.3309C1.49835 11.4025 1.42651 11.4534 1.34383 11.4728C1.26115 11.4923 1.17416 11.4786 1.10137 11.4349C1.02857 11.3911 0.975731 11.3207 0.954081 11.2386C0.832581 10.7296 0.915081 10.1466 1.19058 9.4896L1.19758 9.4721L1.19358 9.4661C1.05817 9.2661 0.957121 9.04489 0.894581 8.8116L0.892081 8.8021C0.816181 8.51105 0.786321 8.2099 0.803581 7.9096C0.825581 7.4546 0.942581 6.9886 1.11458 6.6146L1.12058 6.6016L1.11958 6.6006C0.973081 6.3916 0.864581 6.1241 0.804581 5.8281L0.802081 5.8161C0.719411 5.40323 0.735341 4.97667 0.848581 4.5711C0.979581 4.1136 1.23708 3.7206 1.61658 3.4366C1.64658 3.4141 1.67808 3.3916 1.70958 3.3706C1.63008 2.6241 1.65008 2.0056 1.76558 1.5356C1.82908 1.2766 1.92258 1.0606 2.04658 0.892107C2.18158 0.708107 2.35358 0.581107 2.55408 0.523607C2.68708 0.485607 2.82408 0.494607 2.95258 0.545107ZM5.01058 5.0901C5.47858 5.0901 5.91058 5.2466 6.23358 5.5176C6.54858 5.7811 6.73608 6.1351 6.73608 6.4876C6.73608 6.9316 6.53308 7.2776 6.16958 7.4986C5.85958 7.6861 5.44408 7.7771 4.96808 7.7771C4.46358 7.7771 4.03258 7.6476 3.72158 7.4101C3.41308 7.1751 3.24008 6.8451 3.24008 6.4876C3.24008 6.1341 3.43908 5.7791 3.76808 5.5146C4.10208 5.2461 4.54308 5.0901 5.01058 5.0901ZM5.01058 5.5381C4.6637 5.53508 4.32601 5.64963 4.05258 5.8631C3.82208 6.0481 3.69158 6.2806 3.69158 6.4881C3.69158 6.7021 3.79658 6.9026 3.99658 7.0551C4.22408 7.2286 4.55858 7.3291 4.96808 7.3291C5.36758 7.3291 5.70458 7.2556 5.93408 7.1161C6.16558 6.9761 6.28408 6.7731 6.28408 6.4876C6.28408 6.2761 6.16108 6.0426 5.94258 5.8596C5.70058 5.6571 5.37258 5.5381 5.01058 5.5381ZM5.34158 6.1431L5.34358 6.1451C5.40358 6.2206 5.39108 6.3301 5.31558 6.3901L5.16958 6.5051V6.7281C5.16931 6.77775 5.14937 6.82526 5.11413 6.86022C5.07889 6.89518 5.03122 6.91474 4.98158 6.9146C4.93194 6.91474 4.88427 6.89518 4.84903 6.86022C4.81378 6.82526 4.79384 6.77775 4.79358 6.7281V6.4981L4.65808 6.3891C4.6402 6.37478 4.62534 6.35706 4.61435 6.33697C4.60336 6.31687 4.59646 6.2948 4.59404 6.27202C4.59162 6.24924 4.59374 6.22621 4.60027 6.20426C4.6068 6.1823 4.61761 6.16186 4.63208 6.1441C4.66159 6.10818 4.70409 6.08535 4.75034 6.08057C4.79658 6.0758 4.84285 6.08947 4.87908 6.1186L4.98658 6.2046L5.09658 6.1176C5.13268 6.08908 5.1785 6.07579 5.22426 6.08055C5.27003 6.08531 5.31212 6.10776 5.34158 6.1431ZM2.82158 5.1836C3.06058 5.1836 3.25508 5.3786 3.25508 5.6191C3.25521 5.73439 3.20958 5.84501 3.1282 5.92667C3.04682 6.00832 2.93636 6.05434 2.82108 6.0546C2.70597 6.05421 2.59571 6.0082 2.51446 5.92667C2.4332 5.84513 2.38758 5.73471 2.38758 5.6196C2.38731 5.50432 2.43282 5.39365 2.5141 5.3119C2.59539 5.23014 2.7063 5.184 2.82158 5.1836ZM7.17458 5.1836C7.41458 5.1836 7.60858 5.3786 7.60858 5.6191C7.60871 5.73439 7.56308 5.84501 7.4817 5.92667C7.40032 6.00832 7.28986 6.05434 7.17458 6.0546C7.05947 6.05421 6.94921 6.0082 6.86796 5.92667C6.7867 5.84513 6.74108 5.73471 6.74108 5.6196C6.74081 5.50432 6.78632 5.39365 6.8676 5.3119C6.94889 5.23014 7.0593 5.184 7.17458 5.1836ZM2.72008 1.1501L2.71858 1.1511C2.66065 1.17629 2.61119 1.2176 2.57608 1.2701L2.57358 1.2731C2.50458 1.3676 2.44458 1.5066 2.39958 1.6891C2.31458 2.0351 2.29158 2.5046 2.33758 3.0801C2.55258 3.0161 2.78708 2.9761 3.03958 2.9616L3.04458 2.9611L3.05408 2.9441C3.07708 2.9031 3.10158 2.8636 3.12808 2.8246C3.18958 2.4391 3.13908 1.9786 3.00158 1.6026C2.93458 1.4206 2.85308 1.2776 2.77508 1.1961C2.75898 1.17916 2.74103 1.16407 2.72158 1.1511L2.72008 1.1501ZM7.30708 1.1701L7.30608 1.1706C7.28663 1.18357 7.26868 1.19866 7.25258 1.2156C7.17458 1.2971 7.09258 1.4406 7.02608 1.6226C6.88108 2.0196 6.83258 2.5106 6.91108 2.9086L6.94008 2.9571L6.94408 2.9641H6.95908C7.20723 2.96417 7.45408 2.99987 7.69208 3.0701C7.73508 2.5081 7.71108 2.0486 7.62808 1.7091C7.58308 1.5266 7.52308 1.3876 7.45358 1.2931L7.45158 1.2901C7.41654 1.23741 7.36707 1.19593 7.30908 1.1706L7.30708 1.1701Z" fill="white"/>
</svg>
docs/gpu.mdx 0 → 100644
View file @ 934dd9e1
---
title: Hardware support
---
## Nvidia
Ollama supports Nvidia GPUs with compute capability 5.0+.
Check your compute compatibility to see if your card is supported:
[https://developer.nvidia.com/cuda-gpus](https://developer.nvidia.com/cuda-gpus)
| Compute Capability | Family | Cards |
| ------------------ | ------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| 9.0 | NVIDIA | `H200` `H100` |
| 8.9 | GeForce RTX 40xx | `RTX 4090` `RTX 4080 SUPER` `RTX 4080` `RTX 4070 Ti SUPER` `RTX 4070 Ti` `RTX 4070 SUPER` `RTX 4070` `RTX 4060 Ti` `RTX 4060` |
| | NVIDIA Professional | `L4` `L40` `RTX 6000` |
| 8.6 | GeForce RTX 30xx | `RTX 3090 Ti` `RTX 3090` `RTX 3080 Ti` `RTX 3080` `RTX 3070 Ti` `RTX 3070` `RTX 3060 Ti` `RTX 3060` `RTX 3050 Ti` `RTX 3050` |
| | NVIDIA Professional | `A40` `RTX A6000` `RTX A5000` `RTX A4000` `RTX A3000` `RTX A2000` `A10` `A16` `A2` |
| 8.0 | NVIDIA | `A100` `A30` |
| 7.5 | GeForce GTX/RTX | `GTX 1650 Ti` `TITAN RTX` `RTX 2080 Ti` `RTX 2080` `RTX 2070` `RTX 2060` |
| | NVIDIA Professional | `T4` `RTX 5000` `RTX 4000` `RTX 3000` `T2000` `T1200` `T1000` `T600` `T500` |
| | Quadro | `RTX 8000` `RTX 6000` `RTX 5000` `RTX 4000` |
| 7.0 | NVIDIA | `TITAN V` `V100` `Quadro GV100` |
| 6.1 | NVIDIA TITAN | `TITAN Xp` `TITAN X` |
| | GeForce GTX | `GTX 1080 Ti` `GTX 1080` `GTX 1070 Ti` `GTX 1070` `GTX 1060` `GTX 1050 Ti` `GTX 1050` |
| | Quadro | `P6000` `P5200` `P4200` `P3200` `P5000` `P4000` `P3000` `P2200` `P2000` `P1000` `P620` `P600` `P500` `P520` |
| | Tesla | `P40` `P4` |
| 6.0 | NVIDIA | `Tesla P100` `Quadro GP100` |
| 5.2 | GeForce GTX | `GTX TITAN X` `GTX 980 Ti` `GTX 980` `GTX 970` `GTX 960` `GTX 950` |
| | Quadro | `M6000 24GB` `M6000` `M5000` `M5500M` `M4000` `M2200` `M2000` `M620` |
| | Tesla | `M60` `M40` |
| 5.0 | GeForce GTX | `GTX 750 Ti` `GTX 750` `NVS 810` |
| | Quadro | `K2200` `K1200` `K620` `M1200` `M520` `M5000M` `M4000M` `M3000M` `M2000M` `M1000M` `K620M` `M600M` `M500M` |
For building locally to support older GPUs, see [developer.md](./development.md#linux-cuda-nvidia)
### GPU Selection
If you have multiple NVIDIA GPUs in your system and want to limit Ollama to use
a subset, you can set `CUDA_VISIBLE_DEVICES` to a comma separated list of GPUs.
Numeric IDs may be used, however ordering may vary, so UUIDs are more reliable.
You can discover the UUID of your GPUs by running `nvidia-smi -L` If you want to
ignore the GPUs and force CPU usage, use an invalid GPU ID (e.g., "-1")
### Linux Suspend Resume
On linux, after a suspend/resume cycle, sometimes Ollama will fail to discover
your NVIDIA GPU, and fallback to running on the CPU. You can workaround this
driver bug by reloading the NVIDIA UVM driver with `sudo rmmod nvidia_uvm &&
sudo modprobe nvidia_uvm`
## AMD Radeon
Ollama supports the following AMD GPUs:
### Linux Support
| Family | Cards and accelerators |
| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| AMD Radeon RX | `7900 XTX` `7900 XT` `7900 GRE` `7800 XT` `7700 XT` `7600 XT` `7600` `6950 XT` `6900 XTX` `6900XT` `6800 XT` `6800` `Vega 64` `Vega 56` |
| AMD Radeon PRO | `W7900` `W7800` `W7700` `W7600` `W7500` `W6900X` `W6800X Duo` `W6800X` `W6800` `V620` `V420` `V340` `V320` `Vega II Duo` `Vega II` `VII` `SSG` |
| AMD Instinct | `MI300X` `MI300A` `MI300` `MI250X` `MI250` `MI210` `MI200` `MI100` `MI60` `MI50` |
### Windows Support
With ROCm v6.1, the following GPUs are supported on Windows.
| Family | Cards and accelerators |
| -------------- | ------------------------------------------------------------------------------------------------------------------- |
| AMD Radeon RX | `7900 XTX` `7900 XT` `7900 GRE` `7800 XT` `7700 XT` `7600 XT` `7600` `6950 XT` `6900 XTX` `6900XT` `6800 XT` `6800` |
| AMD Radeon PRO | `W7900` `W7800` `W7700` `W7600` `W7500` `W6900X` `W6800X Duo` `W6800X` `W6800` `V620` |
### Overrides on Linux
Ollama leverages the AMD ROCm library, which does not support all AMD GPUs. In
some cases you can force the system to try to use a similar LLVM target that is
close. For example The Radeon RX 5400 is `gfx1034` (also known as 10.3.4)
however, ROCm does not currently support this target. The closest support is
`gfx1030`. You can use the environment variable `HSA_OVERRIDE_GFX_VERSION` with
`x.y.z` syntax. So for example, to force the system to run on the RX 5400, you
would set `HSA_OVERRIDE_GFX_VERSION="10.3.0"` as an environment variable for the
server. If you have an unsupported AMD GPU you can experiment using the list of
supported types below.
If you have multiple GPUs with different GFX versions, append the numeric device
number to the environment variable to set them individually. For example,
`HSA_OVERRIDE_GFX_VERSION_0=10.3.0` and `HSA_OVERRIDE_GFX_VERSION_1=11.0.0`
At this time, the known supported GPU types on linux are the following LLVM Targets.
This table shows some example GPUs that map to these LLVM targets:
| **LLVM Target** | **An Example GPU** |
|-----------------|---------------------|
| gfx900 | Radeon RX Vega 56 |
| gfx906 | Radeon Instinct MI50 |
| gfx908 | Radeon Instinct MI100 |
| gfx90a | Radeon Instinct MI210 |
| gfx940 | Radeon Instinct MI300 |
| gfx941 | |
| gfx942 | |
| gfx1030 | Radeon PRO V620 |
| gfx1100 | Radeon PRO W7900 |
| gfx1101 | Radeon PRO W7700 |
| gfx1102 | Radeon RX 7600 |
AMD is working on enhancing ROCm v6 to broaden support for families of GPUs in a
future release which should increase support for more GPUs.
Reach out on [Discord](https://discord.gg/ollama) or file an
[issue](https://github.com/ollama/ollama/issues) for additional help.
### GPU Selection
If you have multiple AMD GPUs in your system and want to limit Ollama to use a
subset, you can set `ROCR_VISIBLE_DEVICES` to a comma separated list of GPUs.
You can see the list of devices with `rocminfo`. If you want to ignore the GPUs
and force CPU usage, use an invalid GPU ID (e.g., "-1"). When available, use the
`Uuid` to uniquely identify the device instead of numeric value.
### Container Permission
In some Linux distributions, SELinux can prevent containers from
accessing the AMD GPU devices. On the host system you can run
`sudo setsebool container_use_devices=1` to allow containers to use devices.
### Metal (Apple GPUs)
Ollama supports GPU acceleration on Apple devices via the Metal API.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment