Skip to content

GitLab

"vscode:/vscode.git/clone" did not exist on "f7e99c0873c033832529f18e6596a4527db606a1"
Unverified Commit 430811a7 authored by Karmel Allison's avatar Karmel Allison Committed by GitHub
Browse files

Adding scripts for TensorRT (#3808) 

* Adding scripts for TensorRT

* Fixing README link

* Responding to CR
parent e52a22c9
Show whitespace changes
Inline Side-by-side
Showing with 767 additions and 0 deletions
research/tensorrt/README.md 0 → 100644
View file @ 430811a7
# Running the TensorFlow Official ResNet with TensorRT
[TensorRT](https://developer.nvidia.com/tensorrt) is NVIDIA's inference
optimizer for deep learning. Briefly, TensorRT rewrites parts of the
execution graph to allow for faster prediction times.
Here we provide a sample script that can:
1. Convert a TensorFlow SavedModel to a Frozen Graph.
2. Load a Frozen Graph for inference.
3. Time inference loops using the native TensorFlow graph.
4. Time inference loops using FP32, FP16, or INT8<sup>1</sup> precision modes from TensorRT.
We provide some results below, as well as instructions for running this script.
<sup>1</sup> INT8 mode is a work in progress; please see [INT8 Mode is the Bleeding Edge](#int8-mode-is-the-bleeding-edge) below.
## How to Run This Script
### Step 1: Install Prerequisites
1. [Install TensorFlow.](https://www.tensorflow.org/install/)
2. [Install TensorRT.](http://docs.nvidia.com/deeplearning/sdk/tensorrt-install-guide/index.html)
3. We use the image processing functions from the [Official version of ResNet](/official/resnet/imagenet_preprocessing.py). Please checkout the Models repository if you haven't
already, and add the Official Models to your Python path:
```
git clone https://github.com/tensorflow/models.git
export PYTHONPATH="$PYTHONPATH:/path/to/models"
```
### Step 2: Get a model to test
The provided script runs with the [Official version of ResNet trained with
ImageNet data](/official/resnet), but can be used for other models as well,
as long as you have a SavedModel or a Frozen Graph.
You can download the ResNetv2-ImageNet [SavedModel](http://download.tensorflow.org/models/official/resnetv2_imagenet_savedmodel.tar.gz)
or [Frozen Graph](http://download.tensorflow.org/models/official/resnetv2_imagenet_frozen_graph.pb),
or, if you want to train the model yourself,
pass `--export_dir` to the Official ResNet [imagenet_main.py](/official/resnet/imagenet_main.py).
When running this script, you can pass in a SavedModel directory containing the
Protobuf MetaGraphDef and variables directory to `savedmodel_dir`, or pass in
a Protobuf frozen graph file directly to `frozen_graph`. If you downloaded the
SavedModel linked above, note that you should untar it before passing in to the
script.
### Step 3: Get an image to test
The script can accept a JPEG image file to use for predictions. If none is
provided, random data will be generated. We provide a sample `image.jpg` here
which can be passed in with the `--image_file` flag.
### Step 4: Run the model
You have TensorFlow, TensorRT, a graph def, and a picture.
Now it's time to time.
For the full set of possible parameters, you can run
`python tensorrt.py --help`. Assuming you used the files provided above,
you would run:
```
python tensorrt.py --frozen_graph=resnetv2_imagenet_frozen_graph.pb \
--image_file=image.jpg --native --fp32 --fp16 --output_dir=/my/output
```
This will print the predictions for each of the precision modes that were run
(native, which is the native precision of the model passed in, as well
as the TensorRT version of the graph at precisions of fp32 and fp16):
```
INFO:tensorflow:Starting timing.
INFO:tensorflow:Timing loop done!
Predictions:
Precision: native [u'seashore, coast, seacoast, sea-coast', u'promontory, headland, head, foreland', u'breakwater, groin, groyne, mole, bulwark, seawall, jetty', u'lakeside, lakeshore', u'grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus']
Precision: FP32 [u'seashore, coast, seacoast, sea-coast', u'promontory, headland, head, foreland', u'breakwater, groin, groyne, mole, bulwark, seawall, jetty', u'lakeside, lakeshore', u'sandbar, sand bar']
Precision: FP16 [u'seashore, coast, seacoast, sea-coast', u'promontory, headland, head, foreland', u'lakeside, lakeshore', u'sandbar, sand bar', u'breakwater, groin, groyne, mole, bulwark, seawall, jetty']
```
The script will generate or append to a file in the output_dir, `log.txt`,
which includes the timing information for each of the models:
```
==========================
network: native_resnetv2_imagenet_frozen_graph.pb, batchsize 128, steps 100
fps median: 1041.4, mean: 1056.6, uncertainty: 2.8, jitter: 6.1
latency median: 0.12292, mean: 0.12123, 99th_p: 0.13151, 99th_uncertainty: 0.00024
==========================
network: tftrt_fp32_resnetv2_imagenet_frozen_graph.pb, batchsize 128, steps 100
fps median: 1253.0, mean: 1250.8, uncertainty: 3.4, jitter: 17.3
latency median: 0.10215, mean: 0.10241, 99th_p: 0.11482, 99th_uncertainty: 0.01109
==========================
network: tftrt_fp16_resnetv2_imagenet_frozen_graph.pb, batchsize 128, steps 100
fps median: 2280.2, mean: 2312.8, uncertainty: 10.3, jitter: 100.1
latency median: 0.05614, mean: 0.05546, 99th_p: 0.06103, 99th_uncertainty: 0.00781
```
The script will also output the GraphDefs used for each of the modes run,
for future use and inspection:
```
ls /my/output
log.txt
tftrt_fp16_imagenet_frozen_graph.pb
tftrt_fp32_imagenet_frozen_graph.pb
```
## Troubleshooting and Notes
### INT8 Mode is the Bleeding Edge
Note that currently, INT8 mode results in a segfault using the models provided.
We are working on it.
```
E tensorflow/contrib/tensorrt/log/trt_logger.cc:38] DefaultLogger Parameter check failed at: Network.cpp::addScale::118, condition: shift.count == 0 || shift.count == weightCount
Segmentation fault (core dumped)
```
### GPU/Precision Compatibility
Not all GPUs support the ops required for all precisions. For example, P100s
cannot currently run INT8 precision.
### Label Offsets
Some ResNet models represent 1000 categories, and some represent all 1001, with
the 0th category being "background". The models provided are of the latter type.
If you are using a different model and find that your predictions seem slightly
off, try passing in the `--ids_are_one_indexed` arg, which adjusts the label
alignment for models with only 1000 categories.
## Model Links
[ResNet-v2-ImageNet Frozen Graph](http://download.tensorflow.org/models/official/resnetv2_imagenet_frozen_graph.pb)
[ResNet-v2-ImageNet SavedModel](http://download.tensorflow.org/models/official/resnetv2_imagenet_savedmodel.tar.gz)
[ResNet-v1-ImageNet Frozen Graph](http://download.tensorflow.org/models/official/resnetv1_imagenet_frozen_graph.pb)
[ResNet-v1-ImageNet SavedModel](http://download.tensorflow.org/models/official/resnetv1_imagenet_savedmodel.tar.gz)
research/tensorrt/labellist.json 0 → 100755
View file @ 430811a7
{"0": "background", "1": "tench, Tinca tinca", "2": "goldfish, Carassius auratus", "3": "great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias", "4": "tiger shark, Galeocerdo cuvieri", "5": "hammerhead, hammerhead shark", "6": "electric ray, crampfish, numbfish, torpedo", "7": "stingray", "8": "cock", "9": "hen", "10": "ostrich, Struthio camelus", "11": "brambling, Fringilla montifringilla", "12": "goldfinch, Carduelis carduelis", "13": "house finch, linnet, Carpodacus mexicanus", "14": "junco, snowbird", "15": "indigo bunting, indigo finch, indigo bird, Passerina cyanea", "16": "robin, American robin, Turdus migratorius", "17": "bulbul", "18": "jay", "19": "magpie", "20": "chickadee", "21": "water ouzel, dipper", "22": "kite", "23": "bald eagle, American eagle, Haliaeetus leucocephalus", "24": "vulture", "25": "great grey owl, great gray owl, Strix nebulosa", "26": "European fire salamander, Salamandra salamandra", "27": "common newt, Triturus vulgaris", "28": "eft", "29": "spotted salamander, Ambystoma maculatum", "30": "axolotl, mud puppy, Ambystoma mexicanum", "31": "bullfrog, Rana catesbeiana", "32": "tree frog, tree-frog", "33": "tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui", "34": "loggerhead, loggerhead turtle, Caretta caretta", "35": "leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea", "36": "mud turtle", "37": "terrapin", "38": "box turtle, box tortoise", "39": "banded gecko", "40": "common iguana, iguana, Iguana iguana", "41": "American chameleon, anole, Anolis carolinensis", "42": "whiptail, whiptail lizard", "43": "agama", "44": "frilled lizard, Chlamydosaurus kingi", "45": "alligator lizard", "46": "Gila monster, Heloderma suspectum", "47": "green lizard, Lacerta viridis", "48": "African chameleon, Chamaeleo chamaeleon", "49": "Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis", "50": "African crocodile, Nile crocodile, Crocodylus niloticus", "51": "American alligator, Alligator mississipiensis", "52": "triceratops", "53": "thunder snake, worm snake, Carphophis amoenus", "54": "ringneck snake, ring-necked snake, ring snake", "55": "hognose snake, puff adder, sand viper", "56": "green snake, grass snake", "57": "king snake, kingsnake", "58": "garter snake, grass snake", "59": "water snake", "60": "vine snake", "61": "night snake, Hypsiglena torquata", "62": "boa constrictor, Constrictor constrictor", "63": "rock python, rock snake, Python sebae", "64": "Indian cobra, Naja naja", "65": "green mamba", "66": "sea snake", "67": "horned viper, cerastes, sand viper, horned asp, Cerastes cornutus", "68": "diamondback, diamondback rattlesnake, Crotalus adamanteus", "69": "sidewinder, horned rattlesnake, Crotalus cerastes", "70": "trilobite", "71": "harvestman, daddy longlegs, Phalangium opilio", "72": "scorpion", "73": "black and gold garden spider, Argiope aurantia", "74": "barn spider, Araneus cavaticus", "75": "garden spider, Aranea diademata", "76": "black widow, Latrodectus mactans", "77": "tarantula", "78": "wolf spider, hunting spider", "79": "tick", "80": "centipede", "81": "black grouse", "82": "ptarmigan", "83": "ruffed grouse, partridge, Bonasa umbellus", "84": "prairie chicken, prairie grouse, prairie fowl", "85": "peacock", "86": "quail", "87": "partridge", "88": "African grey, African gray, Psittacus erithacus", "89": "macaw", "90": "sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita", "91": "lorikeet", "92": "coucal", "93": "bee eater", "94": "hornbill", "95": "hummingbird", "96": "jacamar", "97": "toucan", "98": "drake", "99": "red-breasted merganser, Mergus serrator", "100": "goose", "101": "black swan, Cygnus atratus", "102": "tusker", "103": "echidna, spiny anteater, anteater", "104": "platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus", "105": "wallaby, brush kangaroo", "106": "koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus", "107": "wombat", "108": "jellyfish", "109": "sea anemone, anemone", "110": "brain coral", "111": "flatworm, platyhelminth", "112": "nematode, nematode worm, roundworm", "113": "conch", "114": "snail", "115": "slug", "116": "sea slug, nudibranch", "117": "chiton, coat-of-mail shell, sea cradle, polyplacophore", "118": "chambered nautilus, pearly nautilus, nautilus", "119": "Dungeness crab, Cancer magister", "120": "rock crab, Cancer irroratus", "121": "fiddler crab", "122": "king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica", "123": "American lobster, Northern lobster, Maine lobster, Homarus americanus", "124": "spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish", "125": "crayfish, crawfish, crawdad, crawdaddy", "126": "hermit crab", "127": "isopod", "128": "white stork, Ciconia ciconia", "129": "black stork, Ciconia nigra", "130": "spoonbill", "131": "flamingo", "132": "little blue heron, Egretta caerulea", "133": "American egret, great white heron, Egretta albus", "134": "bittern", "135": "crane", "136": "limpkin, Aramus pictus", "137": "European gallinule, Porphyrio porphyrio", "138": "American coot, marsh hen, mud hen, water hen, Fulica americana", "139": "bustard", "140": "ruddy turnstone, Arenaria interpres", "141": "red-backed sandpiper, dunlin, Erolia alpina", "142": "redshank, Tringa totanus", "143": "dowitcher", "144": "oystercatcher, oyster catcher", "145": "pelican", "146": "king penguin, Aptenodytes patagonica", "147": "albatross, mollymawk", "148": "grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus", "149": "killer whale, killer, orca, grampus, sea wolf, Orcinus orca", "150": "dugong, Dugong dugon", "151": "sea lion", "152": "Chihuahua", "153": "Japanese spaniel", "154": "Maltese dog, Maltese terrier, Maltese", "155": "Pekinese, Pekingese, Peke", "156": "Shih-Tzu", "157": "Blenheim spaniel", "158": "papillon", "159": "toy terrier", "160": "Rhodesian ridgeback", "161": "Afghan hound, Afghan", "162": "basset, basset hound", "163": "beagle", "164": "bloodhound, sleuthhound", "165": "bluetick", "166": "black-and-tan coonhound", "167": "Walker hound, Walker foxhound", "168": "English foxhound", "169": "redbone", "170": "borzoi, Russian wolfhound", "171": "Irish wolfhound", "172": "Italian greyhound", "173": "whippet", "174": "Ibizan hound, Ibizan Podenco", "175": "Norwegian elkhound, elkhound", "176": "otterhound, otter hound", "177": "Saluki, gazelle hound", "178": "Scottish deerhound, deerhound", "179": "Weimaraner", "180": "Staffordshire bullterrier, Staffordshire bull terrier", "181": "American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier", "182": "Bedlington terrier", "183": "Border terrier", "184": "Kerry blue terrier", "185": "Irish terrier", "186": "Norfolk terrier", "187": "Norwich terrier", "188": "Yorkshire terrier", "189": "wire-haired fox terrier", "190": "Lakeland terrier", "191": "Sealyham terrier, Sealyham", "192": "Airedale, Airedale terrier", "193": "cairn, cairn terrier", "194": "Australian terrier", "195": "Dandie Dinmont, Dandie Dinmont terrier", "196": "Boston bull, Boston terrier", "197": "miniature schnauzer", "198": "giant schnauzer", "199": "standard schnauzer", "200": "Scotch terrier, Scottish terrier, Scottie", "201": "Tibetan terrier, chrysanthemum dog", "202": "silky terrier, Sydney silky", "203": "soft-coated wheaten terrier", "204": "West Highland white terrier", "205": "Lhasa, Lhasa apso", "206": "flat-coated retriever", "207": "curly-coated retriever", "208": "golden retriever", "209": "Labrador retriever", "210": "Chesapeake Bay retriever", "211": "German short-haired pointer", "212": "vizsla, Hungarian pointer", "213": "English setter", "214": "Irish setter, red setter", "215": "Gordon setter", "216": "Brittany spaniel", "217": "clumber, clumber spaniel", "218": "English springer, English springer spaniel", "219": "Welsh springer spaniel", "220": "cocker spaniel, English cocker spaniel, cocker", "221": "Sussex spaniel", "222": "Irish water spaniel", "223": "kuvasz", "224": "schipperke", "225": "groenendael", "226": "malinois", "227": "briard", "228": "kelpie", "229": "komondor", "230": "Old English sheepdog, bobtail", "231": "Shetland sheepdog, Shetland sheep dog, Shetland", "232": "collie", "233": "Border collie", "234": "Bouvier des Flandres, Bouviers des Flandres", "235": "Rottweiler", "236": "German shepherd, German shepherd dog, German police dog, alsatian", "237": "Doberman, Doberman pinscher", "238": "miniature pinscher", "239": "Greater Swiss Mountain dog", "240": "Bernese mountain dog", "241": "Appenzeller", "242": "EntleBucher", "243": "boxer", "244": "bull mastiff", "245": "Tibetan mastiff", "246": "French bulldog", "247": "Great Dane", "248": "Saint Bernard, St Bernard", "249": "Eskimo dog, husky", "250": "malamute, malemute, Alaskan malamute", "251": "Siberian husky", "252": "dalmatian, coach dog, carriage dog", "253": "affenpinscher, monkey pinscher, monkey dog", "254": "basenji", "255": "pug, pug-dog", "256": "Leonberg", "257": "Newfoundland, Newfoundland dog", "258": "Great Pyrenees", "259": "Samoyed, Samoyede", "260": "Pomeranian", "261": "chow, chow chow", "262": "keeshond", "263": "Brabancon griffon", "264": "Pembroke, Pembroke Welsh corgi", "265": "Cardigan, Cardigan Welsh corgi", "266": "toy poodle", "267": "miniature poodle", "268": "standard poodle", "269": "Mexican hairless", "270": "timber wolf, grey wolf, gray wolf, Canis lupus", "271": "white wolf, Arctic wolf, Canis lupus tundrarum", "272": "red wolf, maned wolf, Canis rufus, Canis niger", "273": "coyote, prairie wolf, brush wolf, Canis latrans", "274": "dingo, warrigal, warragal, Canis dingo", "275": "dhole, Cuon alpinus", "276": "African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus", "277": "hyena, hyaena", "278": "red fox, Vulpes vulpes", "279": "kit fox, Vulpes macrotis", "280": "Arctic fox, white fox, Alopex lagopus", "281": "grey fox, gray fox, Urocyon cinereoargenteus", "282": "tabby, tabby cat", "283": "tiger cat", "284": "Persian cat", "285": "Siamese cat, Siamese", "286": "Egyptian cat", "287": "cougar, puma, catamount, mountain lion, painter, panther, Felis concolor", "288": "lynx, catamount", "289": "leopard, Panthera pardus", "290": "snow leopard, ounce, Panthera uncia", "291": "jaguar, panther, Panthera onca, Felis onca", "292": "lion, king of beasts, Panthera leo", "293": "tiger, Panthera tigris", "294": "cheetah, chetah, Acinonyx jubatus", "295": "brown bear, bruin, Ursus arctos", "296": "American black bear, black bear, Ursus americanus, Euarctos americanus", "297": "ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus", "298": "sloth bear, Melursus ursinus, Ursus ursinus", "299": "mongoose", "300": "meerkat, mierkat", "301": "tiger beetle", "302": "ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle", "303": "ground beetle, carabid beetle", "304": "long-horned beetle, longicorn, longicorn beetle", "305": "leaf beetle, chrysomelid", "306": "dung beetle", "307": "rhinoceros beetle", "308": "weevil", "309": "fly", "310": "bee", "311": "ant, emmet, pismire", "312": "grasshopper, hopper", "313": "cricket", "314": "walking stick, walkingstick, stick insect", "315": "cockroach, roach", "316": "mantis, mantid", "317": "cicada, cicala", "318": "leafhopper", "319": "lacewing, lacewing fly", "320": "dragonfly, darning needle, devil's darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk", "321": "damselfly", "322": "admiral", "323": "ringlet, ringlet butterfly", "324": "monarch, monarch butterfly, milkweed butterfly, Danaus plexippus", "325": "cabbage butterfly", "326": "sulphur butterfly, sulfur butterfly", "327": "lycaenid, lycaenid butterfly", "328": "starfish, sea star", "329": "sea urchin", "330": "sea cucumber, holothurian", "331": "wood rabbit, cottontail, cottontail rabbit", "332": "hare", "333": "Angora, Angora rabbit", "334": "hamster", "335": "porcupine, hedgehog", "336": "fox squirrel, eastern fox squirrel, Sciurus niger", "337": "marmot", "338": "beaver", "339": "guinea pig, Cavia cobaya", "340": "sorrel", "341": "zebra", "342": "hog, pig, grunter, squealer, Sus scrofa", "343": "wild boar, boar, Sus scrofa", "344": "warthog", "345": "hippopotamus, hippo, river horse, Hippopotamus amphibius", "346": "ox", "347": "water buffalo, water ox, Asiatic buffalo, Bubalus bubalis", "348": "bison", "349": "ram, tup", "350": "bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis", "351": "ibex, Capra ibex", "352": "hartebeest", "353": "impala, Aepyceros melampus", "354": "gazelle", "355": "Arabian camel, dromedary, Camelus dromedarius", "356": "llama", "357": "weasel", "358": "mink", "359": "polecat, fitch, foulmart, foumart, Mustela putorius", "360": "black-footed ferret, ferret, Mustela nigripes", "361": "otter", "362": "skunk, polecat, wood pussy", "363": "badger", "364": "armadillo", "365": "three-toed sloth, ai, Bradypus tridactylus", "366": "orangutan, orang, orangutang, Pongo pygmaeus", "367": "gorilla, Gorilla gorilla", "368": "chimpanzee, chimp, Pan troglodytes", "369": "gibbon, Hylobates lar", "370": "siamang, Hylobates syndactylus, Symphalangus syndactylus", "371": "guenon, guenon monkey", "372": "patas, hussar monkey, Erythrocebus patas", "373": "baboon", "374": "macaque", "375": "langur", "376": "colobus, colobus monkey", "377": "proboscis monkey, Nasalis larvatus", "378": "marmoset", "379": "capuchin, ringtail, Cebus capucinus", "380": "howler monkey, howler", "381": "titi, titi monkey", "382": "spider monkey, Ateles geoffroyi", "383": "squirrel monkey, Saimiri sciureus", "384": "Madagascar cat, ring-tailed lemur, Lemur catta", "385": "indri, indris, Indri indri, Indri brevicaudatus", "386": "Indian elephant, Elephas maximus", "387": "African elephant, Loxodonta africana", "388": "lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens", "389": "giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca", "390": "barracouta, snoek", "391": "eel", "392": "coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch", "393": "rock beauty, Holocanthus tricolor", "394": "anemone fish", "395": "sturgeon", "396": "gar, garfish, garpike, billfish, Lepisosteus osseus", "397": "lionfish", "398": "puffer, pufferfish, blowfish, globefish", "399": "abacus", "400": "abaya", "401": "academic gown, academic robe, judge's robe", "402": "accordion, piano accordion, squeeze box", "403": "acoustic guitar", "404": "aircraft carrier, carrier, flattop, attack aircraft carrier", "405": "airliner", "406": "airship, dirigible", "407": "altar", "408": "ambulance", "409": "amphibian, amphibious vehicle", "410": "analog clock", "411": "apiary, bee house", "412": "apron", "413": "ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin", "414": "assault rifle, assault gun", "415": "backpack, back pack, knapsack, packsack, rucksack, haversack", "416": "bakery, bakeshop, bakehouse", "417": "balance beam, beam", "418": "balloon", "419": "ballpoint, ballpoint pen, ballpen, Biro", "420": "Band Aid", "421": "banjo", "422": "bannister, banister, balustrade, balusters, handrail", "423": "barbell", "424": "barber chair", "425": "barbershop", "426": "barn", "427": "barometer", "428": "barrel, cask", "429": "barrow, garden cart, lawn cart, wheelbarrow", "430": "baseball", "431": "basketball", "432": "bassinet", "433": "bassoon", "434": "bathing cap, swimming cap", "435": "bath towel", "436": "bathtub, bathing tub, bath, tub", "437": "beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon", "438": "beacon, lighthouse, beacon light, pharos", "439": "beaker", "440": "bearskin, busby, shako", "441": "beer bottle", "442": "beer glass", "443": "bell cote, bell cot", "444": "bib", "445": "bicycle-built-for-two, tandem bicycle, tandem", "446": "bikini, two-piece", "447": "binder, ring-binder", "448": "binoculars, field glasses, opera glasses", "449": "birdhouse", "450": "boathouse", "451": "bobsled, bobsleigh, bob", "452": "bolo tie, bolo, bola tie, bola", "453": "bonnet, poke bonnet", "454": "bookcase", "455": "bookshop, bookstore, bookstall", "456": "bottlecap", "457": "bow", "458": "bow tie, bow-tie, bowtie", "459": "brass, memorial tablet, plaque", "460": "brassiere, bra, bandeau", "461": "breakwater, groin, groyne, mole, bulwark, seawall, jetty", "462": "breastplate, aegis, egis", "463": "broom", "464": "bucket, pail", "465": "buckle", "466": "bulletproof vest", "467": "bullet train, bullet", "468": "butcher shop, meat market", "469": "cab, hack, taxi, taxicab", "470": "caldron, cauldron", "471": "candle, taper, wax light", "472": "cannon", "473": "canoe", "474": "can opener, tin opener", "475": "cardigan", "476": "car mirror", "477": "carousel, carrousel, merry-go-round, roundabout, whirligig", "478": "carpenter's kit, tool kit", "479": "carton", "480": "car wheel", "481": "cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM", "482": "cassette", "483": "cassette player", "484": "castle", "485": "catamaran", "486": "CD player", "487": "cello, violoncello", "488": "cellular telephone, cellular phone, cellphone, cell, mobile phone", "489": "chain", "490": "chainlink fence", "491": "chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour", "492": "chain saw, chainsaw", "493": "chest", "494": "chiffonier, commode", "495": "chime, bell, gong", "496": "china cabinet, china closet", "497": "Christmas stocking", "498": "church, church building", "499": "cinema, movie theater, movie theatre, movie house, picture palace", "500": "cleaver, meat cleaver, chopper", "501": "cliff dwelling", "502": "cloak", "503": "clog, geta, patten, sabot", "504": "cocktail shaker", "505": "coffee mug", "506": "coffeepot", "507": "coil, spiral, volute, whorl, helix", "508": "combination lock", "509": "computer keyboard, keypad", "510": "confectionery, confectionary, candy store", "511": "container ship, containership, container vessel", "512": "convertible", "513": "corkscrew, bottle screw", "514": "cornet, horn, trumpet, trump", "515": "cowboy boot", "516": "cowboy hat, ten-gallon hat", "517": "cradle", "518": "crane", "519": "crash helmet", "520": "crate", "521": "crib, cot", "522": "Crock Pot", "523": "croquet ball", "524": "crutch", "525": "cuirass", "526": "dam, dike, dyke", "527": "desk", "528": "desktop computer", "529": "dial telephone, dial phone", "530": "diaper, nappy, napkin", "531": "digital clock", "532": "digital watch", "533": "dining table, board", "534": "dishrag, dishcloth", "535": "dishwasher, dish washer, dishwashing machine", "536": "disk brake, disc brake", "537": "dock, dockage, docking facility", "538": "dogsled, dog sled, dog sleigh", "539": "dome", "540": "doormat, welcome mat", "541": "drilling platform, offshore rig", "542": "drum, membranophone, tympan", "543": "drumstick", "544": "dumbbell", "545": "Dutch oven", "546": "electric fan, blower", "547": "electric guitar", "548": "electric locomotive", "549": "entertainment center", "550": "envelope", "551": "espresso maker", "552": "face powder", "553": "feather boa, boa", "554": "file, file cabinet, filing cabinet", "555": "fireboat", "556": "fire engine, fire truck", "557": "fire screen, fireguard", "558": "flagpole, flagstaff", "559": "flute, transverse flute", "560": "folding chair", "561": "football helmet", "562": "forklift", "563": "fountain", "564": "fountain pen", "565": "four-poster", "566": "freight car", "567": "French horn, horn", "568": "frying pan, frypan, skillet", "569": "fur coat", "570": "garbage truck, dustcart", "571": "gasmask, respirator, gas helmet", "572": "gas pump, gasoline pump, petrol pump, island dispenser", "573": "goblet", "574": "go-kart", "575": "golf ball", "576": "golfcart, golf cart", "577": "gondola", "578": "gong, tam-tam", "579": "gown", "580": "grand piano, grand", "581": "greenhouse, nursery, glasshouse", "582": "grille, radiator grille", "583": "grocery store, grocery, food market, market", "584": "guillotine", "585": "hair slide", "586": "hair spray", "587": "half track", "588": "hammer", "589": "hamper", "590": "hand blower, blow dryer, blow drier, hair dryer, hair drier", "591": "hand-held computer, hand-held microcomputer", "592": "handkerchief, hankie, hanky, hankey", "593": "hard disc, hard disk, fixed disk", "594": "harmonica, mouth organ, harp, mouth harp", "595": "harp", "596": "harvester, reaper", "597": "hatchet", "598": "holster", "599": "home theater, home theatre", "600": "honeycomb", "601": "hook, claw", "602": "hoopskirt, crinoline", "603": "horizontal bar, high bar", "604": "horse cart, horse-cart", "605": "hourglass", "606": "iPod", "607": "iron, smoothing iron", "608": "jack-o'-lantern", "609": "jean, blue jean, denim", "610": "jeep, landrover", "611": "jersey, T-shirt, tee shirt", "612": "jigsaw puzzle", "613": "jinrikisha, ricksha, rickshaw", "614": "joystick", "615": "kimono", "616": "knee pad", "617": "knot", "618": "lab coat, laboratory coat", "619": "ladle", "620": "lampshade, lamp shade", "621": "laptop, laptop computer", "622": "lawn mower, mower", "623": "lens cap, lens cover", "624": "letter opener, paper knife, paperknife", "625": "library", "626": "lifeboat", "627": "lighter, light, igniter, ignitor", "628": "limousine, limo", "629": "liner, ocean liner", "630": "lipstick, lip rouge", "631": "Loafer", "632": "lotion", "633": "loudspeaker, speaker, speaker unit, loudspeaker system, speaker system", "634": "loupe, jeweler's loupe", "635": "lumbermill, sawmill", "636": "magnetic compass", "637": "mailbag, postbag", "638": "mailbox, letter box", "639": "maillot", "640": "maillot, tank suit", "641": "manhole cover", "642": "maraca", "643": "marimba, xylophone", "644": "mask", "645": "matchstick", "646": "maypole", "647": "maze, labyrinth", "648": "measuring cup", "649": "medicine chest, medicine cabinet", "650": "megalith, megalithic structure", "651": "microphone, mike", "652": "microwave, microwave oven", "653": "military uniform", "654": "milk can", "655": "minibus", "656": "miniskirt, mini", "657": "minivan", "658": "missile", "659": "mitten", "660": "mixing bowl", "661": "mobile home, manufactured home", "662": "Model T", "663": "modem", "664": "monastery", "665": "monitor", "666": "moped", "667": "mortar", "668": "mortarboard", "669": "mosque", "670": "mosquito net", "671": "motor scooter, scooter", "672": "mountain bike, all-terrain bike, off-roader", "673": "mountain tent", "674": "mouse, computer mouse", "675": "mousetrap", "676": "moving van", "677": "muzzle", "678": "nail", "679": "neck brace", "680": "necklace", "681": "nipple", "682": "notebook, notebook computer", "683": "obelisk", "684": "oboe, hautboy, hautbois", "685": "ocarina, sweet potato", "686": "odometer, hodometer, mileometer, milometer", "687": "oil filter", "688": "organ, pipe organ", "689": "oscilloscope, scope, cathode-ray oscilloscope, CRO", "690": "overskirt", "691": "oxcart", "692": "oxygen mask", "693": "packet", "694": "paddle, boat paddle", "695": "paddlewheel, paddle wheel", "696": "padlock", "697": "paintbrush", "698": "pajama, pyjama, pj's, jammies", "699": "palace", "700": "panpipe, pandean pipe, syrinx", "701": "paper towel", "702": "parachute, chute", "703": "parallel bars, bars", "704": "park bench", "705": "parking meter", "706": "passenger car, coach, carriage", "707": "patio, terrace", "708": "pay-phone, pay-station", "709": "pedestal, plinth, footstall", "710": "pencil box, pencil case", "711": "pencil sharpener", "712": "perfume, essence", "713": "Petri dish", "714": "photocopier", "715": "pick, plectrum, plectron", "716": "pickelhaube", "717": "picket fence, paling", "718": "pickup, pickup truck", "719": "pier", "720": "piggy bank, penny bank", "721": "pill bottle", "722": "pillow", "723": "ping-pong ball", "724": "pinwheel", "725": "pirate, pirate ship", "726": "pitcher, ewer", "727": "plane, carpenter's plane, woodworking plane", "728": "planetarium", "729": "plastic bag", "730": "plate rack", "731": "plow, plough", "732": "plunger, plumber's helper", "733": "Polaroid camera, Polaroid Land camera", "734": "pole", "735": "police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria", "736": "poncho", "737": "pool table, billiard table, snooker table", "738": "pop bottle, soda bottle", "739": "pot, flowerpot", "740": "potter's wheel", "741": "power drill", "742": "prayer rug, prayer mat", "743": "printer", "744": "prison, prison house", "745": "projectile, missile", "746": "projector", "747": "puck, hockey puck", "748": "punching bag, punch bag, punching ball, punchball", "749": "purse", "750": "quill, quill pen", "751": "quilt, comforter, comfort, puff", "752": "racer, race car, racing car", "753": "racket, racquet", "754": "radiator", "755": "radio, wireless", "756": "radio telescope, radio reflector", "757": "rain barrel", "758": "recreational vehicle, RV, R.V.", "759": "reel", "760": "reflex camera", "761": "refrigerator, icebox", "762": "remote control, remote", "763": "restaurant, eating house, eating place, eatery", "764": "revolver, six-gun, six-shooter", "765": "rifle", "766": "rocking chair, rocker", "767": "rotisserie", "768": "rubber eraser, rubber, pencil eraser", "769": "rugby ball", "770": "rule, ruler", "771": "running shoe", "772": "safe", "773": "safety pin", "774": "saltshaker, salt shaker", "775": "sandal", "776": "sarong", "777": "sax, saxophone", "778": "scabbard", "779": "scale, weighing machine", "780": "school bus", "781": "schooner", "782": "scoreboard", "783": "screen, CRT screen", "784": "screw", "785": "screwdriver", "786": "seat belt, seatbelt", "787": "sewing machine", "788": "shield, buckler", "789": "shoe shop, shoe-shop, shoe store", "790": "shoji", "791": "shopping basket", "792": "shopping cart", "793": "shovel", "794": "shower cap", "795": "shower curtain", "796": "ski", "797": "ski mask", "798": "sleeping bag", "799": "slide rule, slipstick", "800": "sliding door", "801": "slot, one-armed bandit", "802": "snorkel", "803": "snowmobile", "804": "snowplow, snowplough", "805": "soap dispenser", "806": "soccer ball", "807": "sock", "808": "solar dish, solar collector, solar furnace", "809": "sombrero", "810": "soup bowl", "811": "space bar", "812": "space heater", "813": "space shuttle", "814": "spatula", "815": "speedboat", "816": "spider web, spider's web", "817": "spindle", "818": "sports car, sport car", "819": "spotlight, spot", "820": "stage", "821": "steam locomotive", "822": "steel arch bridge", "823": "steel drum", "824": "stethoscope", "825": "stole", "826": "stone wall", "827": "stopwatch, stop watch", "828": "stove", "829": "strainer", "830": "streetcar, tram, tramcar, trolley, trolley car", "831": "stretcher", "832": "studio couch, day bed", "833": "stupa, tope", "834": "submarine, pigboat, sub, U-boat", "835": "suit, suit of clothes", "836": "sundial", "837": "sunglass", "838": "sunglasses, dark glasses, shades", "839": "sunscreen, sunblock, sun blocker", "840": "suspension bridge", "841": "swab, swob, mop", "842": "sweatshirt", "843": "swimming trunks, bathing trunks", "844": "swing", "845": "switch, electric switch, electrical switch", "846": "syringe", "847": "table lamp", "848": "tank, army tank, armored combat vehicle, armoured combat vehicle", "849": "tape player", "850": "teapot", "851": "teddy, teddy bear", "852": "television, television system", "853": "tennis ball", "854": "thatch, thatched roof", "855": "theater curtain, theatre curtain", "856": "thimble", "857": "thresher, thrasher, threshing machine", "858": "throne", "859": "tile roof", "860": "toaster", "861": "tobacco shop, tobacconist shop, tobacconist", "862": "toilet seat", "863": "torch", "864": "totem pole", "865": "tow truck, tow car, wrecker", "866": "toyshop", "867": "tractor", "868": "trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi", "869": "tray", "870": "trench coat", "871": "tricycle, trike, velocipede", "872": "trimaran", "873": "tripod", "874": "triumphal arch", "875": "trolleybus, trolley coach, trackless trolley", "876": "trombone", "877": "tub, vat", "878": "turnstile", "879": "typewriter keyboard", "880": "umbrella", "881": "unicycle, monocycle", "882": "upright, upright piano", "883": "vacuum, vacuum cleaner", "884": "vase", "885": "vault", "886": "velvet", "887": "vending machine", "888": "vestment", "889": "viaduct", "890": "violin, fiddle", "891": "volleyball", "892": "waffle iron", "893": "wall clock", "894": "wallet, billfold, notecase, pocketbook", "895": "wardrobe, closet, press", "896": "warplane, military plane", "897": "washbasin, handbasin, washbowl, lavabo, wash-hand basin", "898": "washer, automatic washer, washing machine", "899": "water bottle", "900": "water jug", "901": "water tower", "902": "whiskey jug", "903": "whistle", "904": "wig", "905": "window screen", "906": "window shade", "907": "Windsor tie", "908": "wine bottle", "909": "wing", "910": "wok", "911": "wooden spoon", "912": "wool, woolen, woollen", "913": "worm fence, snake fence, snake-rail fence, Virginia fence", "914": "wreck", "915": "yawl", "916": "yurt", "917": "web site, website, internet site, site", "918": "comic book", "919": "crossword puzzle, crossword", "920": "street sign", "921": "traffic light, traffic signal, stoplight", "922": "book jacket, dust cover, dust jacket, dust wrapper", "923": "menu", "924": "plate", "925": "guacamole", "926": "consomme", "927": "hot pot, hotpot", "928": "trifle", "929": "ice cream, icecream", "930": "ice lolly, lolly, lollipop, popsicle", "931": "French loaf", "932": "bagel, beigel", "933": "pretzel", "934": "cheeseburger", "935": "hotdog, hot dog, red hot", "936": "mashed potato", "937": "head cabbage", "938": "broccoli", "939": "cauliflower", "940": "zucchini, courgette", "941": "spaghetti squash", "942": "acorn squash", "943": "butternut squash", "944": "cucumber, cuke", "945": "artichoke, globe artichoke", "946": "bell pepper", "947": "cardoon", "948": "mushroom", "949": "Granny Smith", "950": "strawberry", "951": "orange", "952": "lemon", "953": "fig", "954": "pineapple, ananas", "955": "banana", "956": "jackfruit, jak, jack", "957": "custard apple", "958": "pomegranate", "959": "hay", "960": "carbonara", "961": "chocolate sauce, chocolate syrup", "962": "dough", "963": "meat loaf, meatloaf", "964": "pizza, pizza pie", "965": "potpie", "966": "burrito", "967": "red wine", "968": "espresso", "969": "cup", "970": "eggnog", "971": "alp", "972": "bubble", "973": "cliff, drop, drop-off", "974": "coral reef", "975": "geyser", "976": "lakeside, lakeshore", "977": "promontory, headland, head, foreland", "978": "sandbar, sand bar", "979": "seashore, coast, seacoast, sea-coast", "980": "valley, vale", "981": "volcano", "982": "ballplayer, baseball player", "983": "groom, bridegroom", "984": "scuba diver", "985": "rapeseed", "986": "daisy", "987": "yellow lady's slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum", "988": "corn", "989": "acorn", "990": "hip, rose hip, rosehip", "991": "buckeye, horse chestnut, conker", "992": "coral fungus", "993": "agaric", "994": "gyromitra", "995": "stinkhorn, carrion fungus", "996": "earthstar", "997": "hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa", "998": "bolete", "999": "ear, spike, capitulum", "1000": "toilet tissue, toilet paper, bathroom tissue"}
\ No newline at end of file
research/tensorrt/tensorrt.py 0 → 100644
View file @ 430811a7
# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
"""Methods for running the Official Models with TensorRT.
Please note that all of these methods are in development, and subject to
rapid change.
"""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import argparse
import imghdr
import json
import os
import sys
import time
import numpy as np
import tensorflow as tf
from tensorflow.contrib.saved_model.python.saved_model import reader
import tensorflow.contrib.tensorrt as trt
from official.resnet import imagenet_preprocessing # pylint: disable=g-bad-import-order
_GPU_MEM_FRACTION = 0.50
_WARMUP_NUM_LOOPS = 50
_LOG_FILE = "log.txt"
_LABELS_FILE = "labellist.json"
_GRAPH_FILE = "frozen_graph.pb"
################################################################################
# Prep the image input to the graph.
################################################################################
def preprocess_image(file_name, output_height=224, output_width=224,
num_channels=3):
"""Run standard ImageNet preprocessing on the passed image file.
Args:
file_name: string, path to file containing a JPEG image
output_height: int, final height of image
output_width: int, final width of image
num_channels: int, depth of input image
Returns:
Float array representing processed image with shape
[output_height, output_width, num_channels]
Raises:
ValueError: if image is not a JPEG.
"""
if imghdr.what(file_name) != "jpeg":
raise ValueError("At this time, only JPEG images are supported. "
"Please try another image.")
image_buffer = tf.read_file(file_name)
normalized = imagenet_preprocessing.preprocess_image(
image_buffer=image_buffer,
bbox=None,
output_height=output_height,
output_width=output_width,
num_channels=num_channels,
is_training=False)
with tf.Session(config=get_gpu_config()) as sess:
result = sess.run([normalized])
return result[0]
def batch_from_image(file_name, batch_size, output_height=224, output_width=224,
num_channels=3):
"""Produce a batch of data from the passed image file.
Args:
file_name: string, path to file containing a JPEG image
batch_size: int, the size of the desired batch of data
output_height: int, final height of data
output_width: int, final width of data
num_channels: int, depth of input data
Returns:
Float array representing copies of the image with shape
[batch_size, output_height, output_width, num_channels]
"""
image_array = preprocess_image(
file_name, output_height, output_width, num_channels)
tiled_array = np.tile(image_array, [batch_size, 1, 1, 1])
return tiled_array
def batch_from_random(batch_size, output_height=224, output_width=224,
num_channels=3):
"""Produce a batch of random data.
Args:
batch_size: int, the size of the desired batch of data
output_height: int, final height of data
output_width: int, final width of data
num_channels: int, depth of output data
Returns:
Float array containing random numbers with shape
[batch_size, output_height, output_width, num_channels]
"""
shape = [batch_size, output_height, output_width, num_channels]
return np.random.random_sample(shape)
################################################################################
# Utils for handling Frozen Graphs.
################################################################################
def get_serving_meta_graph_def(savedmodel_dir):
"""Extract the SERVING MetaGraphDef from a SavedModel directory.
Args:
savedmodel_dir: the string path to the directory containing the .pb
and variables for a SavedModel. This is equivalent to the subdirectory
that is created under the directory specified by --export_dir when
running an Official Model.
Returns:
MetaGraphDef that should be used for tag_constants.SERVING mode.
Raises:
ValueError: if a MetaGraphDef matching tag_constants.SERVING is not found.
"""
# We only care about the serving graph def
tag_set = set([tf.saved_model.tag_constants.SERVING])
serving_graph_def = None
saved_model = reader.read_saved_model(savedmodel_dir)
for meta_graph_def in saved_model.meta_graphs:
if set(meta_graph_def.meta_info_def.tags) == tag_set:
serving_graph_def = meta_graph_def
if not serving_graph_def:
raise ValueError("No MetaGraphDef found for tag_constants.SERVING. "
"Please make sure the SavedModel includes a SERVING def.")
return serving_graph_def
def write_graph_to_file(graph_name, graph_def, output_dir):
"""Write Frozen Graph file to disk."""
output_path = os.path.join(output_dir, graph_name)
with tf.gfile.GFile(output_path, "wb") as f:
f.write(graph_def.SerializeToString())
def convert_savedmodel_to_frozen_graph(savedmodel_dir, output_dir):
"""Convert a SavedModel to a Frozen Graph.
A SavedModel includes a `variables` directory with variable values,
and a specification of the MetaGraph in a ProtoBuffer file. A Frozen Graph
takes the variable values and inserts them into the graph, such that the
SavedModel is all bundled into a single file. TensorRT and TFLite both
leverage Frozen Graphs. Here, we provide a simple utility for converting
a SavedModel into a frozen graph for use with these other tools.
Args:
savedmodel_dir: the string path to the directory containing the .pb
and variables for a SavedModel. This is equivalent to the subdirectory
that is created under the directory specified by --export_dir when
running an Official Model.
output_dir: string representing path to the output directory for saving
the frozen graph.
Returns:
Frozen Graph definition for use.
"""
meta_graph = get_serving_meta_graph_def(savedmodel_dir)
signature_def = tf.contrib.saved_model.get_signature_def_by_key(
meta_graph,
tf.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY)
outputs = [v.name for v in signature_def.outputs.itervalues()]
output_names = [node.split(":")[0] for node in outputs]
graph = tf.Graph()
with tf.Session(graph=graph) as sess:
tf.saved_model.loader.load(
sess, meta_graph.meta_info_def.tags, savedmodel_dir)
frozen_graph_def = tf.graph_util.convert_variables_to_constants(
sess, graph.as_graph_def(), output_names)
write_graph_to_file(_GRAPH_FILE, frozen_graph_def, output_dir)
return frozen_graph_def
def get_frozen_graph(graph_file):
"""Read Frozen Graph file from disk."""
with tf.gfile.FastGFile(graph_file, "rb") as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
return graph_def
def get_tftrt_name(graph_name, precision_string):
return "tftrt_{}_{}".format(precision_string.lower(), graph_name)
def get_trt_graph(graph_name, graph_def, precision_mode, output_dir,
output_node, batch_size=128, workspace_size=1<<30):
"""Create and save inference graph using the TensorRT library.
Args:
graph_name: string, name of the graph to be used for saving.
graph_def: GraphDef, the Frozen Graph to be converted.
precision_mode: string, the precision that TensorRT should convert into.
Options- FP32, FP16, INT8.
output_dir: string, the path to where files should be written.
output_node: string, the names of the output node that will
be returned during inference.
batch_size: int, the number of examples that will be predicted at a time.
workspace_size: long, size in bytes that can be used during conversion.
Returns:
GraphDef for the TensorRT inference graph.
"""
trt_graph = trt.create_inference_graph(
graph_def, [output_node], max_batch_size=batch_size,
max_workspace_size_bytes=workspace_size,
precision_mode=precision_mode)
write_graph_to_file(graph_name, trt_graph, output_dir)
return trt_graph
def get_trt_graph_from_calib(graph_name, calib_graph_def, output_dir):
"""Convert a TensorRT graph used for calibration to an inference graph."""
trt_graph = trt.calib_graph_to_infer_graph(calib_graph_def)
write_graph_to_file(graph_name, trt_graph, output_dir)
return trt_graph
################################################################################
# Run the graph in various precision modes.
################################################################################
def get_gpu_config():
"""Share GPU memory between image preprocessing and inference."""
gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=_GPU_MEM_FRACTION)
return tf.ConfigProto(gpu_options=gpu_options)
def get_iterator(data):
"""Wrap numpy data in a dataset."""
dataset = tf.data.Dataset.from_tensors(data).repeat()
return dataset.make_one_shot_iterator()
def time_graph(graph_def, data, input_node, output_node, num_loops=100):
"""Run and time the inference graph.
This function sets up the input and outputs for inference, warms up by
running inference for _WARMUP_NUM_LOOPS, then times inference for num_loops
loops.
Args:
graph_def: GraphDef, the graph to be timed.
data: ndarray of shape [batch_size, height, width, depth], data to be
predicted.
input_node: string, the label of the input node where data will enter the
graph.
output_node: string, the names of the output node that will
be returned during inference.
num_loops: int, number of batches that should run through for timing.
Returns:
A tuple consisting of a list of num_loops inference times, and the
predictions that were output for the batch.
"""
tf.logging.info("Starting execution")
tf.reset_default_graph()
g = tf.Graph()
with g.as_default():
iterator = get_iterator(data)
return_tensors = tf.import_graph_def(
graph_def=graph_def,
input_map={input_node: iterator.get_next()},
return_elements=[output_node]
)
# Unwrap the returned output node. For now, we assume we only
# want the tensor with index `:0`, which is the 0th element of the
# `.outputs` list.
output = return_tensors[0].outputs[0]
timings = []
with tf.Session(graph=g, config=get_gpu_config()) as sess:
tf.logging.info("Starting Warmup cycle")
for _ in range(_WARMUP_NUM_LOOPS):
sess.run([output])
tf.logging.info("Starting timing.")
for _ in range(num_loops):
tstart = time.time()
val = sess.run([output])
timings.append(time.time() - tstart)
tf.logging.info("Timing loop done!")
return timings, val[0]
def log_stats(graph_name, log_buffer, timings, batch_size):
"""Write stats to the passed log_buffer.
Args:
graph_name: string, name of the graph to be used for reporting.
log_buffer: filehandle, log file opened for appending.
timings: list of floats, times produced for multiple runs that will be
used for statistic calculation
batch_size: int, number of examples per batch
Returns:
list of two floats representing frames per second and 99th percentile
time per batch
"""
times = np.array(timings)
steps = len(times)
speeds = batch_size / times
time_mean = np.mean(times)
time_med = np.median(times)
time_99th = np.percentile(times, 99)
time_99th_uncertainty = np.abs(np.percentile(times[0::2], 99) -
np.percentile(times[1::2], 99))
speed_mean = np.mean(speeds)
speed_med = np.median(speeds)
speed_uncertainty = np.std(speeds, ddof=1) / np.sqrt(float(steps))
speed_jitter = 1.4826 * np.median(np.abs(speeds - np.median(speeds)))
msg = ("\n==========================\n"
"network: %s,\t batchsize %d, steps %d\n"
" fps \tmedian: %.1f, \tmean: %.1f, \tuncertainty: %.1f, \tjitter: %.1f\n" # pylint: disable=line-too-long
" latency \tmedian: %.5f, \tmean: %.5f, \t99th_p: %.5f, \t99th_uncertainty: %.5f\n" # pylint: disable=line-too-long
) % (graph_name, batch_size, steps,
speed_med, speed_mean, speed_uncertainty, speed_jitter,
time_med, time_mean, time_99th, time_99th_uncertainty)
log_buffer.write(msg)
return [speed_mean, time_99th]
def time_and_log_graph(graph_name, graph_def, data, log_buffer, flags):
timings, result = time_graph(
graph_def, data, flags.input_node, flags.output_node, flags.num_loops)
log_stats(graph_name, log_buffer, timings, flags.batch_size)
return result
def run_trt_graph_for_mode(
graph_name, graph_def, mode, data, log_buffer, flags):
"""Convert, time, and log the graph at `mode` precision using TensorRT."""
g_name = get_tftrt_name(graph_name, mode)
graph = get_trt_graph(
g_name, graph_def, mode, flags.output_dir, flags.output_node,
flags.batch_size, flags.workspace_size)
result = time_and_log_graph(g_name, graph, data, log_buffer, flags)
return result
################################################################################
# Parse predictions
################################################################################
def get_labels():
"""Get the set of possible labels for classification."""
with open(_LABELS_FILE, "r") as labels_file:
labels = json.load(labels_file)
return labels
def top_predictions(result, n):
"""Get the top n predictions given the array of softmax results."""
# We only care about the first example.
probabilities = result[0]
# Get the ids of most probable labels. Reverse order to get greatest first.
ids = np.argsort(probabilities)[::-1]
return ids[:n]
def get_labels_for_ids(labels, ids, ids_are_one_indexed=False):
"""Get the human-readable labels for given ids.
Args:
labels: dict, string-ID to label mapping from ImageNet.
ids: list of ints, IDs to return labels for.
ids_are_one_indexed: whether to increment passed IDs by 1 to account for
the background category. See ArgParser `--ids_are_one_indexed`
for details.
Returns:
list of category labels
"""
return [labels[str(x + int(ids_are_one_indexed))] for x in ids]
def print_predictions(results, ids_are_one_indexed=False, preds_to_print=5):
"""Given an array of mode, graph_name, predicted_ID, print labels."""
labels = get_labels()
print("Predictions:")
for mode, result in results:
pred_ids = top_predictions(result, preds_to_print)
pred_labels = get_labels_for_ids(labels, pred_ids, ids_are_one_indexed)
print("Precision: ", mode, pred_labels)
################################################################################
# Run this script
################################################################################
def main(argv):
parser = TensorRTParser()
flags = parser.parse_args(args=argv[1:])
# Load the data.
if flags.image_file:
data = batch_from_image(flags.image_file, flags.batch_size)
else:
data = batch_from_random(flags.batch_size)
# Load the graph def
if flags.frozen_graph:
frozen_graph_def = get_frozen_graph(flags.frozen_graph)
elif flags.savedmodel_dir:
frozen_graph_def = convert_savedmodel_to_frozen_graph(
flags.savedmodel_dir, flags.output_dir)
else:
raise ValueError(
"Either a Frozen Graph file or a SavedModel must be provided.")
# Get a name for saving TensorRT versions of the graph.
graph_name = os.path.basename(flags.frozen_graph or _GRAPH_FILE)
# Write to a single file for all tests, continuing from previous logs.
log_buffer = open(os.path.join(flags.output_dir, _LOG_FILE), "a")
# Run inference in all desired modes.
results = []
if flags.native:
mode = "native"
print("Running {} graph".format(mode))
g_name = "{}_{}".format(mode, graph_name)
result = time_and_log_graph(
g_name, frozen_graph_def, data, log_buffer, flags)
results.append((mode, result))
if flags.fp32:
mode = "FP32"
print("Running {} graph".format(mode))
result = run_trt_graph_for_mode(
graph_name, frozen_graph_def, mode, data, log_buffer, flags)
results.append((mode, result))
if flags.fp16:
mode = "FP16"
print("Running {} graph".format(mode))
result = run_trt_graph_for_mode(
graph_name, frozen_graph_def, mode, data, log_buffer, flags)
results.append((mode, result))
if flags.int8:
mode = "INT8"
print("Running {} graph".format(mode))
save_name = get_tftrt_name(graph_name, "INT8_calib")
calib_graph = get_trt_graph(
save_name, frozen_graph_def, mode, flags.output_dir, flags.output_node,
flags.batch_size, flags.workspace_size)
time_graph(calib_graph, data, flags.input_node, flags.output_node,
num_loops=1)
g_name = get_tftrt_name(graph_name, mode)
int8_graph = get_trt_graph_from_calib(g_name, calib_graph, flags.output_dir)
result = time_and_log_graph(g_name, int8_graph, data, log_buffer, flags)
results.append((mode, result))
# Print prediction results to the command line.
print_predictions(
results, flags.ids_are_one_indexed, flags.predictions_to_print)
class TensorRTParser(argparse.ArgumentParser):
"""Parser to contain flags for running the TensorRT timers."""
def __init__(self):
super(TensorRTParser, self).__init__()
self.add_argument(
"--frozen_graph", "-fg", default=None,
help="[default: %(default)s] The location of a Frozen Graph "
"protobuf file that will be used for inference. Note that either "
"savedmodel_dir or frozen_graph should be passed in, and "
"frozen_graph will take precedence.",
metavar="<FG>",
)
self.add_argument(
"--savedmodel_dir", "-sd", default=None,
help="[default: %(default)s] The location of a SavedModel directory "
"to be converted into a Frozen Graph. This is equivalent to the "
"subdirectory that is created under the directory specified by "
"--export_dir when running an Official Model. Note that either "
"savedmodel_dir or frozen_graph should be passed in, and "
"frozen_graph will take precedence.",
metavar="<SD>",
)
self.add_argument(
"--output_dir", "-od", default="/tmp",
help="[default: %(default)s] The location where output files will "
"be saved.",
metavar="<OD>",
)
self.add_argument(
"--output_node", "-on", default="softmax_tensor",
help="[default: %(default)s] The names of the graph output node "
"that should be used when retrieving results. Assumed to be a softmax.",
metavar="<ON>",
)
self.add_argument(
"--input_node", "-in", default="input_tensor",
help="[default: %(default)s] The name of the graph input node where "
"the float image array should be fed for prediction.",
metavar="<ON>",
)
self.add_argument(
"--batch_size", "-bs", type=int, default=128,
help="[default: %(default)s] Batch size for inference. If an "
"image file is passed, it will be copied batch_size times to "
"imitate a batch.",
metavar="<BS>"
)
self.add_argument(
"--image_file", "-if", default=None,
help="[default: %(default)s] The location of a JPEG image that will "
"be passed in for inference. This will be copied batch_size times to "
"imitate a batch. If not passed, random data will be used.",
metavar="<IF>",
)
self.add_argument(
"--native", action="store_true",
help="[default: %(default)s] If set, benchmark the model "
"with it's native precision and without TensorRT."
)
self.add_argument(
"--fp32", action="store_true",
help="[default: %(default)s] If set, benchmark the model with TensorRT "
"using fp32 precision."
)
self.add_argument(
"--fp16", action="store_true",
help="[default: %(default)s] If set, benchmark the model with TensorRT "
"using fp16 precision."
)
self.add_argument(
"--int8", action="store_true",
help="[default: %(default)s] If set, benchmark the model with TensorRT "
"using int8 precision."
)
self.add_argument(
"--num_loops", "-nl", type=int, default=100,
help="[default: %(default)s] Number of inferences to time per "
"benchmarked model.",
metavar="<NL>"
)
self.add_argument(
"--workspace_size", "-ws", type=long, default=2<<30,
help="[default: %(default)s] Workspace size in bytes.",
metavar="<WS>"
)
self.add_argument(
"--ids_are_one_indexed", action="store_true",
help="[default: %(default)s] Some ResNet models include a `background` "
"category, and others do not. If the model used includes `background` "
"at index 0 in the output and represents all 1001 categories, "
"this should be False. If the model used omits the `background` label "
"and has only 1000 categories, this should be True."
)
self.add_argument(
"--predictions_to_print", "-pp", type=int, default=5,
help="[default: %(default)s] Number of predicted labels to predict.",
metavar="<PP>"
)
if __name__ == "__main__":
tf.logging.set_verbosity(tf.logging.INFO)
main(argv=sys.argv)
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment