Commit 5852200b authored by mibaumgartner's avatar mibaumgartner
Browse files

env collection, smaller changes

parent b37e01f3
......@@ -18,6 +18,32 @@ The resulting self-configuring method, nnDetection, adapts itself without any ma
We demonstrate the effectiveness of nnDetection on two public benchmarks, ADAM and LUNA16, and propose 10 further public data sets for a comprehensive evaluation of medical object detection methods.
# Installation
## Docker Installation
The easiest way to get started with nnDetection is the provided is to build a Docker Container with the provided Dockerfile.
Please install docker and [nvidia-docker2](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) before continuing.
All projects which are based on nnDetection assume that the base image was built with the following tagging scheme `nnDetection:[version]`.
To build a container (nnDetection Version 0.1) run the following command from the base directory:
```bash
docker build -t nndetection:0.1 --build-arg env_det_num_threads=6 --build-arg env_det_verbose=1 .
```
(`--build-arg env_det_num_threads=6` and `--build-arg env_det_verbose=1` are optional and are used to overwrite the provided default parameters)
The docker container expects data and models in its own `/opt/data` and `/opt/models` directories respectively.
The directories need to be mounted via docker `-v`. For simplicity and speed, the ENV variables `det_data` and `det_models` can be set in the host system to point to the desired directories. To run:
```bash
docker run --gpus all -v ${det_data}:/opt/data -v ${det_models}:/opt/models -it --shm-size=24gb nndetection:0.1 /bin/bash
```
Warning:
When running a training inside the container it is necessary to [increase the shared memory](https://stackoverflow.com/questions/30210362/how-to-increase-the-size-of-the-dev-shm-in-docker-container) (via --shm-size).
## Source Installation
1. Install CUDA (>10.1) and cudnn (make sure to select [compatible versions](https://docs.nvidia.com/deeplearning/cudnn/support-matrix/index.html)!)
2. [Optional] Depending on your GPU you might need to set `TORCH_CUDA_ARCH_LIST`, check [compute capabilities](https://developer.nvidia.com/cuda-gpus) here.
3. Install [torch](https://pytorch.org/) (make sure to match the pytorch and CUDA versions!) (requires pytorch >1.7+)
......@@ -52,34 +78,6 @@ To test the whole installation please run the Toy Dataset example.
To get the best possible performance we recommend using CUDA 11.0+ with cuDNN 8.1.X+ and a (!)locally compiled version(!) of Pytorch 1.7+
</details>
<details close>
<summary>Docker Container</summary>
<br>
The provided Dockerfile can be used to setup quick development environments or deploy nnDetection.
Please install docker and [nvidia-docker2](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) before continuing.
All projects which are based on nnDetection assume that the base image was built with the following tagging scheme `nnDetection:[version]`.
To build a container (nnDetection Version 0.1) run the following command from the base directory:
```bash
docker build -t nndetection:0.1 --build-arg env_det_num_threads=6 --build-arg env_det_verbose=1 .
```
(`--build-arg env_det_num_threads=6` and `--build-arg env_det_verbose=1` are optional and are used to overwrite the provided default parameters)
The docker container expects data and models in its own `/opt/data` and `/opt/models` directories respectively.
The directories need to be mounted via docker `-v`. For simplicity and speed, the ENV variables `det_data` and `det_models` can be set in the host system to point to the desired directories. To run:
```bash
docker run --gpus all -v ${det_data}:/opt/data -v ${det_models}:/opt/models -it --shm-size=24gb nndetection:0.1 /bin/bash
```
Warning:
When running a training inside the container it is necessary to [increase the shared memory](https://stackoverflow.com/questions/30210362/how-to-increase-the-size-of-the-dev-shm-in-docker-container) (via --shm-size).
</details>
# nnDetection
<div align="center">
<img src=docs/source/nnDetectionFunctional.svg width="600px">
......@@ -134,11 +132,12 @@ After running the generation script follow the `Planning`, `Training` and `Infer
</div>
## Adding New Datasets
nnDetection relie on a standardized input format which is very similar to the [nnU-Net](https://github.com/MIC-DKFZ/nnUNet) format and allows easy integration of new datasets.
The format is explained below.
nnDetection relies on a standardized input format which is very similar to the [nnU-Net](https://github.com/MIC-DKFZ/nnUNet) format and allows easy integration of new datasets.
More details about the format can be found below.
### Folders
All datasets should reside inside `Task[Number]_[Name]` folder inside the specified detection data folder (et the path to this folder with the `det_data` environment flag).
To avoid conflicts with our provided pretrained models we recommend to use task numbers starting from 100.
An overview is provided below ([Name] symbolise folder, `-` symbolise files, indents refer to substructures)
```text
......@@ -317,7 +316,7 @@ nndet_eval 000 RetinaUNetV001_D3V001_3d 0 --boxes --analyze_boxes
```
### Inference
After running all fold it is time to collect the models and creat a unified inference plan.
After running all folds it is time to collect the models and creat a unified inference plan.
The following command will copy all the models and predictions per fold and by adding the `sweep` options the empiricaly hyperparameter optimization across all fold can be started.
This will generate a unified plan for all models which will be used during inference.
......@@ -331,6 +330,7 @@ nndet_consolidate 000 RetinaUNetV001_D3V001_3d --sweep_boxes
# /experiments/consolidate.py - main()
```
For the final test set predictions simply select the best model according to the validation scores and run the prediction command below.
Data which is located in `raw_splitted/imagesTs` will be automatically preprocessed and predicted by running the following command:
```bash
nndet_predict [task] [model] [--fold] [--num_models] [--num_tta] [--no_preprocess]
......
......@@ -155,7 +155,7 @@ def analyze_instances_per_case(analyzer: DatasetAnalyzer,
props["num_instances"] = count_instances(props, all_classes)
props["has_classes"] = list(set(props["instances"].values()))
props["volume_per_class"], props["region_volume_per_class"] = \
instance_class_and_region_sizes(iseg, props, all_classes)
instance_class_and_region_sizes(case_id, iseg, props, all_classes)
props["boxes"] = iseg_to_boxes(iseg)
props["all_ious"], props["class_ious"] = case_ious(props["boxes"], props)
return props
......@@ -178,6 +178,7 @@ def count_instances(props: dict, all_classes: Sequence[int]) -> Dict[int, int]:
def instance_class_and_region_sizes(
case_id: str,
iseg: np.ndarray,
props: dict,
all_classes: Sequence[int],
......@@ -209,7 +210,8 @@ def instance_class_and_region_sizes(
ids = np.unique(iseg)
ids = ids[ids > 0]
if len(ids) != len(list(instance_classes.keys())):
logger.warning(f"Instance lost. Found {instance_classes} in properties but {ids} in seg.")
logger.warning(f"Instance lost. Found {instance_classes} in "
f"properties but {ids} in seg of {case_id}.")
volumer_per_instance = {c: np.sum(iseg == c) * vol_per_voxel for c in ids}
for instance_id, instance_vol in volumer_per_instance.items():
......@@ -278,13 +280,10 @@ def case_ious(boxes: np.ndarray, props: dict) -> Tuple[np.ndarray, Dict[int, np.
class_ious = OrderedDict()
case_classes = list(set(map(int, props["instances"].values())))
case_instances = sorted(list(map(int, props["instances"].keys())))
try: # TODO: fix bug in case of missing instances
for cls in case_classes:
cls_box_indices = [props["instances"][str(ci)] == cls for ci in case_instances]
class_ious[cls] = compute_each_iou(boxes[cls_box_indices])
except:
pass
# from IPython import embed; embed()
for cls in case_classes:
cls_box_indices = [props["instances"][str(ci)] == cls for ci in case_instances]
class_ious[cls] = compute_each_iou(boxes[cls_box_indices])
else:
all_ious = np.array([])
class_ious = {}
......
......@@ -178,3 +178,21 @@ def unpack():
p = args.path
num_processes = args.num_processes
unpack_dataset(p, num_processes, False)
def env():
import os
import torch
print(f"PyTorch Version: {torch.version}")
print(f"PyTorch CUDA: {torch.version.cuda}")
print(f"PyTorch Backend cudnn: {torch.backends.cudnn.version()}")
print(f"PyTorch CUDA Arch List: {torch.cuda.get_arch_list()}")
print(f"PyTorch Current Device Capability: {torch.cuda.get_device_capability()}")
print(f"PyTorch CUDA available: {torch.cuda.is_available()}")
stream = os.popen('nvcc --version')
output = stream.read()
print(f"System NVCC: {output}")
print(f"System Arch List: {os.getenv('TORCH_CUDA_ARCH_LIST', None)}")
print(f"System OMP_NUM_THREADS: {os.getenv('OMP_NUM_THREADS', None)}")
print(f"System CUDA_HOME is None: {os.getenv('CUDA_HOME', None) is None}")
......@@ -128,6 +128,7 @@ setup(
'nndet_boxes2nii = scripts.utils:boxes2nii',
'nndet_seg2nii = scripts.utils:seg2nii',
'nndet_unpack = scripts.utils:unpack',
'nndet_env = scripts.utils:env',
]
},
)
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment