English | [简体中文](README_CN.md)

# 3D Medical image Segmentaion Solution

## Contents
0. Introduction
1. News
2. EISeg-Med3D
3. Performance
4. Quick Start
5. Full Tutorial
6. Train On Your Dataset
7. Structure
8. TODO
9. License
10. Acknowledgement

## <img src="https://user-images.githubusercontent.com/34859558/190043857-bfbdaf8b-d2dc-4fff-81c7-e0aac50851f9.png" width="25"/> Introduction

Medical image segmentation is a pixel-wise/voxel-wise classification of images generated by medical imaging, so that different organs or tissues can be distinguished. It is widely used in medical diagnosis and treatment planning.

Medical image segmentation can be divided into 2D medical image segmentation and 3D medical image segmentation. 2D medical image segmentation is supported in PaddleSeg. For details, please see [Fundus Data Segmentation Instructions](../../configs/unet/), while 3D image segmentation is handled by MedicalSeg.

**MedicalSeg is an easy-to-use 3D medical image segmentation solution** supporting the whole segmentation process including data preprocessing, model training, and model deployment.

The MedicalSeg panorama is as follows, and its main features include:
* Contains the APIs of the whole process medical image segmentation process from data labeling, training, to deployment.
* Including the 3D medical image annotation platform EISeg-Med3D to achieve efficient, accurate and easy-to-use labeling.
* Support six cutting-edge models nnUNet, nnFormer, SwinUNet, TransUNet, UNETR, VNet and corresponding high-precision pre-training models.

<p align="center">
<img src="https://github.com/shiyutang/files/raw/main/ezgif.com-gif-maker%20(1).gif" width="30.6%" height="20%"><img src="https://github.com/shiyutang/files/raw/main/ezgif.com-gif-maker.gif" width="40.6%" height="20%">
<p align="center">
    VNet segmentation result on COVID-19 CT scans (mDice on evalset is 97.04%) & MRISpineSeg (16 class mDice on evalset is 89.14%)
</p>
</p>


<p align="center">
<img src="https://github.com/shiyutang/files/raw/main/meeicalsegall.png" width="70.6%" height="20%">
<p align="center">
    <b>MedicalSeg Whole View</b>
</p>
</p>

## <img src="https://user-images.githubusercontent.com/34859558/190043516-eed25535-10e8-4853-8601-6bcf7ff58197.png" width="25"/> News
- [2022-9] Release MedicalSegV2. It provides 3 cutting-edge models to support whole process deployment applications, including nnformer, TransUnet and nnUnet, allowing you to experience a stronger and more accurate segmentation effect; a new 3D medical image intelligent annotation platform [EISeg-Med3D]( ../../EISeg/med3d/README_en.md) to quickly and easily achieve accurate 3D medical image annotation.
- [2022-4] Release MedicalSegV1. It provides the whole process from data preprocessing in 3D medical image segmentation to training and deployment, including native support for five datasets, and high-precision preprocessing on vertebrae and lungs Train the model.

## <img src="../../docs/images/chat.png" width="25"/> Communicate with us
**MedicalSeg just released! If you find any problem using it or want to share any future develop suggestions, please open a github issue or join us by scanning the following QR code.**

<div align="center">
<img src="https://user-images.githubusercontent.com/48433081/174770518-e6b5319b-336f-45d9-9817-da12b1961fb1.jpg"  width = "200" />  
</div>



## <img src="https://user-images.githubusercontent.com/34859558/188419267-bd117697-7456-4c72-8cbe-1272264d4fe4.png" width="25"/>  EISeg-Med3D
In order to solve the problem of low efficiency of 3D medical manual annotation, and to truly empower medical care with AI starting from data annotation, we built [EISeg-Med3D](../../EISeg/med3d/README_en.md), a user-friendly, efficient and intelligent 3D medical image annotation platform, which realizes intelligent and efficient 3D medical data annotation by integrating 3D interactive segmentation models in the annotation process. The main features are as follows:

* **Efficient**：Each category only needs a few clicks to generate 3d segmentation results, ten times efficient compares to time-consuming and laborious manual annotation.

* **Accurate**：The mIOU can reach 0.85 with only 3 clicks. with the segmentation editor equipped with machine learning algorithm and manual annotation, 100% accuracy is right on your hand.

* **Convenient**：Install our plugin within three steps; labeling results and progress are automatically saved; the transparency of labeling results can be adjusted to improve labeling accuracy; user-friendly interface interaction makes labeling worry-free and hassle-free。

The detailed doc of using EISeg-Med3D is [here](../../EISeg/med3d/README_en.md)

<div align="center">
<p align="center">
  <img src="https://user-images.githubusercontent.com/34859558/188415269-10526530-0415-4632-8223-0e5d755db29c.gif"  align="middle" width = 600"/>
</p>
</div>

------------------


## <img src="https://user-images.githubusercontent.com/34859558/190044217-8f6befc2-7f20-473d-b356-148e06265205.png" width="25"/> Model Performance

###  1. Accuracy

We have added cutting-edge models including nnUNet, nnFormer, SwinUNet, and TransUNet, and all of them surpassed the original paper in terms of segmentation accuracy to varying degrees. Among them, the accuracy of the reproduced TransUNet exceeded the original paper by 3.6%. 81.8% mDice segmentation accuracy.

Below we show our existing models, pre-trained model parameters and accuracy in the form of a table, welcome to download the log to view the results or load the pre-trained model to improve the training effect on the relevant data set :).


|Model| Segment | Dataset | mDice | README | Links |
|:-:|:-:|:-:|:-:|:-:|:-:|
|[nnFormer](https://arxiv.org/abs/2109.03201)| Heart |[ACDC](https://acdc.creatis.insa-lyon.fr/#phase/5846c3ab6a3c7735e84b67f2)|91.8%|[README](configs/acdc/README.md)|[model](https://paddleseg.bj.bcebos.com/paddleseg3d/acdc/nnformer_acdc_160_160_14_250k_4e-4/model.pdparams) \| [log](https://paddleseg.bj.bcebos.com/paddleseg3d/acdc/nnformer_acdc_160_160_14_250k_4e-4/train.log)\| [vdl](https://www.paddlepaddle.org.cn/paddle/visualdl/service/app/scalar?id=b9a90b8aba579997a6f088b840a6e96d)|
|[Vnet](https://arxiv.org/abs/1606.04797)|Lung|[COVID-19 CT scans](https://www.kaggle.com/andrewmvd/covid19-ct-scans)|97.0%|[README](configs/lung_coronavirus/README.md)|[model](https://bj.bcebos.com/paddleseg/paddleseg3d/lung_coronavirus/vnet_lung_coronavirus_128_128_128_15k_1e-3/model.pdparams) \| [log](https://bj.bcebos.com/paddleseg/paddleseg3d/lung_coronavirus/vnet_lung_coronavirus_128_128_128_15k_1e-3/train.log) \| [vdl](https://paddlepaddle.org.cn/paddle/visualdl/service/app?id=9db5c1e11ebc82f9a470f01a9114bd3c)|
|[nnUNet](https://www.nature.com/articles/s41592-020-01008-z)|Lung Tumor|[MSD-Lung](http://medicaldecathlon.com/)|67.9%|[README](configs/nnunet/msd_lung/README.md)|[model](https://aistudio.baidu.com/aistudio/datasetdetail/162872)  \| [log](https://aistudio.baidu.com/aistudio/datasetdetail/150774)|
|[Vnet](https://arxiv.org/abs/1606.04797)|Spine|[MRISpineSeg](https://www.spinesegmentation-challenge.com/)|74.7%|[README](configs/mri_spine_seg/README.md)|[model](https://bj.bcebos.com/paddleseg/paddleseg3d/mri_spine_seg/vnet_mri_spine_seg_512_512_12_15k_5e-1/model.pdparams) \| [log](https://bj.bcebos.com/paddleseg/paddleseg3d/mri_spine_seg/vnet_mri_spine_seg_512_512_12_15k_5e-1/train.log) \| [vdl](https://www.paddlepaddle.org.cn/paddle/visualdl/service/app/index?id=08b0f9f62ebb255cdfc93fd6bd8f2c06)|
|[UNETR](https://arxiv.org/abs/2103.10504)|Brain Tumor|[MSD-brain](http://medicaldecathlon.com/)|71.8%|[README](configs/msd_brain_seg/README.md)|[model](https://bj.bcebos.com/paddleseg/paddleseg/medicalseg/msd_brain_seg/unetr_msd_brain_seg_1e-4/model.pdparams) \| [log](https://bj.bcebos.com/paddleseg/paddleseg/medicalseg/msd_brain_seg/unetr_msd_brain_seg_1e-4/train.log) \| [vdl](https://www.paddlepaddle.org.cn/paddle/visualdl/service/app/scalar?id=04e012eef21ea8478bdc03f9c5b1032f)|
|[SwinUNet](https://arxiv.org/abs/2105.05537)|Multi-orgrans|[Synapse](https://www.synapse.org/#!Synapse:syn3193805/files/)|82.1%|[README](configs/synapse/README.md)|[model](https://paddleseg.bj.bcebos.com/paddleseg3d/synapse/abdomen/swinunet_abdomen_224_224_1_14k_5e-2/model.pdparams) \| [log](https://paddleseg.bj.bcebos.com/paddleseg3d/synapse/abdomen/swinunet_abdomen_224_224_1_14k_5e-2/train.log) \| [vdl](https://www.paddlepaddle.org.cn/paddle/visualdl/service/app/scalar?id=f62f69b8e9e9210c680dcfc862e3b65b) |****
|[TransUNet](https://arxiv.org/abs/2102.04306)|Multi-orgrans|[Synapse](https://www.synapse.org/#!Synapse:syn3193805/files/)|81.1%|[README](configs/synapse/README.md)|[model](https://paddleseg.bj.bcebos.com/paddleseg3d/synapse/abdomen/transunet_abdomen_224_224_1_14k_1e-2/model.pdparams) \| [log](https://paddleseg.bj.bcebos.com/paddleseg3d/synapse/abdomen/transunet_abdomen_224_224_1_14k_1e-2/train.log) \| [vdl](https://www.paddlepaddle.org.cn/paddle/visualdl/service/app/scalar?id=d933d970394436aa6969c9c00cf8a6da)|


### 2. Speed
We add GPU acceleration in data preprocess using [CuPy](https://docs.cupy.dev/en/stable/index.html). Compared with preprocess data on CPU, acceleration enable us to use about 40% less time in data prepeocessing. The following shows the time we spend in process COVID-19 CT scans.

<center>

| Device | Time(s) |
|:-:|:-:|
|CPU|50.7|
|GPU|31.4( &#8595; 38%)|

</center>


## <img src="https://user-images.githubusercontent.com/34859558/190043857-bfbdaf8b-d2dc-4fff-81c7-e0aac50851f9.png" width="25"/> QuickStart
This part introduce a easy to use the demo on COVID-19 CT scans dataset. This demo is available on our [Aistudio project](https://aistudio.baidu.com/aistudio/projectdetail/3519594) as well. Detailed steps on training and add your own dataset can refer to this [tutorial](documentation/tutorial.md).
- Download our repository.
    ```
    git clone https://github.com/PaddlePaddle/PaddleSeg.git

    cd contrib/MedicalSeg/
    ```
- Install requirements:
    ```
    pip install -r requirements.txt
    ```
- (Optional) Install CuPY if you want to accelerate the preprocess process. [CuPY installation guide](https://docs.cupy.dev/en/latest/install.html)

- Get and preprocess the data. Remember to replace prepare_lung_coronavirus.py with different python script that you need [here](./tools):
    - change the GPU setting [here](tools/preprocess_globals.yml) to True if you installed CuPY and want to use GPU to accelerate.
    ```
    python tools/prepare_lung_coronavirus.py
    ```

- Run the train and validation example. (Refer to the [tutorial](documentation/tutorial.md) for details.)
   ```
   sh run-vnet.sh
   ```


## <img src="https://user-images.githubusercontent.com/34859558/188439970-18e51958-61bf-4b43-a73c-a9de3eb4fc79.png" width="25"/> FullTutorial
This part shows the details on how to use our repository from setting configurations to deploy.

## 1. Set configuration
Change configuration about loss, optimizer, dataset, and so on here. Our configurations is organized as follows:
```bash
├── _base_                   # base config, set your data path here and make sure you have enough space under this path.
│   └── global_configs.yml
├── lung_coronavirus         # each dataset has one config directory.
│   ├── lung_coronavirus.yml # all the config besides model is here, you can change configs about loss, optimizer, dataset, and so on.
│   ├── README.md  
│   └── vnet_lung_coronavirus_128_128_128_15k.yml    # model related config is here
└── schedulers              # the two stage scheduler, we have not use this part yet
    └── two_stage_coarseseg_fineseg.yml
```


## 2. Prepare the data
We use the data preparation script to download, preprocess, convert, and split the data automatically. If you want to prepare the data as we did, you can run the data prepare file like the following:

```
python tools/prepare_lung_coronavirus.py # take the CONVID-19 CT scans as example.
```

## 3. Train & Validate

After changing your config, you are ready to train your model. A basic training and validation example is [run-vnet.sh](../run-vnet.sh). Let's see some of the training and validation configurations in this file.

```bash
# set your GPU ID here
export CUDA_VISIBLE_DEVICES=0

# set the config file name and save directory here
yml=vnet_lung_coronavirus_128_128_128_15k
save_dir=saved_model/${yml}
mkdir save_dir

# Train the model: see the train.py for detailed explanation on script args
python3 train.py --config configs/lung_coronavirus/${yml}.yml \
--save_dir  $save_dir \
--save_interval 500 --log_iters 100 \
--num_workers 6 --do_eval --use_vdl \
--keep_checkpoint_max 5  --seed 0  >> $save_dir/train.log

# Validate the model: see the val.py for detailed explanation on script args
python3 val.py --config configs/lung_coronavirus/${yml}.yml \
--save_dir  $save_dir/best_model --model_path $save_dir/best_model/model.pdparams

```


## 4. deploy the model

With a trained model, we support deploying it with paddle inference to boost the inference speed. The instruction to do so is as follows, and you can see a detailed python deployment tutorial [here](../../docs/deployment/inference/python_inference.md) and a cpp deployment tutorial [here](../../docs/deployment/inference/cpp_inference.md).

```bash
cd MedicalSeg/

# Export the model with trained parameter
python export.py --config configs/lung_coronavirus/vnet_lung_coronavirus_128_128_128_15k.yml --model_path /path/to/your/trained/model

# Infer it with Paddle Inference Python API
python deploy/python/infer.py \
    --config /path/to/model/deploy.yaml \
    --image_path /path/to/image/path/or/dir/
    --benchmark True   # Use it after installed AutoLog, to record the speed, see ../deploy/python/README.md for detail to install AutoLog.

```
If you see the "finish" output, you have sucessfully upgrade your model's infer speed.

## <img src="https://user-images.githubusercontent.com/34859558/190044556-ad04dc0e-3ec9-41c4-b6a5-a3d251f5cad2.png" width="25"/> TrainOnYourOnData

If you want to train on your dataset, simply add a [dataset file](../medicalseg/datasets/lung_coronavirus.py), a [data preprocess file](../tools/prepare_lung_coronavirus.py), a [configuration directory](../configs/lung_coronavirus), a [training](run-vnet.sh) script and you are good to go. Details on how to add can refer to the links above.

### 1 Add a configuration directory
As we mentioned, every dataset has its own configuration directory. If you want to add a new dataset, you can replicate the lung_coronavirus directory and change relevant names and configs.
```
├── _base_
│   └── global_configs.yml
├── lung_coronavirus
│   ├── lung_coronavirus.yml
│   ├── README.md
│   └── vnet_lung_coronavirus_128_128_128_15k.yml
```

### 2 Add a new data preprocess file
Your data needs to be convert into numpy array and split into trainset and valset as our format. You can refer to the [prepare script](../tools/prepare_lung_coronavirus.py):

```python
├── lung_coronavirus_phase0  # the preprocessed file
│   ├── images
│   │   ├── imagexx.npy
│   │   ├── ...
│   ├── labels
│   │   ├── labelxx.npy
│   │   ├── ...
│   ├── train_list.txt       # put all train data names here, each line contains:  /path/to/img_name_xxx.npy /path/to/label_names_xxx.npy
│   └── val_list.txt         # put all val data names here, each line contains:  img_name_xxx.npy label_names_xxx.npy
```

### 3 Add a dataset file
Our dataset file inherits MedicalDataset base class, where data split is based on the train_list.txt and val_list.txt you generated from previous step. For more details, please refer to the [dataset script](../medicalseg/datasets/lung_coronavirus.py).

### 4 Add a run script
The run script is used to automate a series of process. To add your config file, just replicate the [run-vnet.sh](run-vnet.sh) and change it based on your thought. Here is the content of what they mean:
```bash
# set your GPU ID here
export CUDA_VISIBLE_DEVICES=0

# set the config file name and save directory here
yml=lung_coronavirus/vnet_lung_coronavirus_128_128_128_15k  # relative path to your yml from config dir
config_name = vnet_lung_coronavirus_128_128_128_15k         # name of the config yml
save_dir_all=saved_model                                    # overall save dir
save_dir=saved_model/${config_name}                         # savedir of this exp
```


## <img src="https://user-images.githubusercontent.com/34859558/190046287-31b0467c-1a7e-4bf2-9e5e-40ff3eed94ee.png" width="25"/> Structure
This part shows you the whole picture of our repository, which is easy to expand with different model and datasets. Our file tree is as follows:

```bash
├── configs         # All configuration stays here. If you use our model, you only need to change this and run-vnet.sh.
├── data            # Data stays here.
├── deploy          # deploy related doc and script.
├── medicalseg  
│   ├── core        # the core training, val and test file.
│   ├── datasets  
│   ├── models  
│   ├── transforms  # the online data transforms
│   └── utils       # all kinds of utility files
├── export.py
├── run-vnet.sh     # the script to reproduce our project, including training, validate, infer and deploy
├── tools           # Data preprocess including fetch data, process it and split into training and validation set
├── train.py
├── val.py
└── visualize.ipynb # You can try to visualize the result use this file.
```

## <img src="https://user-images.githubusercontent.com/34859558/190046674-53e22678-7345-4bf1-ac0c-0cc99718b3dd.png" width="25"/> TODO
We have several thoughts in mind about what should our repo focus on. Your contribution will be very much welcomed.
- [✔️] Add PP-nnunet with acceleration in preprocess, automatic configuration for all dataset and better performance compared to nnunet.
- [✔️] Add EISeg-Med3D, a AI-assisted annotation platform.
- [ ] Add pretrain model on various dataset.

## <img src="https://user-images.githubusercontent.com/34859558/188446853-6e32659e-8939-4e65-9282-68909a38edd7.png" width="25"/> License

License of MedicalSeg is [Apache 2.0 license](LICENSE).

##  <img src="https://user-images.githubusercontent.com/34859558/188446803-06c54d50-f2aa-4a53-8e08-db2253df52fd.png" width="25"/> Acknowledgement
- Many thanks to [Lin Han](https://github.com/linhandev), [Lang Du](https://github.com/justld), [onecatcn](https://github.com/onecatcn) for their contribution in  our repository
- Many thanks to [itkwidgets](https://github.com/InsightSoftwareConsortium/itkwidgets) for their powerful visualization toolkit that we used to present our visualizations.
- Many thans to <a href="https://www.flaticon.com/free-icons/idea" title="idea icons"> Idea icons created by Vectors Market - Flaticon</a>  for their beautiful icons.
