Skip to content

GitLab

Switch branch/tag
Find file Normal viewHistoryPermalink
README.md 5.54 KB
Newer Older
xx's avatar
pasd  
xx committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178 
# PASD

## 论文

`Pixel-Aware Stable Diffusion for Realistic Image Super-Resolution and Personalized Stylization`

* https://arxiv.org/pdf/2308.14469

## 模型结构

该模型基于`stable diffusion`结构，并在此基础上添加了`Degradation Removal`，`PACA`以及`ANS`模块。

![alt text](readme_imgs/model.png)

## 算法原理

该算法基于`stable diffusion`，通过增加`PACA`模块使用`cross-attention`的方式将`ControlNet`的特征与`Unet`的特征融合，保留了图像的细节特征。

![alt text](readme_imgs/alg.png)

## 环境配置

### Docker（方法一）
    
    docker pull image.sourcefind.cn:5000/dcu/admin/base/pytorch:2.1.0-ubuntu20.04-dtk24.04.1-py3.10

    docker run --shm-size 50g --network=host --name=pasd --privileged --device=/dev/kfd --device=/dev/dri --group-add video --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v 项目地址(绝对路径):/home/ -v /opt/hyhal:/opt/hyhal:ro -it <your IMAGE ID> bash

    pip install -r requirements.txt

    python setup.py install

    cd BasicSR && python setup.py install


### Dockerfile（方法二）

    docker build -t <IMAGE_NAME>:<TAG> .

    docker run --shm-size 50g --network=host --name=pasd --privileged --device=/dev/kfd --device=/dev/dri --group-add video --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v 项目地址(绝对路径):/home/ -v /opt/hyhal:/opt/hyhal:ro -it <your IMAGE ID> bash

    pip install -r requirements.txt

    python setup.py install

    cd BasicSR && python setup.py install

### Anaconda (方法三)

1、关于本项目DCU显卡所需的特殊深度学习库可从光合开发者社区下载安装：
https://developer.hpccube.com/tool/

    DTK驱动：dtk24.04.1
    python：python3.10
    torch: 2.1.0
    torchvision: 0.16.0

Tips：以上dtk驱动、python、torch等DCU相关工具版本需要严格一一对应

2、其它非特殊库参照requirements.txt安装

    pip install -r requirements.txt

    python setup.py install

    cd BasicSR && python setup.py install

## 数据集

无 

## 训练

无

## 推理

```bash
export HF_ENDPOINT=https://hf-mirror.com
```

### stable diffusion v1-5

### webui
```bash
python gradio_pasd.py
```

### 命令行

```bash
python test_pasd.py --image_path <path or folder> --output_dir <path/to/output_dir>
```

### sdxl

```bash
python test_pasd_sdxl.py --image_path <path/to/image/folder or path> --output_dir <path/to/save/folder>
```

注意：sdxl版本功能较少但性能较好。

## result

|模型|输入|输出|
|:---:|:---:|:---:|
|SD1.5|<img src="readme_imgs/d4f59e89c1011bc4.png">|<img src="readme_imgs/d4f59e89c1011bc4_15.png">|
|SDXL|<img src="readme_imgs/d4f59e89c1011bc4.png">|<img src="readme_imgs/d4f59e89c1011bc4_sdxl.png" style="zoom:50%;">|


### 精度

无

## 应用场景

### 算法类别

`图像超分`

### 热点应用行业

`电商,绘画,交通`

## 预训练权重

```bash
cd pasd/annotator/ckpts && wget https://public-vigen-video.oss-cn-shanghai.aliyuncs.com/robin/models/RetinaFace-R50.pth
```

### stable diffusion v1-5

- base_model: [huggingface](https://hf-mirror.com/benjamin-paine/stable-diffusion-v1-5/tree/main) | [SCNet高速下载通道](http://113.200.138.88:18080/aimodels/stable-diffusion-v1-5)

- pasd: [huggingface](https://hf-mirror.com/yangtao9009/PASD/tree/main/pasd) | [SCNet高速下载通道](http://113.200.138.88:18080/aimodels/findsource-dependency/pasd)
- pasd_light: [原链接](https://public-vigen-video.oss-cn-shanghai.aliyuncs.com/robin/models/PASD/pasd_light.zip) | [SCNet高速下载通道](http://113.200.138.88:18080/aimodels/findsource-dependency/pasd)
- pasd_rrdb: [原链接](https://public-vigen-video.oss-cn-shanghai.aliyuncs.com/robin/models/PASD/pasd_rrdb.zip) | [SCNet高速下载通道](http://113.200.138.88:18080/aimodels/findsource-dependency/pasd)
- pasd_light_rrdb: [原链接](https://public-vigen-video.oss-cn-shanghai.aliyuncs.com/robin/models/PASD/pasd_light_rrdb.zip) | [SCNet高速下载通道](http://113.200.138.88:18080/aimodels/findsource-dependency/pasd)

- dreambooth_lora
    - toonyou_beta6: [huggingface](https://huggingface.co/frankjoshua/toonyou_beta6/tree/main) | [SCNet高速下载通道](http://113.200.138.88:18080/aimodels/frankjoshua/toonyou_beta6)

|模型名称|存储路径|
|:---:|:---:|
|base_model|checkpoints/stable-diffusion-v1-5|
|pasd*|runs/*|
|dreambooth_lora|checkpoints/personalized_models/*.safetensors|

注意：上述模型都需要下载`.safetensors`版本。


### sdxl

- pasd_sdxl: [huggingface](https://hf-mirror.com/yangtao9009/PASD-SDXL/tree/main) | [SCNet高速下载通道](http://113.200.138.88:18080/aimodels/findsource-dependency/pasd)

- stable-diffusion-xl-base-1.0: [huggingface](https://hf-mirror.com/stabilityai/stable-diffusion-xl-base-1.0) | [SCNet高速下载通道](http://113.200.138.88:18080/aimodels/stable-diffusion-xl-base-1.0)

- stable-diffusion-xl-refiner-1.0: [huggingface](https://hf-mirror.com/stabilityai/stable-diffusion-xl-refiner-1.0) | [SCNet高速下载通道](http://113.200.138.88:18080/aimodels/stable-diffusion-xl-refiner-1.0)

- sdxl-vae-fp16-fix: [huggingface](https://hf-mirror.com/madebyollin/sdxl-vae-fp16-fix) | [SCNet高速下载通道](http://113.200.138.88:18080/aimodels/sdxl-vae-fp16-fix)


|模型名称|存储路径|
|:---:|:---:|
|base_model|checkpoints/stable-diffusion-xl-base(refiner)-1.0|
|pasd_sdxl|runs/pasd_sdxl|
|vae|checkpoints/sdxl-vae-fp16-fix|

注意：上述模型都需要下载`.safetensors`版本。

## 源码仓库及问题反馈

* https://developer.hpccube.com/codes/modelzoo/pasd_pytorch

## 参考资料

* https://github.com/yangxy/PASD/tree/main