README.md

# 数据集

coco2017

```
https://cocodataset.org/#home
http://images.cocodataset.org/zips/train2017.zip # train dataset
http://images.cocodataset.org/zips/val2017.zip # validation dataset
http://images.cocodataset.org/zips/test2017.zip # test dataset
http://images.cocodataset.org/zips/unlabeled2017.zip
http://images.cocodataset.org/annotations/annotations_trainval2017.zip
http://images.cocodataset.org/annotations/stuff_annotations_trainval2017.zip
http://images.cocodataset.org/annotations/image_info_test2017.zip
http://images.cocodataset.org/annotations/image_info_unlabeled2017.zip
```

宝可梦
```
https://huggingface.co/datasets/lambdalabs/pokemon-blip-captions

```


## 数据集处理

deal_coco.py 
```
#!/usr/bin/env python
import argparse
import json
import os
from typing import Iterator, Tuple


def read_coco_annotations(path: str, image_root: str) -> Iterator[Tuple[str, str]]:
    with open(path, "r") as f:
        content = json.load(f)["annotations"]
        for record in content:
            image_id = record["image_id"]
            caption = record["caption"]
            image_name = f"{image_id:012d}.jpg"
            image_path = os.path.join(image_root, image_name)
            if not os.path.isfile(image_path):
                print(f"Cannot find `{image_path}`, skip.")
                continue
            caption = caption.strip().replace(",", "").replace("\n", "")
            yield image_name, caption


def main():
    parser = argparse.ArgumentParser(description="Converting COCO json annatotion into plain txt")
    parser.add_argument("--label", required=True, help="Path of the label file")
    parser.add_argument("--image", required=True, help="Path of the image root")
    parser.add_argument("--out", default="metadata.csv", help="Output path of the txt file")
    args = parser.parse_args()

    with open(args.out, "w") as f:
        f.write("file_name,text\n")
        for image_name, caption in read_coco_annotations(args.label, args.image):
            f.write(f"{image_name},{caption}\n")


if __name__ == "__main__":
    main()
```

处理数据
```
python  deal_coco.py  --lablel  captions_train2017.json   --images train2017/
```





# 模型
```
https://github.com/huggingface/diffusers.git
```




# 环境搭建

```
#将requiments.txt中torchvision注释
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
python setup.py install 





```


# 训练


lora 训练
```
export MODEL_NAME="/datasets/custom_model/stable-diffusion-xl-base-1.0"                                                                                                                                                                                                                                                                                
 
export OUTPUT_DIR="/path/to/sd_xl"                                                                                                                                                                                                                                                                                                                   
#export DATASET_NAME="/datasets/custom_datasets/pokemon-blip-captions"                                                                                                                                                                                                                                                                                 
                                                                                                                                                             
export DATASET_NAME="/datasets/custom_datasets/coco2017/images/train2017"   
export VAE_NAME="/datasets/custom_model/sdxl-vae-fp16-fix"
accelerate launch --multi_gpu  examples/text_to_image/train_text_to_image_sdxl.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --pretrained_vae_model_name_or_path=$VAE_NAME \
  --dataset_name=$DATASET_NAME \
  --enable_xformers_memory_efficient_attention \
  --resolution=512 --center_crop --random_flip \
  --proportion_empty_prompts=0.2 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 --gradient_checkpointing \
  --max_train_steps=10000 \
  --use_8bit_adam \
  --learning_rate=1e-06 --lr_scheduler="constant" --lr_warmup_steps=0 \
  --mixed_precision="fp16" \
  --validation_prompt="a cute Sundar Pichai creature" --validation_epochs 5 \
  --checkpointing_steps=5000 \
  --output_dir=$OUTPUT_DIR

```

# 推理



# 参考
 https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/README_sdxl.md