MakeDataset.md

# Use the IndexKits library to create datasets.

[TOC]

## Introdution

The IndexKits library offers a command-line tool `idk` for creating datasets and viewing their statistical information. You can view the instructions by using `idk -h`. Here, it refers to creating an Index V2 format dataset from a series of arrow files.

## 1. Creating a Base Index V2 Dataset

### 1.1 Create a Base Index V2 dataset using `idk`

When creating a Base Index V2 dataset, you need to specify the path to a configuration file using the `-c` parameter and a save path using the `-t` parameter.

```shell
idk base -c base_config.yaml -t base_dataset.json
```

### 1.2 Basic Configuration

Next, let’s discuss how to write the configuration file. The configuration file is in `yaml` format, and below is a basic example:

Filename: `base_config.yaml`

```yaml
source:
    - /HunYuanDiT/dataset/porcelain/arrows/00000.arrow
```

|    Field Name    | Type  |     Description     |
|:---------:|:---:|:----------:|
|  source   | Optional  |  Arrow List  |


We provide an example that includes all features and fields in [full_config.yaml](./docs/full_config.yaml).

### 1.3 Filtering Criteria

`idk` offers two types of filtering capabilities during the dataset creation process: (1) based on columns in Arrow, and (2) based on MD5 files. 

To enable filtering criteria, add the `filter` field in the configuration file.

#### 1.3.1 Column Filtering

To enable column filtering, add the `column` field under the `filter` section. 
Multiple column filtering criteria can be applied simultaneously, with the intersection of multiple conditions being taken.

For example, to select data where both the length and width are greater than or equal to 512, 
with the default being 1024 if the length and width are invalid:

```yaml
filter:
    column:
        -   name: height
            type: int
            action: ge
            target: 512
            default: 1024
        -   name: width
            type: int
            action: ge
            target: 512
            default: 1024
```

This filtering condition is equivalent to `table['height'].to_int(default=1024) >= 512 && table['width'].to_int(default=1024) >= 512`.

Each filtering criterion includes the following fields:

|      Field Name    | Type  |   Description     |          Value Range          |
|:------------------:|:---:|:---------------:|:----------------------:|
|        name        | Required  |   Column name in Arrow    |       Column in Arrow       |
|        type        | Required  |    Type of Elements in the Column     | `int`, `float` or `str` |
|       action       | Required  |     Filtering Criteria     |    See the table below for possible values      |
|       target       | Required  |    Filtering Criteria    |       Numeric or String       |
|      default       | Required  | Default value when the element is invalid  |       Numeric or String       |
| arrow_file_keyword | Optional  | Keywords in the Arrow file path |           -            |

Below are the specific meanings of “action” and the optional values under different circumstances::

|    Action     |                     Description                      |         Type          |    Action    |                 Description                  |         Type          |
|:-------------:|:----------------------------------------------------:|:---------------------:|:------------:|:--------------------------------------------:|:---------------------:|
|      eq       |                     equal, `==`                      | `int`, `float`, `str` |      ne      |               not equal, `!=`                | `int`, `float`, `str` |
|      gt       |                   great than, `>`                    |    `int`, `float`     |      lt      |                less than, `<`                |    `int`, `float`     |
|      ge       |                 great or equal, `>=`                 |    `int`, `float`     |      le      |             less or equal, `<=`              |    `int`, `float`     |
|    len_eq     |           str length equal, `str.len()==`            |         `str`         |    len_ne    |     str length not equal, `str.len()!=`      |         `str`         |
|    len_gt     |         str length great than, `str.len()>`          |         `str`         |    len_lt    |      str length less than, `str.len()<`      |         `str`         |
|    len_ge     |       str length great or equal, `str.len()>=`       |         `str`         |    len_le    |   str length less or equal, `str.len()<=`    |         `str`         |
|   contains    |         str contains, `str.contains(target)`         |         `str`         | not_contains | str not contains, `str.not_contains(target)` |         `str`         |
|      in       |              str in,  `str.in(target)`               |         `str`         |    not_in    |       str not in, `str.not_in(target)`       |         `str`         |
| lower_last_in | lower str last char in, `str.lower()[-1].in(target)` |         `str`         |              |                                              |                       |

#### 1.3.2 MD5 Filtering

Add an `md5` field under the `filter` section to initiate MD5 filtering. Multiple MD5 filtering criteria can be applied simultaneously, with the intersection of multiple conditions being taken. 

例如:
* `badcase.txt` is a list of MD5s, aiming to filter out the entries listed in these lists.
* `badcase.json` is a dictionary, where the key is the MD5 and the value is a text-related `tag`. The goal is to filter out specific `tags`.

```shell
filter:
    md5:
        - name: badcase1
          path:
            - badcase1.txt
          type: list
          action: in
          is_valid: false
        - name: badcase2
          path: badcase2.json
          type: dict
          action: eq
          target: 'Specified tag'
          is_valid: false
```

Each filtering criterion includes five fields:

|        Field Name         | Type  |                                Description                                |                                        Value Range                                        |
|:------------------:|:---:|:-----------------------------------------------------------------:|:------------------------------------------------------------------------------------:|
|        name        | Required  |     The name of the filtering criterion, which can be customized for ease of statistics.                       |                                          -                                           |
|        path        | Required  | The path to the filtering file, which can be a single path or multiple paths provided in a list format. Supports `.txt`, `.json`, `.pkl` formats. |                                          -                                           |
|        type        | Required  |                         The type of records in the filtering file.                       |                                   `list` or `dict`                               | 
|       action       | Required  |                           The filtering action.                             | For `list`: `in`, `not_in`; For `dict`: `eq`, `ne`, `gt`, `lt`, `ge`, `le`|
|       target       | Optional  |                         The filtering criterion.                              |                               Required when type is `dict`                               |
|      is_valid      | Required  |                 Whether a hit on action+target is considered valid or invalid.                  |                                  `true` or `false`                           |
| arrow_file_keyword | Optional  |                       Keywords in the Arrow file path.                       |                                          -                                           |

### 1.4 Advanced Filtering

`idk` also supports some more advanced filtering functions.

#### 1.4.1  Filtering Criteria Applied to Part of Arrow Files

Using the `arrow_file_keyword` parameter, filtering criteria can be applied only to part of the Arrow files. 

For example:

* The filtering criterion `height>=0` applies only to arrows whose path includes `human`.
* The filtering criterion “keep samples in goodcase.txt” applies only to arrows whose path includes `human`.

```yaml
filter:
    column:
        - name: height
          type: int
          action: ge
          target: 512
          default: 1024
          arrow_file_keyword:
              - human

    md5:
        - name: goodcase
          path: goodcase.txt
          type: list
          action: in
          is_valid: true
          arrow_file_keyword:
            - human
```

#### 1.4.2 The “or” Logic in Filtering Criteria

By default, filtering criteria follow an “and” logic. If you want two or more filtering criteria to follow an “or” logic, you should use the `logical_or` field. The column filtering criteria listed under this field will be combined using an “or” logic.

```yaml
filter:  
    column:  
        - logical_or:  
            -   name: md5  
                type: str  
                action: lower_last_in  
                target: '02468ace'  
                default: ''  
            -   name: text_zh  
                type: str  
                action: contains  
                target: 'turtle|rabbit|duck|swan|peacock|hedgehog|wolf|fox|seal|crow|deer'  
                default: ''
```

Special Note: The `logical_or` field is applicable only to the `column` filtering criteria within `filter`.

1.4.3 Excluding Certain Arrows from the Source

While wildcards can be used to fetch multiple arrows at once, there might be instances where we want to exclude some of them. This can be achieved through the `exclude` field. Keywords listed under `exclude`, if found in the path of the current group of arrows, will result in those arrows being excluded.

```yaml
source:
    - /HunYuanDiT/dataset/porcelain/arrows/*.arrow:
        exclude:
            - arrow1
            - arrow2
```

### 1.5 Repeating Samples  
  
`idk` offers the capability to repeat either all samples or specific samples during dataset creation. There are three types of repeaters:  
* Directly repeating the source  
* Based on keywords in the Arrow file name (enable repeat conditions by adding the `repeater` field in the configuration file)  
* Based on an MD5 file (enable repeat conditions by adding the `repeater` field in the configuration file)  
  
**Special Note:** The above three conditions can be used simultaneously. If a sample meets multiple repeat conditions, the highest number of repeats will be taken.  
  
#### 1.5.1 Repeating the Source  
  
In the source, for the Arrow(s) you want to repeat (which can include wildcards like `*`, `?`), add `repeat: n` to mark the number of times to repeat.  
  
```yaml  
source:  
    - /HunYuanDiT/dataset/porcelain/arrows/*.arrow:  
        repeat: 10  
```

**Special Note: Add a colon at the end of the Arrow path**

#### 1.5.2 Arrow Keywords

Add an `arrow_file_keyword` field under the `repeater` section.

```yaml
repeater:  
    arrow_file_keyword:  
        - repeat: 8  
          keyword:  
            - Lolita anime style  
            - Minimalist style  
        - repeat: 5  
          keyword:  
              - Magical Barbie style  
              - Disney style  
```

Each repeat condition includes two fields:


|   Field Name	   | Type  |       Description        | Value Range |
|:-------:|:---:|:---------------:|:----:|
| repeat  | Required  |  The number of times to repeat  |  Number  |
| keyword | Required  | Keywords in the Arrow file path |  -    |


#### 1.5.3 MD5 File

Add an `md5` field under the `repeater` section.

```yaml
repeater:
    md5:
        - name: goodcase1
          path: /HunYuanDiT/dataset/porcelain/md5_repeat_1.json
          type: dict
          plus: 3
        - name: goodcase2
          path: /HunYuanDiT/dataset/porcelain/md5_repeat_2.json
          type: list
          repeat: 6
```

Each repeat condition includes the following fields:


|  Field Name   | Type  |                       Description	                     |            Value Range               |
|:------:|:---:|:----------------------------------------------:|:---------------------------:|
|  name  | Required  |               Custom name for the repeat condition                |              -              |
|  path  | Required  |                 Path to the file containing MD5s               | `.txt`, `.json`, or `.pkl` formats |
|  type  | Required  |                  Type of the MD5 file                 |       `list` or `dict`       |
| repeat | Optional  |     Number of times to repeat, this will override plus     |             Integer              |
|  plus  | Optional  | Adds this value to the value obtained from the MD5 file as the number of repeats |             Integer              |

### 1.6 Deduplication

To deduplicate, add `remove_md5_dup` and set it to `true`. For example:

```yaml
remove_md5_dup: true
```

**Special Note:** Deduplication is performed after repeat conditions, so using them together will make repeat ineffective.

### 1.7 Creating a Base Index V2 Dataset with Python Code

```python
from index_kits import IndexV2Builder

builder = IndexV2Builder(arrow_files)
builder.save('data_v2.json')
```



## 2. Creating a Multireso Index V2 Dataset

### 2.1 Creating a Multireso Index V2 Dataset with `idk`
To create a multi-resolution dataset, you can do so through a configuration file:

```shell
src:
    - /HunYuanDiT/dataset/porcelain/jsons/a.json
    - /HunYuanDiT/dataset/porcelain/jsons/b.json
    - /HunYuanDiT/dataset/porcelain/jsons/c.json
base_size: 512
reso_step: 32
min_size: 512
```

The fields are as follows:

|      Field Name      | Type  |                           Description                           |              Value Range               |
|:-------------:|:---:|:------------------------------------------------------:|:-------------------------------:|
|      src      | Required  |       Path(s) to the Base Index V2 file, can be single or multiple           |                -                |
|   base_size   | Required  |              The base resolution (n, n) from which to create multiple resolutions               |        Recommended values: 256/512/1024      |
|   reso_step   | Optional  | The step size for traversing multiple resolutions. Choose either this or target_ratios	 |          Recommended values: 16/32/64         |
| target_ratios | Optional  | Target aspect ratios, a list. Choose either this or reso_step	 | Recommended values: 1:1, 4:3, 3:4, 16:9, 9:16  |
|     align     | Optional  | When using target_ratios, the multiple to align the target resolution to	 | Recommended value: 16 (2x patchify, 8x VAE) |
|   min_size    | Optional  | The minimum resolution filter for samples when creating from Base to Multireso Index V2	 |       Recommended values: 256/512/1024      |
|   md5_file    | Optional  | A pre-calculated dictionary of image sizes in pkl format, key is MD5, value is (h, w)	 |                -                |


### 2.2 Creating a Multireso Index V2 Dataset with Python Code

First, import the necessary function

```python
from index_kits import build_multi_resolution_bucket  
  
md5_hw = None  
# If many arrows in your Index V2 lack height and width, you can pre-calculate them and pass through the md5_hw parameter.  
# md5_hw = {'c67be1d8f30fd0edcff6ac99b703879f': (720, 1280), ...}  
# with open('md5_hw.pkl', 'rb') as f:  
#     md5_hw = pickle.load(f)  
  
index_v2_file_path = 'data_v2.json'  
index_v2_save_path = 'data_multireso.json'  
  
# Method 1: Given base_size and reso_step, automatically calculate all resolution buckets.  
build_multi_resolution_bucket(base_size=1024,  
                              reso_step=64,  
                              min_size=1024,  
                              src_index_files=index_v2_file_path,   
                              save_file=index_v2_save_path,   
                              md5_hw=md5_hw)  
  
# Method 2: Given a series of target aspect ratios, automatically calculate all resolution buckets.  
build_multi_resolution_bucket(base_size=1024,  
                              target_ratios=["1:1", "3:4", "4:3", "16:9", "9:16"],  
                              align=16,  
                              min_size=1024,  
                              src_index_files=index_v2_file_path,   
                              save_file=index_v2_save_path,   
                              md5_hw=md5_hw)  
```

**Note：** If both reso_step and target_ratios are provided, target_ratios will be prioritized.