# Nunchaku Tests
Nunchaku uses pytest as its testing framework.

## Setting Up Test Environments
After installing `nunchaku` as described in the [README](../README.md#installation), you can install the test dependencies with:
```shell
pip install -r tests/requirements.txt
```

## Running the Tests
```shell
HF_TOKEN=$YOUR_HF_TOKEN pytest -v tests/flux/test_flux_memory.py
HF_TOKEN=$YOUR_HF_TOKEN pytest -v tests/flux --ignore=tests/flux/test_flux_memory.py
HF_TOKEN=$YOUR_HF_TOKEN pytest -v tests/sana
```

> **Note:** `$YOUR_HF_TOKEN` refers to your Hugging Face access token, required to download models and datasets. You can create one at [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens).
>  If you've already logged in using `huggingface-cli login`, you can skip setting this environment variable.

Some tests generate images using the original 16-bit models. You can cache these results to speed up future test runs by setting the environment variable `NUNCHAKU_TEST_CACHE_ROOT`. If not set, the images will be saved in `test_results/ref`.

## Writing Tests

When adding a new feature, please include corresponding test cases in the [`tests`](./) directory. **Please avoid modifying existing tests.**

To test visual output correctness, you can:

1. **Generate reference images:** Use the original 16-bit model to produce a small number of reference images (e.g., 4).

2. **Generate comparison images:** Run your method using the **same inputs and seeds** to ensure deterministic outputs. You can control the seed by setting the `generator` parameter in the diffusers pipeline.

3. **Compute similarity:** Evaluate the similarity between your outputs and the reference images using the [LPIPS](https://arxiv.org/abs/1801.03924) metric. Use the `compute_lpips` function provided in [`tests/flux/utils.py`](flux/utils.py):

   ```shell
   lpips = compute_lpips(dir1, dir2)
   ```

   Here, `dir1` should point to the directory containing the reference images, and `dir2` should contain the images generated by your method. 

### Setting the LPIPS Threshold

To pass the test, the LPIPS score must be below a predefined threshold—typically **< 0.3**. We recommend first running the comparison locally to observe the LPIPS value, and then setting the threshold slightly above that value to allow for minor variations. Since the test is based on a small sample of images, slight fluctuations are expected; a margin of **+0.04** is generally sufficient.

## Acknowledgments

This contribution guide is adapted from [SGLang](https://github.com/sgl-project/sglang/tree/main/test). We thank them for the inspiration.