GitLab

Summary:
# Problem

In CT / State Encoding, we expect a scenario where we would like to render a batch of topologies where each of them would have different number of vertices and triangles. Currently the only way to support this with DRTK is to iterate over the batch in a for loop for each topology and render it.

In a series of diffs we would like to solve this issue by making drtk consume a batch of triangles as opposed to just 1 set of triangles. **However, we would like to achieve this behavior without affecting the most common single topology case by a lot**.

# How do we pass in multiple topologies in a single batch?
- We will provide a `TopologyBatch` structure in xrcia/lib/graphics/structures where we will provide functionality to create a `Batch x MaxTriangles x 3` and `Batch x MaxVertices x 3`.
- Padded vertices will be 0s and padded triangles will have MaxVertices - 1 as their value. But these will discarded as degenerate in rasterization / rendering.

# In this diff
- Extend `rasterize_kernel` and `rasterize_lines_kernel` to support a batch dimension as default.
- `rasterize` will now unsqueeze the batch dimension when using a single topo
- We access the vertex indices of triangles by walking an additional `batch stride * n` in the triangles data pointer.
- Add an extra condition to check to see if the triangles are degenerate; this happens when padding the batch.
- We show that the we don't cause too much overhead in GPU by introducing these 3 extra operations (Same profiling as in D68194200)

Differential Revision: D68388659

fbshipit-source-id: b4f8a7daab8b133b8538f7e5db4f730f70b71deb

Summary:
The purpos of this stack is to enable LUT in rosetta2. To do this:
1. extend project_fisheye_distort_62 to support LUT
2. propagate lut in DRTK rendering functions
3. load the LUT in HMCCalibrationSCIOVRS
4. register LUT as an asset for loading
5. extend the rendering pipeline in rosetta2 with  LUT
6. hook the care cmd string generated in fblearner pipeline with assets defined in 4.

This diff implements 2. I start from the rendering layer defined in drtk and add lut as optional defaults to None.

Reviewed By: patricksnape

Differential Revision: D68207715

fbshipit-source-id: bb9250ec13eaa78bc2615d1845a0b612fbfc12e4

Summary:
The purpos of this stack is to enable LUT in rosetta2. To do this:
1. extend project_fisheye_distort_62 to support LUT
2. propagate lut in DRTK rendering functions
3. load the LUT in HMCCalibrationSCIOVRS
4. register LUT as an asset for loading
5. extend the rendering pipeline in rosetta2 with  LUT
6. hook the care cmd string generated in fblearner pipeline with assets defined in 4.

This diff implements 1. The look up table contains two parts:
1. lut_vector_field: N x 2 X H_lut x W_lut stores the offset for each camera, e.g. 2 x 41 x 41 for FETA camera
2. lut_spacing: N x 2 stores the spacing of each grid in lut_vector_field, e.g. 10.0 for FETA 400 x 400 camera. Usually spacing is the same for both with and height directions in calibration file but might be different if we resize the field.

The offset for each pixel is found by grid sampling

Reviewed By: patricksnape

Differential Revision: D62097271

fbshipit-source-id: 1655cb1dfc8d5e34bc97c4bab16b07a9ee2cf636

Summary:
X-link: https://github.com/facebookexternal/DRTK/pull/2

Pull Request resolved: https://github.com/facebookresearch/DRTK/pull/13

X-link: https://github.com/pytorch/pytorch/pull/138626

Was it a typo? Since we already have `at::detail` in `ATen/Dispatch.h`

Deleting `::detail` all together, we don't need the functions in it anymore.

Reviewed By: malfet

Differential Revision: D64642080

fbshipit-source-id: 9ae48a888a854ec85e1caef53d87e6e3cc81591b

Reviewed By: kit1980, kiran1228

Differential Revision: D67462406

fbshipit-source-id: 6371719adba0c5a7690530b997874a2ce1e045fa

Summary:
Converts the directory specified to use the Ruff formatter in pyfmt

ruff_dog

If this diff causes merge conflicts when rebasing, please run
`hg status -n -0 --change . -I '**/*.{py,pyi}' | xargs -0 arc pyfmt`
on your diff, and amend any changes before rebasing onto latest.
That should help reduce or eliminate any merge conflicts.

allow-large-files

Reviewed By: tpolasek

Differential Revision: D66375521

fbshipit-source-id: 0f4ae27d79e8ae43ee54f319e451fb9ca88da58a

Summary: Added teaser video

Reviewed By: una-dinosauria

Differential Revision: D63808117

fbshipit-source-id: 5302c43f2ddffd5b573a44b446a4aa40f4b113b5

Summary:
Added two more tutorials. For mesh rendering and for hand fitting

Also fixed missing copyright in css file

Reviewed By: una-dinosauria

Differential Revision: D63805824

fbshipit-source-id: e232299b1b4572adc9573d441e1d9b34414ac8e3

Summary: Added another tutorial

Reviewed By: una-dinosauria

Differential Revision: D63753117

fbshipit-source-id: d1fc0f4ca85b97da341b22c3bda71adb787a7140

Summary:
Fixing Automated checkup

https://www.internalfb.com/intern/opensource/github/repo/978273877280726/checkup/

Reviewed By: una-dinosauria

Differential Revision: D63753101

fbshipit-source-id: 550464583a0a81c79a87c351de0e0aae2041db5f

Summary: Added sphinx docs and github workflow to build gh-pages

Reviewed By: HapeMask

Differential Revision: D63498319

fbshipit-source-id: 401aabb6dc1624a26c2c9231ab2a1e6b98458190

Summary:
This unblocks building DRTK in a Docker build since BuildKit doesn't yet support GPUs: https://github.com/moby/buildkit/issues/1436

This works because CUDA is needed at run-time but not at build-time. When CUDA is present at build-time, the currently installed cards set which archs the extensions are built for. However, the archs are manually fixed in setup.py, so this commit also adds support for respecting TORCH_CUDA_ARCH_LIST which controls archs within torch.utils.cpp_extension.CUDAExtension

Pull Request resolved: https://github.com/facebookresearch/DRTK/pull/5

Test Plan:
Build without TORCH_CUDA_ARCH_LIST and observe using `ps` that compilation gets the flags specified in setup.py.

Build with TORCH_CUDA_ARCH_LIST=Turing and TORCH_CUDA_ARCH_LIST=8.6 and observe using `ps` that compilation gets only the flags for the requested archs.

Build within a Dockerfile using `docker-compose up --build`, which is using BuildKit, and observe that compilation succeeds and the library is usable when later run with CUDA GPUs available.

Reviewed By: HapeMask

Differential Revision: D63513797

Pulled By: podgorskiy

fbshipit-source-id: 54ce6765ccc37317999f18690a73823eb074f08f

Summary: This is necessary if we want to run building docs in github workflow

Differential Revision: D63497765

fbshipit-source-id: 018157c205a66584dd040882124588e499439893

Summary: As title says. The is for the sphinx documentation.

Reviewed By: HapeMask

Differential Revision: D63440496

fbshipit-source-id: 483fdfc6cbc14ce8f88e6d048553488f1a0f8ed3

Summary:
Sequence is not TorchScript compatible, replacing with a List.

Also the original annotation `Optional[Sequence[str]]` was not correct, as implementation was actially allowing usage of `str` type.

Wrapped `project_points` with decorator `torch.jit.ignore` as it violates TorchScript too much (uses sets, changes types of variables, uses None with other types aggressively)

Differential Revision: D63444533

fbshipit-source-id: 46604cbd239ed8b9051ad371779fb81106987dc5

Summary:
Make `rasterize_with_depth` and `transform_with_v_cam`  accessible from the root drtk module as:

```
from drtk import rasterize_with_depth, transform_with_v_cam
```

instead of

```
from drtk.rasterize import rasterize_with_depth
from drtk.transform import transform_with_v_cam
```

This has greater importance for DGX and GitHub rather than prod, since because of how buck is setup, it still won't work.

Reviewed By: una-dinosauria

Differential Revision: D63440390

fbshipit-source-id: 7f6c8c16fb8ef5aa10bef3491fd59727f1000d0b

Differential Revision: D63440224

fbshipit-source-id: 0a3647fd710f2d980b22f1954e439fa27a1b4864

Summary: Got legal approval 🥳

Reviewed By: una-dinosauria

Differential Revision: D63428775

fbshipit-source-id: 7568ef2861ef10c2bd0367a7195cbbedf96ec8be

The internal and external repositories are out of sync. This Pull Request attempts to brings them back in sync by patching the GitHub repository. Please carefully review this patch. You must disable ShipIt for your project in order to merge this pull request. DO NOT IMPORT this pull request. Instead, merge it directly on GitHub using the MERGE BUTTON. Re-enable ShipIt after merging.

Summary: This isn't the real fix it's just moving the files into a folder we can exclude in shipit. See D62289505 where we exclude this directory from shipit. This project is exported to open source (Github) so we need to not ship these files -- I didn't realize this when I wrote the tests.

Reviewed By: jermenkoo

Differential Revision: D62289557

fbshipit-source-id: 60afb00fd8a6aa54aabe12a8a3fcc161a65b0495

Summary:
Currently on master we have a bug in that we undistort images into a pinhole frame but then optimize in Rosetta using fisheye. Undistorting to pinhole is totally fine but then we should be optimizing in pinhole in Rosetta.

So this is the "other" version of the fix I did in D58979100. In D58979100 I "fixed" the cameras by defining the undistort method to be fisheye rather than pinhole. In this diff I did the other way round - I've left the CameraOperator alone and I've fixed the distortion model we propagate after undistortion to be pinhole.

Reviewed By: AlexRadionovMeta

Differential Revision: D60927099

fbshipit-source-id: 316aa0d99396c49d61c31d743112f1aa0eef85cb

Summary:
I wrote the unit test attached to triple check we were matching the perception cameras - but we were not. So I ported the code exactly and now we exactly match the perception cameras behavior.

Looking at the code it seems to be a legit bug - and when I read the code my mental model says this may make a significant difference if the radial distortion parameters are high (which they are in BTL). However - Rosetta continues to be an enigma to me and it seems to hardly make a difference...

Differential Revision: D60927097

fbshipit-source-id: 15e04b50634d8c236bfaf89fcf6f43aeff1ede7d

Summary:
Adds `grid_scatter` op that is similar to `grid_sample` but the grid points to the destination location instead of the source.

`grid_scatter` is indeed dual to `grid_sample`. Forward of `grid_scatter` is backward of `grid_sample` and backward of  `grid_scatter` is forward of `grid_sample` (with the exception for the gradient with respect to grid) which is reflected in the reference implementation in `drtk/grid_scatter.py`.

```python
def grid_scatter(
    input: th.Tensor,
    grid: th.Tensor,
    output_height: int,
    output_width: int,
    mode: str = "bilinear",
    padding_mode: str = "border",
    align_corners: Optional[bool] = None,
) -> th.Tensor:
```

Where :
* `input` [N x C x H x W]: is the input tensor values from which will be transferred to the result.
* `grid` [N x H x W x 2]: is the grid tensor that points to the location where the values from the  input tensor should be copied to. The `W`, `H` sizes of grid should match the corresponding sizes of the `input` tensor.
*  `output_height`, `output_width`: size of the output, where output will be: [N x C x `output_height` x `output_width`]. In contrast to `grid_sample`, we can no longer rely on the sizes of the `grid` for this information.
* `mode`, `padding_mode`, `align_corners` same as for the `grid_sample`, but now for the reverse operation - splatting (or scattering).

At the moment does not support "nearest" mode, which is rarely needed. Maybe will add later.

Ideally, we would also want to support autocast mode where the `input` and output tensors are float16 while the `grid` is float32. This is not the case at the moment, but I'll add that later.

## Example usage

Let's assume that we loaded mesh into `v, vi, vt, vti`, have defined `image_width, image_height`, `cam_pos`, `cam_rot`, `focal`, `princpt`, and computed normals for the mesh `normals`. We also define a shading function, e.g.:

```lang=python
def shade(
    vn_img: th.Tensor,
    light_dir: th.Tensor,
    ambient_intensity: float = 1.0,
    direct_intensity: float = 1.0,
    shadow_img: Optional[th.Tensor] = None,
):
    ambient = (vn_img[:, 1:2] * 0.5 + 0.5) * th.as_tensor([0.45, 0.5, 0.7]).cuda()[
        None, :, None, None
    ]
    direct = (
        th.sum(vn_img.mul(thf.normalize(light_dir, dim=1)), dim=1, keepdim=True).clamp(
            min=0.0
        )
        * th.as_tensor([0.65, 0.6, 0.5]).cuda()[None, :, None, None]
    )
    if shadow_img is not None:
        direct = direct * shadow_img
    return th.pow(ambient * ambient_intensity + direct * direct_intensity, 1 / 2.2)
```

And we can render the image as:

```lang=python
v_pix = transform(v, cam_pos, cam_rot, focal, princpt)
index_img = rasterize(v_pix, vi, image_height, image_width)
_, bary_img = render(v_pix, vi, index_img)

# mask image
mask: th.Tensor = (index_img != -1)[:, None]

# compute vt image
vt_img = interpolate(vt.mul(2.0).sub(1.0)[None], vti, index_img, bary_img)

# compute normals
vn_img = interpolate(normals, vi, index_img, bary_img)

diffuse = (
    shade(vn_img, th.as_tensor([0.5, 0.5, 0.0]).cuda()[None, :, None, None]) * mask
)
```

 {F1801805545}

## Shadow mapping

We can use  `grid_scatter` to compute mesh visibility from the camera view:

```lang=python
texel_weight = grid_scatter(
    mask.float(),
    vt_img.permute(0, 2, 3, 1),
    output_width=512,
    output_height=512,
    mode="bilinear",
    padding_mode="border",
    align_corners=False,
)
threshold = 0.1  # texel_weight is proportional to how much pixel are the texel covers. We can specify a threshold of how much covered pixel area counts as visible.
visibility = (texel_weight > threshold).float()
```
 {F1801810094}

Now we can render the scene from different angle and use the visibility mask for shadows:

```lang=python
v_pix = transform(v, cam_pos_new, cam_rot_new, focal, princpt)
index_img = rasterize(v_pix, vi, image_height, image_width)
_, bary_img = render(v_pix, vi, index_img)

# mask image
mask: th.Tensor = (index_img != -1)[:, None]

# compute vt image
vt_img = interpolate(vt.mul(2.0).sub(1.0)[None], vti, index_img, bary_img)

# compute v image (for near-field)
v_img = interpolate(v, vi, index_img, bary_img)

# shadow
shadow_img = thf.grid_sample(visibility, vt_img.permute(0, 2, 3, 1), mode="bilinear", padding_mode="border", align_corners=False)

# compute normals
vn_img = interpolate(normals, vi, index_img, bary_img)

diffuse = shade(vn_img, cam_pos[:, :, None, None] - v_img, 0.05, 0.4, shadow_img) * mask
```
 {F1801811232}

## Texture projection

Let's load a test image:

```lang=python
import skimage
test_image = (
    th.as_tensor(skimage.data.coffee(), dtype=th.float32).permute(2, 0, 1)[None, ...].mul(1 / 255).contiguous().cuda()
)

test_image = thf.interpolate(test_image, scale_factor=2.0, mode="bilinear", align_corners=False)
```

{F1801814094}

We can use `grid_scatter` to project the image onto the uv space:

```lang=python
camera_image_extended = (
    th.cat([test_image, th.ones_like(test_image[:, :1])], dim=1) * mask
)

texture_weight = grid_scatter(
    camera_image_extended,
    vt_img.permute(0, 2, 3, 1),
    output_width=512,
    output_height=512,
    mode="bilinear",
    padding_mode="border",
    align_corners=False,
)

texture = texture_weight[:, :3] / texture_weight[:, 3:4].clamp(min=1e-4)
```

{F1801816367}

And if we render the scene from a different angle using the projected texture:

 {F1801817130}

Reviewed By: HapeMask

Differential Revision: D61006613

fbshipit-source-id: 98c83ba4eda531e9d73cb9e533176286dc699f63

Summary:
For unused warps we always write zeros to `bary_img_grad`.
However it is possible that the warp is used, but only portion of the threads are used. In this case, for unused threads we do not write zeros to `bary_img_grad`.

For efficiency, `bary_img_grad` is created with `at::empty`, thus  those before mentioned entries, will still have uninitialized values.

This is not an issue, because the `render` function will never read those entries, however it is possible that the uninitialized values will coincide with `nan` and it will trigger a false positive detection in auto grad anomaly detection.  Please see more details in D60904848 about the issue.

Differential Revision: D60912474

fbshipit-source-id: 6eda5a07789db456c17eb60de222dd4c7b1c53d2

Summary:
Apply fisheye62 distortion to HQLP camera renders.

Visualization of results. 4 rows are in order: HMC, drtk render, pixel render, pinhole img.

https://www.internalfb.com/manifold/explorer/codec-avatars-scratch/tree/hmc_nvs/20240516_rosetta_nearfield/runs/TEST_run_hqlp_rosetta_render_ojwang_v0719_002/dome_hmc_nvs_render/MLU130_20220509--0000_codec/vrs_comparison/000002__MLU130-brows_lowered-2022-05-09-16-08-12.png

Debug distortion document, tracked the work: https://docs.google.com/document/d/1nSnFMrGWXVoICDEstpL9JZAnRtiuSMnWjnfGDzIfrNE/edit?pli=1

Reviewed By: euclid1767

Differential Revision: D58825272

fbshipit-source-id: 463f8c60014bd9cde898d72875bec49ffc75e453

Summary: In order to work properly with `torch.compile()`, we need to decorate any function that calls a custom C++/CUDA extension with `torch.compiler.disable` so that it knows to insert a graph break.

Reviewed By: podgorskiy

Differential Revision: D59776177

fbshipit-source-id: d80eb43858836f8b8647d2a35b30d0b863989e94

Summary: Created from CodeHub with https://fburl.com/edit-in-codehub

Reviewed By: una-dinosauria

Differential Revision: D58369133

fbshipit-source-id: cf6a0e4710be579f60f6d569d23c50183dfd6018

Summary: Created from CodeHub with https://fburl.com/edit-in-codehub

Reviewed By: tsimk

Differential Revision: D58368652

fbshipit-source-id: 8f3f426f0825255e5e6d6b32b8f1bdb2cde6e161

fbshipit-source-id: afc575e8e7d8e2796a3f77d8b1c6c4fcb999558d