GitLab

Summary: Implement the sample_pdf function from the NeRF project as compiled operators.. The binary search (in searchsorted) is replaced with a low tech linear search, but this is not a problem for the envisaged numbers of bins.

Reviewed By: gkioxari

Differential Revision: D26312535

fbshipit-source-id: df1c3119cd63d944380ed1b2657b6ad81d743e49

Summary: Copy the sample_pdf operation from the NeRF project in to PyTorch3D, in preparation for optimizing it.

Reviewed By: gkioxari

Differential Revision: D27117930

fbshipit-source-id: 20286b007f589a4c4d53ed818c4bc5f2abd22833

Summary:
Add a CPU version to the benchmarks.

```
Benchmark                                                               Avg Time(μs)      Peak Time(μs) Iterations
--------------------------------------------------------------------------------
ADD_POINTS_TO_VOLUMES_cpu_10_trilinear_[25, 25, 25]_1000                    10100           46422             50
ADD_POINTS_TO_VOLUMES_cpu_10_trilinear_[25, 25, 25]_10000                   28442           32100             18
ADD_POINTS_TO_VOLUMES_cpu_10_trilinear_[25, 25, 25]_100000                 241127          254269              3
ADD_POINTS_TO_VOLUMES_cpu_10_trilinear_[101, 111, 121]_1000                 54149           79480             10
ADD_POINTS_TO_VOLUMES_cpu_10_trilinear_[101, 111, 121]_10000               125459          212734              4
ADD_POINTS_TO_VOLUMES_cpu_10_trilinear_[101, 111, 121]_100000              512739          512739              1
ADD_POINTS_TO_VOLUMES_cpu_10_nearest_[25, 25, 25]_1000                       2866           13365            175
ADD_POINTS_TO_VOLUMES_cpu_10_nearest_[25, 25, 25]_10000                      7026           12604             72
ADD_POINTS_TO_VOLUMES_cpu_10_nearest_[25, 25, 25]_100000                    48822           55607             11
ADD_POINTS_TO_VOLUMES_cpu_10_nearest_[101, 111, 121]_1000                   38098           38576             14
ADD_POINTS_TO_VOLUMES_cpu_10_nearest_[101, 111, 121]_10000                  48006           54120             11
ADD_POINTS_TO_VOLUMES_cpu_10_nearest_[101, 111, 121]_100000                131563          138536              4
ADD_POINTS_TO_VOLUMES_cpu_100_trilinear_[25, 25, 25]_1000                   64615           91735              8
ADD_POINTS_TO_VOLUMES_cpu_100_trilinear_[25, 25, 25]_10000                 228815          246095              3
ADD_POINTS_TO_VOLUMES_cpu_100_trilinear_[25, 25, 25]_100000               3086615         3086615              1
ADD_POINTS_TO_VOLUMES_cpu_100_trilinear_[101, 111, 121]_1000               464298          465292              2
ADD_POINTS_TO_VOLUMES_cpu_100_trilinear_[101, 111, 121]_10000             1053440         1053440              1
ADD_POINTS_TO_VOLUMES_cpu_100_trilinear_[101, 111, 121]_100000            6736236         6736236              1
ADD_POINTS_TO_VOLUMES_cpu_100_nearest_[25, 25, 25]_1000                     11940           12440             42
ADD_POINTS_TO_VOLUMES_cpu_100_nearest_[25, 25, 25]_10000                    56641           58051              9
ADD_POINTS_TO_VOLUMES_cpu_100_nearest_[25, 25, 25]_100000                  711492          711492              1
ADD_POINTS_TO_VOLUMES_cpu_100_nearest_[101, 111, 121]_1000                 326437          329846              2
ADD_POINTS_TO_VOLUMES_cpu_100_nearest_[101, 111, 121]_10000                418514          427911              2
ADD_POINTS_TO_VOLUMES_cpu_100_nearest_[101, 111, 121]_100000              1524285         1524285              1
ADD_POINTS_TO_VOLUMES_cuda:0_10_trilinear_[25, 25, 25]_1000                  5949           13602             85
ADD_POINTS_TO_VOLUMES_cuda:0_10_trilinear_[25, 25, 25]_10000                 5817           13001             86
ADD_POINTS_TO_VOLUMES_cuda:0_10_trilinear_[25, 25, 25]_100000               23833           25971             21
ADD_POINTS_TO_VOLUMES_cuda:0_10_trilinear_[101, 111, 121]_1000               9029           16178             56
ADD_POINTS_TO_VOLUMES_cuda:0_10_trilinear_[101, 111, 121]_10000             11595           18601             44
ADD_POINTS_TO_VOLUMES_cuda:0_10_trilinear_[101, 111, 121]_100000            46986           47344             11
ADD_POINTS_TO_VOLUMES_cuda:0_10_nearest_[25, 25, 25]_1000                    2554            9747            196
ADD_POINTS_TO_VOLUMES_cuda:0_10_nearest_[25, 25, 25]_10000                   2676            9537            187
ADD_POINTS_TO_VOLUMES_cuda:0_10_nearest_[25, 25, 25]_100000                  6567           14179             77
ADD_POINTS_TO_VOLUMES_cuda:0_10_nearest_[101, 111, 121]_1000                 5840           12811             86
ADD_POINTS_TO_VOLUMES_cuda:0_10_nearest_[101, 111, 121]_10000                6102           13128             82
ADD_POINTS_TO_VOLUMES_cuda:0_10_nearest_[101, 111, 121]_100000              11945           11995             42
ADD_POINTS_TO_VOLUMES_cuda:0_100_trilinear_[25, 25, 25]_1000                 7642           13671             66
ADD_POINTS_TO_VOLUMES_cuda:0_100_trilinear_[25, 25, 25]_10000               25190           25260             20
ADD_POINTS_TO_VOLUMES_cuda:0_100_trilinear_[25, 25, 25]_100000             212018          212134              3
ADD_POINTS_TO_VOLUMES_cuda:0_100_trilinear_[101, 111, 121]_1000             40421           45692             13
ADD_POINTS_TO_VOLUMES_cuda:0_100_trilinear_[101, 111, 121]_10000            92078           92132              6
ADD_POINTS_TO_VOLUMES_cuda:0_100_trilinear_[101, 111, 121]_100000          457211          457229              2
ADD_POINTS_TO_VOLUMES_cuda:0_100_nearest_[25, 25, 25]_1000                   3574           10377            140
ADD_POINTS_TO_VOLUMES_cuda:0_100_nearest_[25, 25, 25]_10000                  7222           13023             70
ADD_POINTS_TO_VOLUMES_cuda:0_100_nearest_[25, 25, 25]_100000                48127           48165             11
ADD_POINTS_TO_VOLUMES_cuda:0_100_nearest_[101, 111, 121]_1000               34732           35295             15
ADD_POINTS_TO_VOLUMES_cuda:0_100_nearest_[101, 111, 121]_10000              43050           51064             12
ADD_POINTS_TO_VOLUMES_cuda:0_100_nearest_[101, 111, 121]_100000            106028          106058              5
--------------------------------------------------------------------------------
```

Reviewed By: patricklabatut

Differential Revision: D29522830

fbshipit-source-id: 1e857db03613b0c6afcb68a58cdd7ba032e1a874

Summary: Fixes to a couple of comments on points to volumes, make the mask work in round_points_to_volumes, and remove a duplicate rand calculation

Reviewed By: nikhilaravi

Differential Revision: D29522845

fbshipit-source-id: 86770ba37ef3942b909baf63fd73eed1399635b6

Summary: Much of the code is actually available during the conda tests, as long as we look in the right place. We enable some of them.

Reviewed By: nikhilaravi

Differential Revision: D30249357

fbshipit-source-id: 01c57b6b8c04442237965f23eded594aeb90abfb

Summary:
Implementation of ball query from PointNet++.  This function is similar to KNN (find the neighbors in p2 for all points in p1). These are the key differences:
-  It will return the **first** K neighbors within a specified radius as opposed to the **closest** K neighbors.
- As all the points in p2 do not need to be considered to find the closest K, the algorithm is much faster than KNN when p2 has a large number of points.
- The neighbors are not sorted
- Due to the radius threshold it is not guaranteed that there will be K neighbors even if there are more than K points in p2.
- The padding value for `idx` is -1 instead of 0.

# Note:
- Some of the code is very similar to KNN so it could be possible to modify the KNN forward kernels to support ball query.
- Some users might want to use kNN with ball query - for this we could provide a wrapper function around the current `knn_points` which enables applying the radius threshold afterwards as an alternative. This could be called `ball_query_knn`.

Reviewed By: jcjohnson

Differential Revision: D30261362

fbshipit-source-id: 66b6a7e0114beff7164daf7eba21546ff41ec450

Summary: New test that notes and tutorials are listed in the website metadata, so that they will be included in the website build.

Reviewed By: nikhilaravi

Differential Revision: D30223799

fbshipit-source-id: 2dca9730b54e68da2fd430a7b47cb7e18814d518

Summary: Fix to resolve GitHub issue #796 - the cameras were being passed in the renderer forward pass instead of at initialization. The rasterizer was correctly using the cameras passed in the `kwargs` for the projection, but the `cameras` are still part of the `kwargs` for the `get_screen_to_ndc_transform` and `get_ndc_to_screen_transform` functions which is causing issues about duplicate arguments.

Reviewed By: bottler

Differential Revision: D30175679

fbshipit-source-id: 547e88d8439456e728fa2772722df5fa0fe4584d

Summary:
API fix for NDC/screen cameras and compatibility with PyTorch3D renderers.

With this new fix:
* Users can define cameras and `transform_points` under any coordinate system conventions. The transformation applies the camera K and RT to the input points, not regarding for PyTorch3D conventions. So this makes cameras completely independent from PyTorch3D renderer.

* Cameras can be defined either in NDC space or screen space. For existing ones, FoV cameras are in NDC space. Perspective/Orthographic can be defined in NDC or screen space.

* The interface with PyTorch3D renderers happens through `transform_points_ndc` which transforms points to the NDC space and assumes that input points are provided according to PyTorch3D conventions.

* Similarly, `transform_points_screen` transforms points to screen space and again assumes that input points are under PyTorch3D conventions.

* For Orthographic/Perspective cameras, if they are defined in screen space, the `get_ndc_camera_transform` allows points to be converted to NDC for use for the renderers.

Reviewed By: nikhilaravi

Differential Revision: D26932657

fbshipit-source-id: 1a964e3e7caa54d10c792cf39c4d527ba2fb2e79

Summary: A bad env var check meant these tests were not being run. Fix that, and fix the copyright test for the new message format.

Reviewed By: patricklabatut

Differential Revision: D29734562

fbshipit-source-id: a1a9bb68901b09c71c7b4ff81a04083febca8d50

Summary:
# Background
There is an unstable error during training (it can happen after several minutes or after several hours).
The error is connected to `torch.det()` function in  `_check_valid_rotation_matrix()`.

if I remove the function `torch.det()` in `_check_valid_rotation_matrix()` or remove the whole functions `_check_valid_rotation_matrix()` the error is disappeared (D29555876).

# Solution
Replace `torch.det()` with manual implementation for 3x3 matrix.

Reviewed By: patricklabatut

Differential Revision: D29655924

fbshipit-source-id: 41bde1119274a705ab849751ece28873d2c45155

Summary:
Context: in the code we are releasing with CO3D dataset, we use  `cuda()` on TensorProperties like Pointclouds and Cameras where we recursively move batch to a GPU. It would be good to push it to a release so we don’t need to depend on the nightly build.

Additionally, I aligned the logic of `.to("cuda")` without device index to the one of `torch.Tensor` where the current device is populated to index. It should not affect any actual use cases but some tests had to be changed.

Reviewed By: bottler

Differential Revision: D29659529

fbshipit-source-id: abe58aeaca14bacc68da3e6cf5ae07df3353e3ce

Summary: This commit adds a new camera conversion function for OpenCV style parameters to Pulsar parameters to the library. Using this function it addresses a bug reported here: https://fb.workplace.com/groups/629644647557365/posts/1079637302558095, by using the PyTorch3D->OpenCV->Pulsar chain instead of the original direct conversion function. Both conversions are well-tested and an additional test for the full chain has been added, resulting in a more reliable solution requiring less code.

Reviewed By: patricklabatut

Differential Revision: D29322106

fbshipit-source-id: 13df13c2e48f628f75d9f44f19ff7f1646fb7ebd

Summary: Use rotation matrices for OpenCV / PyTorch3D conversions: this avoids hiding issues with conversions to / from axis-angle vectors and ensure new conversion functions have a consistent interface.

Reviewed By: bottler, classner

Differential Revision: D29634099

fbshipit-source-id: 40b28357914eb563fedea60a965dcf69e848ccfa

Summary: Enable this benchmark to be run on its own, like others.

Reviewed By: patricklabatut

Differential Revision: D29522846

fbshipit-source-id: c7b3b5c9a0fcdeeb79d8b2ec197684b4380aa547

Summary: Fixing recent lint problems.

Reviewed By: patricklabatut

Differential Revision: D29522647

fbshipit-source-id: 9bd89fbfa512ecd7359ec355cf12b16fb7024b47

Summary: solve and lstsq have moved around in torch. Cope with both.

Reviewed By: patricklabatut

Differential Revision: D29302316

fbshipit-source-id: b34f0b923e90a357f20df359635929241eba6e74

Summary: Deprecate the `so3_exponential_map()` function in favor of its alias `so3_exp_map()`: this aligns with the naming of `so3_log_map()` and the recently introduced `se3_exp_map()` / `se3_log_map()` pair.

Reviewed By: bottler

Differential Revision: D29329966

fbshipit-source-id: b6f60b9e86b2995f70b1fbeb16f9feea05c55de9

Summary: Add functionality to to save an `.obj` file with associated UV textures: `.png` image and `.mtl` file as well as saving verts_uvs and faces_uvs to the `.obj` file.

Reviewed By: bottler

Differential Revision: D29337562

fbshipit-source-id: 86829b40dae9224088b328e7f5a16eacf8582eb5

Summary:
Refactor of all functions to compute laplacian matrices in one file.
Support for:
* Standard Laplacian
* Cotangent Laplacian
* Norm Laplacian

Reviewed By: nikhilaravi

Differential Revision: D29297466

fbshipit-source-id: b96b88915ce8ef0c2f5693ec9b179fd27b70abf9

Summary: This diff implements the inverse of D28992470 (https://github.com/facebookresearch/pytorch3d/commit/8006842f2a5ab1546a90797a6394f875adce045c): a function to extract OpenCV convention camera parameters from a PyTorch3D `PerspectiveCameras` object. This is the first part of the new PyTorch3d<>OpenCV<>Pulsar conversion functions.

Reviewed By: patricklabatut

Differential Revision: D29278411

fbshipit-source-id: 68d4555b508dbe8685d8239443f839d194cc2484

Summary: License lint codebase

Reviewed By: theschnitz

Differential Revision: D29001799

fbshipit-source-id: 5c59869911785b0181b1663bbf430bc8b7fb2909

Summary: Lint codebase

Reviewed By: bottler

Differential Revision: D29263057

fbshipit-source-id: ac97f01d2a79fead3b09c2cbb21b50ce688a577d

Summary: Because rotations and (rotation) quaternions live on curved manifolds, it doesn't make sense to optimize them directly. Having a prominent option to require gradient on random ones may cause people to try, and isn't particularly useful.

Reviewed By: theschnitz

Differential Revision: D29160734

fbshipit-source-id: fc9e320672349fe334747c5b214655882a460a62

Summary:
Change the cow gltf loading test to validate the texture values and not to validate the renderer output because it has an unstable pixel.

Also a couple of lints.

Reviewed By: patricklabatut

Differential Revision: D29131260

fbshipit-source-id: 5e11f066a2a638588aacb09776cc842173ef669f

Summary: As noted in #710, save_ply was failing with some values of the faces tensor. It was assuming the faces were contiguous in using view() to change them. Here we avoid doing that.

Reviewed By: patricklabatut

Differential Revision: D29159655

fbshipit-source-id: 47214a7ce915bab8d81f109c2b97cde464fd57d8

Summary: Implements a conversion function between OpenCV and PyTorch3D cameras.

Reviewed By: patricklabatut

Differential Revision: D28992470

fbshipit-source-id: dbcc9f213ec293c2f6938261c704aea09aad3c90

Summary:
Implements the SE3 logarithm and exponential maps.
(this is a second part of the split of D23326429)

Outputs of `bm_se3`:
```
Benchmark         Avg Time(μs)      Peak Time(μs) Iterations
--------------------------------------------------------------------------------
SE3_EXP_1                738             885            678
SE3_EXP_10               717             877            698
SE3_EXP_100              718             847            697
SE3_EXP_1000             729            1181            686
--------------------------------------------------------------------------------

Benchmark          Avg Time(μs)      Peak Time(μs) Iterations
--------------------------------------------------------------------------------
SE3_LOG_1               1451            2267            345
SE3_LOG_10              2185            2453            229
SE3_LOG_100             2217            2448            226
SE3_LOG_1000            2455            2599            204
--------------------------------------------------------------------------------
```

Reviewed By: patricklabatut

Differential Revision: D27852557

fbshipit-source-id: e42ccc9cfffe780e9cad24129de15624ae818472

Summary:
Improves so3 functions to make gradient computation stable:
- Instead of `torch.acos`, uses `acos_linear_extrapolation` which has finite gradients of reasonable magnitude for all inputs.
- Adds tests for the latter.

The tests of the finiteness of the gradient in `test_so3_exp_singularity`, `test_so3_exp_singularity`, `test_so3_cos_bound` would fail if the `so3` functions would call `torch.acos` instead of `acos_linear_extrapolation`.

Reviewed By: bottler

Differential Revision: D23326429

fbshipit-source-id: dc296abf2ae3ddfb3942c8146621491a9cb740ee

Summary:
Implements a backprop-safe version of `torch.acos` that linearly extrapolates the function outside bounds.

Below is a plot of the extrapolated acos for different bounds:
{F611339485}

Reviewed By: bottler, nikhilaravi

Differential Revision: D27945714

fbshipit-source-id: fa2e2385b56d6fe534338d5192447c4a3aec540c

Summary:
Fix small face issue for point_mesh distance computation.

The issue lies in the computation of `IsInsideTriangle` which is unstable and non-symmetrical when faces with small areas are given as input. This diff fixes the issue by returning `False` for `IsInsideTriangle` when small faces are given as input.

Reviewed By: bottler

Differential Revision: D29163052

fbshipit-source-id: be297002f26b5e6eded9394fde00553a37406bee

Summary:
- Add MANIFEST.in and `include_package_data=True` to include dataset .json files in the installation
Fix https://github.com/facebookresearch/pytorch3d/issues/435
- Fix `load_textures=False` for ShapeNetDataset with a test
Fix https://github.com/facebookresearch/pytorch3d/issues/450, partly fix https://github.com/facebookresearch/pytorch3d/issues/444. I've set the textures to `None`, if they should be all white instead, let me know.

Pull Request resolved: https://github.com/facebookresearch/pytorch3d/pull/593

Reviewed By: patricklabatut

Differential Revision: D29116264

Pulled By: nikhilaravi

fbshipit-source-id: 1fb0198e616b7f834dfeaf7168bb5e6e530810d1

Summary:
As reported on github, `matrix_to_quaternion` was incorrect for rotations by 180˚. We resolved the sign of the component `i` based on the sign of `i*r`, assuming `r > 0`, which is untrue if `r == 0`.

This diff handles special cases and ensures we use the non-zero elements to copy the sign from.

Reviewed By: bottler

Differential Revision: D29149465

fbshipit-source-id: cd508cc31567fc37ea3463dd7e8c8e8d5d64a235

Summary: Increase code coverage of shader and re-include them in code coverage test

Reviewed By: nikhilaravi

Differential Revision: D29097503

fbshipit-source-id: 2791989ee1562cfa193f3addea0ce72d6840614a

Summary:
There was a bug when `z_clip_value` is not None but there are no faces which are actually visible in the image due to culling.  In `rasterize_meshes.py` a function `convert_clipped_rasterization_to_original_faces` is called to convert the clipped face indices etc back to the unclipped versions, but the case where there is no clipping was not handled correctly.

Fixes Github Issue #632

Reviewed By: bottler

Differential Revision: D29116150

fbshipit-source-id: fae82a0b4848c84b3ed7c7b04ef5c9848352cf5c

Summary: Fixed multiple issues with shape broadcasting in lighting, shading and blending and updated the tests.

Reviewed By: bottler

Differential Revision: D28997941

fbshipit-source-id: d3ef93f979344076b1d9098a86178b4da63844c8

Summary: Increase code coverage of subdivide_meshes and re-include it in code coverage test

Reviewed By: bottler

Differential Revision: D29097476

fbshipit-source-id: 3403ae38a90c4b53f24188eed11faae202a235b5

Summary:
When `z_clip_value = None` and faces are outside the view frustum the shape of one of the tensors in `clip.py` is incorrect.

`faces_num_clipped_verts` should be (F,) but it was (F,3).  Added a new test to ensure this case is handled.

Reviewed By: bottler

Differential Revision: D29051282

fbshipit-source-id: 5f4172ba4d4a75d928404dde9abf48aef18c68bd

Summary:
When textures are set on `Meshes` we need to check if the dimensions actually match that of the verts/faces in the mesh. There was a github issue where someone tried to set the attribute after construction of the `Meshes` object and ran into an error when trying to sample textures.

The desired usage is to initialize the class with the textures (not set an attribute afterwards) but in any case we need to check the dimensions match before sampling textures.

Reviewed By: bottler

Differential Revision: D29020249

fbshipit-source-id: 9fb8a5368b83c9ec53652df92b96fc8b2613f591

Summary: Make Transform3d.to() not ignore a different dtype when device is the same and no copy is requested. Fix other methods where dtype is ignored.

Reviewed By: nikhilaravi

Differential Revision: D28981171

fbshipit-source-id: 4528e6092f4a693aecbe8131ede985fca84e84cf