GitLab

Summary:
User reported that cloned cameras fail to save. The error with latest PyTorch is

```
pickle.PicklingError: Can't pickle ~T_destination: attribute lookup T_destination on torch.nn.modules.module failed
```

This fixes it.

Reviewed By: btgraham

Differential Revision: D39692258

fbshipit-source-id: 75bbf3b8dfa0023dc28bf7d4cc253ca96e46a64d

Summary:
Amend FisheyeCamera by adding tests for all
combination of params and for different batch_sizes.

Reviewed By: kjchalup

Differential Revision: D39176747

fbshipit-source-id: 830d30da24beeb2f0df52db0b17a4303ed53b59c

Summary: Address comments to add benchmarkings for cameras and the new fisheye cameras. The dependency functions in test_cameras have been updated in Diff 1. The following two snapshots show benchmarking results.

Reviewed By: kjchalup

Differential Revision: D38991914

fbshipit-source-id: 51fe9bb7237543e4ee112c9f5068a4cf12a9d482

Summary:
1. A Fisheye camera model that generalizes pinhole camera by considering distortions (i.e. radial, tangential and thin-prism distortions).

2. Added tests against perspective cameras when distortions are off and Aria data points when distortions are on.

3. Address comments to test unhandled shapes between points and transforms. Added tests for __FIELDS, shape broadcasts, cuda etc.

4. Address earlier comments for code efficiency (e.g., adopted torch.norm; torch.solve for matrix inverse; removed inplace operations; unnecessary clone; expand in place of repeat etc).

Reviewed By: jcjohnson

Differential Revision: D38407094

fbshipit-source-id: a3ab48c85c496ac87af692d5d461bb3fc2a2db13

Summary: Reasonable to expect bool indexing.

Reviewed By: bottler, kjchalup

Differential Revision: D38741446

fbshipit-source-id: 22b607bf13110043c5624196c66ca1484fdbce6c

Summary: Move testing targets from pytorch3d/tests/TARGETS to pytorch3d/TARGETS.

Reviewed By: shapovalov

Differential Revision: D36186940

fbshipit-source-id: a4c52c4d99351f885e2b0bf870532d530324039b

Summary:
Applies new import merging and sorting from µsort v1.0.

When merging imports, µsort will make a best-effort to move associated
comments to match merged elements, but there are known limitations due to
the diynamic nature of Python and developer tooling. These changes should
not produce any dangerous runtime changes, but may require touch-ups to
satisfy linters and other tooling.

Note that µsort uses case-insensitive, lexicographical sorting, which
results in a different ordering compared to isort. This provides a more
consistent sorting order, matching the case-insensitive order used when
sorting import statements by module name, and ensures that "frog", "FROG",
and "Frog" always sort next to each other.

For details on µsort's sorting and merging semantics, see the user guide:
https://usort.readthedocs.io/en/stable/guide.html#sorting

Reviewed By: bottler

Differential Revision: D35553814

fbshipit-source-id: be49bdb6a4c25264ff8d4db3a601f18736d17be1

Reviewed By: zsol

Differential Revision: D34526295

fbshipit-source-id: f511370dc3186bc396d68a2e6d5e0931facbeb42

Summary:
Function to join a list of cameras objects into a single batched object.

FB: In the next diff I will remove the `concatenate_cameras` function in implicitron and update the callsites.

Reviewed By: nikhilaravi

Differential Revision: D33198209

fbshipit-source-id: 0c9f5f5df498a0def9dba756c984e6a946618158

Summary: Manual adjustments for license changes.

Reviewed By: patricklabatut

Differential Revision: D33405657

fbshipit-source-id: 8a21735726f3aece9f9164da9e3b272b27db8032

Summary:
Added a custom `__getitem__` method to `CamerasBase` which returns an instance of the appropriate camera instead of the `TensorAccessor` class.

Long term we should deprecate the `TensorAccessor` and the `__getitem__` method on `TensorProperties`

FB: In the next diff I will update the uses of `select_cameras` in implicitron.

Reviewed By: bottler

Differential Revision: D33185885

fbshipit-source-id: c31995d0eb126981e91ba61a6151d5404b263f67

Summary: Fix some comments to match the recent change to transform_points_screen.

Reviewed By: patricklabatut

Differential Revision: D33243697

fbshipit-source-id: dc8d182667a9413bca2c2e3657f97b2f7a47c795

Summary:
All the renderers in PyTorch3D (pointclouds including pulsar, meshes, raysampling) use align_corners=False style. NDC space goes between the edges of the outer pixels. For a non square image with W>H, the vertical NDC space goes from -1 to 1 and the horizontal from -W/H to W/H.

However it was recently pointed out that functionality which deals with screen space inside the camera classes is inconsistent with this. It unintentionally uses align_corners=True. This fixes that.

This would change behaviour of the following:
- If you create a camera in screen coordinates, i.e. setting in_ndc=False, then anything you do with the camera which touches NDC space may be affected, including trying to use renderers. The transform_points_screen function will not be affected...
- If you call the function “transform_points_screen” on a camera defined in NDC space results will be different. I have illustrated in the diff how to get the old results from the new results but this probably isn’t the right long-term solution..

Reviewed By: gkioxari

Differential Revision: D32536305

fbshipit-source-id: 377325a9137282971dcb7ca11a6cba3fc700c9ce

Summary:
API fix for NDC/screen cameras and compatibility with PyTorch3D renderers.

With this new fix:
* Users can define cameras and `transform_points` under any coordinate system conventions. The transformation applies the camera K and RT to the input points, not regarding for PyTorch3D conventions. So this makes cameras completely independent from PyTorch3D renderer.

* Cameras can be defined either in NDC space or screen space. For existing ones, FoV cameras are in NDC space. Perspective/Orthographic can be defined in NDC or screen space.

* The interface with PyTorch3D renderers happens through `transform_points_ndc` which transforms points to the NDC space and assumes that input points are provided according to PyTorch3D conventions.

* Similarly, `transform_points_screen` transforms points to screen space and again assumes that input points are under PyTorch3D conventions.

* For Orthographic/Perspective cameras, if they are defined in screen space, the `get_ndc_camera_transform` allows points to be converted to NDC for use for the renderers.

Reviewed By: nikhilaravi

Differential Revision: D26932657

fbshipit-source-id: 1a964e3e7caa54d10c792cf39c4d527ba2fb2e79

Summary: Deprecate the `so3_exponential_map()` function in favor of its alias `so3_exp_map()`: this aligns with the naming of `so3_log_map()` and the recently introduced `se3_exp_map()` / `se3_log_map()` pair.

Reviewed By: bottler

Differential Revision: D29329966

fbshipit-source-id: b6f60b9e86b2995f70b1fbeb16f9feea05c55de9

Summary: License lint codebase

Reviewed By: theschnitz

Differential Revision: D29001799

fbshipit-source-id: 5c59869911785b0181b1663bbf430bc8b7fb2909

Summary: Small update to the cameras and rasterizer to correctly infer the type of camera (perspective vs orthographic).

Reviewed By: jcjohnson

Differential Revision: D26267225

fbshipit-source-id: a58ed3bc2ab25553d2a4307c734204c1d41b5176

Summary: Allowing usort, isort and black to coexist without fighting means we can't have imports commented as deprecated from the same module as other imports.

Reviewed By: nikhilaravi

Differential Revision: D25372970

fbshipit-source-id: 637f5a0025c0df9fbec47cba73ce5387f4f8b467

Reviewed By: zertosh

Differential Revision: D24880203

fbshipit-source-id: 2034cdfc2712209e86d3d05c119c58f979b05c52

Summary: To initialize the Cameras class currently we require the principal point, focal length and other parameters to be specified from which we calculate the intrinsic matrix. In some cases the matrix might be directly available e.g. from a dataset and the associated metadata for an image.

Reviewed By: nikhilaravi

Differential Revision: D24489509

fbshipit-source-id: 1b411f19c5f6c8074bcfbf613f3339d5e242c119

Summary: When the camera is vertically oriented, calculating the look_at x-axis (also known as the "right" vector) does not succeed, resulting in the x-axis being placed at the origin. Adds a check to correctly calculate the x-axis if this case occurs.

Reviewed By: nikhilaravi, sbranson

Differential Revision: D23511859

fbshipit-source-id: ee5145cdbecdbe2f7c7d288588bd0899480cb327

Summary:
adds `corresponding_cameras_alignment` function that estimates a similarity transformation between two sets of cameras.

The function is essential for computing camera errors in SfM pipelines.

```
Benchmark                                                   Avg Time(μs)      Peak Time(μs) Iterations
--------------------------------------------------------------------------------
CORRESPONDING_CAMERAS_ALIGNMENT_10_centers_False                32219           36211             16
CORRESPONDING_CAMERAS_ALIGNMENT_10_centers_True                 32429           36063             16
CORRESPONDING_CAMERAS_ALIGNMENT_10_extrinsics_False              5548            8782             91
CORRESPONDING_CAMERAS_ALIGNMENT_10_extrinsics_True               6153            9752             82
CORRESPONDING_CAMERAS_ALIGNMENT_100_centers_False               33344           40398             16
CORRESPONDING_CAMERAS_ALIGNMENT_100_centers_True                34528           37095             15
CORRESPONDING_CAMERAS_ALIGNMENT_100_extrinsics_False             5576            7187             90
CORRESPONDING_CAMERAS_ALIGNMENT_100_extrinsics_True              6256            9166             80
CORRESPONDING_CAMERAS_ALIGNMENT_1000_centers_False              32020           37247             16
CORRESPONDING_CAMERAS_ALIGNMENT_1000_centers_True               32776           37644             16
CORRESPONDING_CAMERAS_ALIGNMENT_1000_extrinsics_False            5336            8795             94
CORRESPONDING_CAMERAS_ALIGNMENT_1000_extrinsics_True             6266            9929             80
--------------------------------------------------------------------------------
```

Reviewed By: shapovalov

Differential Revision: D22946415

fbshipit-source-id: 8caae7ee365b304d8aa1f8133cf0dd92c35bc0dd

Summary:
The look_at_view_transform did not give the correct results when the object location `at` was not (0,0,0).

The problem was on computing the cameras' location in world's coordinate `C`. It only took into account the camera position from spherical angles, but ignored the object location in the world's coordinate system. I simply modified the C tensor to take into account the object's location which is not necessarily in the origin.

I ran unit tests and all but 4 failed with the same error message: `RuntimeError: CUDA error: invalid device ordinal`. However the same happens before this patch, so I believe these errors are unrelated.

Pull Request resolved: https://github.com/facebookresearch/pytorch3d/pull/230

Reviewed By: gkioxari

Differential Revision: D23278126

Pulled By: nikhilaravi

fbshipit-source-id: c06e891bc46de8222325ee7b37aa43cde44648e8

Summary:
Refactor cameras
* CamerasBase was enhanced with `transform_points_screen` that transforms projected points from NDC to screen space
* OpenGLPerspective, OpenGLOrthographic -> FoVPerspective, FoVOrthographic
* SfMPerspective, SfMOrthographic -> Perspective, Orthographic
* PerspectiveCamera can optionally be constructred with screen space parameters
* Note on Cameras and coordinate systems was added

Reviewed By: nikhilaravi

Differential Revision: D23168525

fbshipit-source-id: dd138e2b2cc7e0e0d9f34c45b8251c01266a2063

Summary:
Ran the linter.
TODO: need to update the linter as per D21353065.

Reviewed By: bottler

Differential Revision: D21362270

fbshipit-source-id: ad0e781de0a29f565ad25c43bc94a19b1828c020

Summary: Made a CameraBase class. Added `unproject_points` method for each camera class.

Reviewed By: nikhilaravi

Differential Revision: D20373602

fbshipit-source-id: 7e3da5ae420091b5fcab400a9884ef29ad7a7343

Summary:
Fixed a bug in creating the projection matrix for the SfMPerspectiveCameras class. Also fixed the corresponding test in test_cameras.py.

The p0x, p0y are 2D coordinates for the principal point in the NDCS, and therefore should be added AFTER the perspective z divide. I.e. we expect

<a href="https://www.codecogs.com/eqnedit.php?latex=\begin{bmatrix}&space;x\&space;y\&space;z&space;\end{bmatrix}_{\text{NDCS}}&space;=&space;\begin{bmatrix}&space;f_xX/Z&space;&plus;&space;p_x\&space;f_yY/Z&space;&plus;&space;p_y\&space;1&space;/&space;Z\&space;\end{bmatrix}&space;=&space;\text{normalize}\left(&space;\begin{bmatrix}&space;f_xX&space;&plus;&space;p_xZ\&space;f_yY&space;&plus;&space;p_yZ\&space;1\&space;Z&space;\end{bmatrix}&space;\right)" target="_blank"><img src="https://latex.codecogs.com/gif.latex?\begin{bmatrix}&space;x\&space;y\&space;z&space;\end{bmatrix}_{\text{NDCS}}&space;=&space;\begin{bmatrix}&space;f_xX/Z&space;&plus;&space;p_x\&space;f_yY/Z&space;&plus;&space;p_y\&space;1&space;/&space;Z\&space;\end{bmatrix}&space;=&space;\text{normalize}\left(&space;\begin{bmatrix}&space;f_xX&space;&plus;&space;p_xZ\&space;f_yY&space;&plus;&space;p_yZ\&space;1\&space;Z&space;\end{bmatrix}&space;\right)" title="\begin{bmatrix} x\ y\ z \end{bmatrix}_{\text{NDCS}} = \begin{bmatrix} f_xX/Z + p_x\ f_yY/Z + p_y\ 1 / Z\ \end{bmatrix} = \text{normalize}\left( \begin{bmatrix} f_xX + p_xZ\ f_yY + p_yZ\ 1\ Z \end{bmatrix} \right)" /></a>

The current behavior is

<a href="https://www.codecogs.com/eqnedit.php?latex=\begin{bmatrix}&space;x\&space;y\&space;z&space;\end{bmatrix}_{\text{NDCS}}&space;=&space;\begin{bmatrix}&space;f_xX/Z&space;&plus;&space;p_x/Z\&space;f_yY/Z&space;&plus;&space;p_y/Z\&space;1&space;/&space;Z\&space;\end{bmatrix}&space;=&space;\text{normalize}\left(&space;\begin{bmatrix}&space;f_xX&space;&plus;&space;p_x\&space;f_yY&space;&plus;&space;p_y\&space;1\&space;Z&space;\end{bmatrix}&space;\right)" target="_blank"><img src="https://latex.codecogs.com/gif.latex?\begin{bmatrix}&space;x\&space;y\&space;z&space;\end{bmatrix}_{\text{NDCS}}&space;=&space;\begin{bmatrix}&space;f_xX/Z&space;&plus;&space;p_x/Z\&space;f_yY/Z&space;&plus;&space;p_y/Z\&space;1&space;/&space;Z\&space;\end{bmatrix}&space;=&space;\text{normalize}\left(&space;\begin{bmatrix}&space;f_xX&space;&plus;&space;p_x\&space;f_yY&space;&plus;&space;p_y\&space;1\&space;Z&space;\end{bmatrix}&space;\right)" title="\begin{bmatrix} x\ y\ z \end{bmatrix}_{\text{NDCS}} = \begin{bmatrix} f_xX/Z + p_x/Z\ f_yY/Z + p_y/Z\ 1 / Z\ \end{bmatrix} = \text{normalize}\left( \begin{bmatrix} f_xX + p_x\ f_yY + p_y\ 1\ Z \end{bmatrix} \right)" /></a>

which is incorrect.
Pull Request resolved: https://github.com/facebookresearch/pytorch3d/pull/148

Reviewed By: gkioxari

Differential Revision: D21039003

Pulled By: davnov134

fbshipit-source-id: 3e19ac22adbcc39b731ae14052a72fd4ddda2af5

Summary: Address black + isort fbsource linter warnings from D20558374 (previous diff)

Reviewed By: nikhilaravi

Differential Revision: D20558373

fbshipit-source-id: d3607de4a01fb24c0d5269634563a7914bddf1c8

Summary: use assertClose in some tests, which enforces shape equality. Fixes some small problems, including graph_conv on an empty graph.

Reviewed By: nikhilaravi

Differential Revision: D20556912

fbshipit-source-id: 60a61eafe3c03ce0f6c9c1a842685708fb10ac5b

Summary: Run `/dev/linter.sh` to fix linting

Reviewed By: nikhilaravi

Differential Revision: D20584037

fbshipit-source-id: 69e45b33d22e3d54b6d37c3c35580bb3e9dc50a5

Summary:
Create extrinsic parameters from eye point.
Create the rotation and translation from an eye point, look-at point and up vector.
see:
https://www.khronos.org/registry/OpenGL-Refpages/gl2.1/xhtml/gluLookAt.xml

It is arguably easier to initialise a camera position as a point in the world rather than an angle.
Pull Request resolved: https://github.com/facebookresearch/pytorch3d/pull/65

Reviewed By: bottler

Differential Revision: D20419652

Pulled By: nikhilaravi

fbshipit-source-id: 9caa1330860bb8bde1fb5c3864ed4cde836a5d19

Summary: The shebang line `#!<path to interpreter>` is only required for Python scripts, so remove it on source files for class or function definitions. Additionally explicitly mark as executable the actual Python scripts in the codebase.

Reviewed By: nikhilaravi

Differential Revision: D20095778

fbshipit-source-id: d312599fba485e978a243292f88a180d71e1b55a

Summary:
## Updates

- Defined the world and camera coordinates according to this figure. The world coordinates are defined as having +Y up, +X left and +Z in.

{F230888499}

- Removed all flipping from blending functions.
- Updated the rasterizer to return images with +Y up and +X left.
- Updated all the mesh rasterizer tests
    - The expected values are now defined in terms of the default +Y up, +X left
    - Added tests where the triangles in the meshes are non symmetrical so that it is clear which direction +X and +Y are

## Questions:
- Should we have **scene settings** instead of raster settings?
    - To be more correct we should be [z clipping in the rasterizer based on the far/near clipping planes](https://github.com/ShichenLiu/SoftRas/blob/master/soft_renderer/cuda/soft_rasterize_cuda_kernel.cu#L400) - these values are also required in the blending functions so should we make these scene level parameters and have a scene settings tuple which is available to the rasterizer and shader?

Reviewed By: gkioxari

Differential Revision: D20208604

fbshipit-source-id: 55787301b1bffa0afa9618f0a0886cc681da51f3

fbshipit-source-id: ad58e416e3ceeca85fae0583308968d04e78fe0d