0.5.0: Rewrite all the underlying CUDA. Speedup and Benchmarking. (#182)

* importance_sampling with test

* package importance_sampling

* compute_intervals tested and packaged

* compute_intervals_v2

* bicycle is failing

* fix cut in compute_intervals_v2, test pass for rendering

* hacky way to get opaque_bkgd work

* reorg ING

* PackedRaySegmentsSpec

* chunk_ids -> ray_ids

* binary -> occupied

* test_traverse_grid_basic checked

* fix traverse_grid with step size, checked

* support max_step_size, not verified

* _cuda and cuda; upgrade ray_marching

* inclusive scan

* test_exclusive_sum but seems to have numeric error

* inclusive_sum_backward verified

* exclusive sum backward

* merge fwd and bwd for scan

* inclusive & exclusive prod verified

* support normal scan with torch funcs

* rendering and tests

* a bit clean up

* importance_sampling verified

* stratified for importance_sampling

* importance_sampling in pdf.py

* RaySegmentsSpec in data_specs; fix various bugs

* verified with _proposal_packed.py

* importance sampling support batch input/output. need to verify

* prop script with batch samples

* try to use cumsum  instead of cumprod

* searchsorted

* benchmarking prop

* ray_aabb_intersect untested

* update prop benchmark numbers

* minor fixes

* batched ray_aabb_intersect

* ray_aabb_intersect and traverse with grid(s)

* tiny optimize for traverse_grids kernels

* traverse_grids return intervals and samples

* cub not verified

* cleanup

* propnet and occgrid as estimators

* training print iters 10k

* prop is good now

* benchmark in google sheet.

* really cleanup: scan.py and test

* pack.py and test

* rendering and test

* data_specs.py and pdf.py docs

* data_specs.py and pdf.py docs

* init and headers

* grid.py and test for it

* occ grid docs

* generated docs

* example docs for pack and scan function.

* doc fix for volrend.py

* doc fix for pdf.py

* fix doc for rendering function

* docs

* propnet docs

* update scripts

* docs: index.rst

* methodology docs

* docs for examples

* mlp nerf script

* update t-nerf script

* rename dnerf to tnerf

* misc update

* bug fix: pdf_loss with test

* minor fix

* update readme with submodules

* fix format

* update gitingore file

* fix doc failure. teaser png to jpg

* docs in examples/

docs/source/apis/generated/nerfacc.render_visibility.rst

-nerfacc.render\_visibility
-==========================
-
-.. currentmodule:: nerfacc
-
-.. autofunction:: render_visibility
\ No newline at end of file

docs/source/apis/generated/nerfacc.render_visibility_from_alpha.rst

+nerfacc.render\_visibility\_from\_alpha
+=======================================
+
+.. currentmodule:: nerfacc
+
+.. autofunction:: render_visibility_from_alpha
\ No newline at end of file

docs/source/apis/generated/nerfacc.render_visibility_from_density.rst

+nerfacc.render\_visibility\_from\_density
+=========================================
+
+.. currentmodule:: nerfacc
+
+.. autofunction:: render_visibility_from_density
\ No newline at end of file

docs/source/apis/generated/nerfacc.unpack_data.rst → docs/source/apis/generated/nerfacc.searchsorted.rst

-nerfacc.unpack\_data
+nerfacc.searchsorted
 ====================
 
 .. currentmodule:: nerfacc
 
-.. autofunction:: unpack_data
\ No newline at end of file
+.. autofunction:: searchsorted
\ No newline at end of file

docs/source/apis/generated/nerfacc.traverse_grids.rst

+nerfacc.traverse\_grids
+=======================
+
+.. currentmodule:: nerfacc
+
+.. autofunction:: traverse_grids
\ No newline at end of file

docs/source/apis/generated/nerfacc.unpack_info.rst

-nerfacc.unpack\_info
-====================
-
-.. currentmodule:: nerfacc
-
-.. autofunction:: unpack_info
\ No newline at end of file

docs/source/apis/rendering.rst

-Volumetric Rendering
+Rendering
 ===================================
 
-In `nerfacc`, the volumetric rendering pipeline is broken down into 2 steps:
-
-1. **Raymarching**: This is the process of shooting a ray through the scene and
-   generate samples along the way. To perform efficient volumetric rendering, here we aim
-   at skipping as many areas as possible. The emtpy space is skipped by using the cached
-   occupancy grid (see :class:`nerfacc.OccupancyGrid`), and the invisible space is skipped by
-   checking the transmittance of the ray while marching. Almost in all cases, those skipping
-   won't result in a noticeable loss of quality as they would contribute very little to the
-   final rendered image. But they will bring a significant speedup.
-
-2. **Rendering**: This is the process of accumulating samples along the rays into final image.
-   In this step we also need to query the attributes (a.k.a. color and density) of those samples
-   generated by raymarching. Early stoping is supported in this step.
-
-|
-
 .. currentmodule:: nerfacc
 
-.. autofunction:: ray_marching
 .. autofunction:: rendering

docs/source/apis/utils.rst

 Utils
 ===================================
 
+Below are the basic functions that supports sampling and rendering. 
+
 .. currentmodule:: nerfacc
 
 .. autosummary::
    :nosignatures:
    :toctree: generated/
 
-   ray_aabb_intersect
-   unpack_info
+   inclusive_prod
+   exclusive_prod
+   inclusive_sum
+   exclusive_sum
 
-   accumulate_along_rays
-   render_transmittance_from_density
-   render_transmittance_from_alpha
-   render_weight_from_density
+   pack_info
+
+   render_visibility_from_alpha
+   render_visibility_from_density
    render_weight_from_alpha
-   render_visibility
+   render_weight_from_density
+   render_transmittance_from_alpha
+   render_transmittance_from_density
+   accumulate_along_rays
+
+   importance_sampling
+   searchsorted
 
-   pack_data
-   unpack_data
+   ray_aabb_intersect
+   traverse_grids
    
\ No newline at end of file

docs/source/examples/camera.rst

+Camera Optimization NeRFs
+===================================
+
+Performance Overview
+--------------------
+
++----------------------+----------------+----------------------------------+-----------------------+--------------------------+------------------------+------------------------+
+| Methods              | Dataset        | Training Time :math:`\downarrow` | PSNR :math:`\uparrow` | LPIPS :math:`\downarrow` | E_R :math:`\downarrow` | E_T :math:`\downarrow` |
++======================+================+==================================+=======================+==========================+========================+========================+
+| BARF `[1]`_          | NeRF-Synthetic | 586min                           | 28.83                 | 0.054                    | 0.19                   | 0.74                   |
++----------------------+----------------+----------------------------------+-----------------------+--------------------------+------------------------+------------------------+
+| *+nerfacc (occgrid)* |                | 130min                           | 30.11                 | 0.044                    | 0.07                   | 0.35                   |
++----------------------+----------------+----------------------------------+-----------------------+--------------------------+------------------------+------------------------+
+
+Implementation Details
+----------------------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   camera/*
+
+.. _`[1]`: https://arxiv.org/abs/2104.06405

docs/source/examples/camera/barf.rst

+BARF
+====================
+
+More to come...
\ No newline at end of file

docs/source/examples/dynamic.rst

+Dynamic NeRFs
+===================================
+
+Performance Overview
+--------------------
+
++----------------------+-----------+----------------------------------+-----------------------+--------------------------+
+| Methods              | Dataset   | Training Time :math:`\downarrow` | PSNR :math:`\uparrow` | LPIPS :math:`\downarrow` |
++======================+===========+==================================+=======================+==========================+
+| TiNeuVox `[1]`_      | HyperNeRF | 56.3min                          | 24.19                 | 0.425                    |
++----------------------+-----------+----------------------------------+-----------------------+--------------------------+
+| *+nerfacc (occgrid)* |           | 33.0min                          | 24.19                 | 0.434                    |
++----------------------+-----------+----------------------------------+-----------------------+--------------------------+
+| *+nerfacc (propnet)* |           | 34.3min                          | 24.26                 | 0.398                    |
++----------------------+-----------+----------------------------------+-----------------------+--------------------------+
+| TiNeuVox `[1]`_      | D-NeRF    | 11.8min                          | 31.14                 | 0.050                    |
++----------------------+-----------+----------------------------------+-----------------------+--------------------------+
+| *+nerfacc (occgrid)* |           | 4.2min                           | 31.75                 | 0.038                    |
++----------------------+-----------+----------------------------------+-----------------------+--------------------------+
+| K-Planes `[2]`_      | D-NeRF    | 63.9min                          | 30.28                 | 0.043                    |
++----------------------+-----------+----------------------------------+-----------------------+--------------------------+
+| *+nerfacc (occgrid)* |           | 38.8min                          | 30.35                 | 0.042                    |
++----------------------+-----------+----------------------------------+-----------------------+--------------------------+
+| T-NeRF `[3]`_        | D-NeRF    | 20hours                          | 28.78                 | 0.069                    |
++----------------------+-----------+----------------------------------+-----------------------+--------------------------+
+| *+nerfacc (occgrid)* |           | 58min                            | 32.22                 | 0.040                    |
++----------------------+-----------+----------------------------------+-----------------------+--------------------------+
+
+Implementation Details
+----------------------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   dynamic/*
+
+|
+
+3rd-Party Use Cases
+-------------------
+
+- `Representing Volumetric Videos as Dynamic MLP Maps, CVPR 2023 <https://github.com/zju3dv/mlp_maps>`_.
+
+.. _`[1]`: https://arxiv.org/abs/2205.15285
+.. _`[2]`: https://arxiv.org/abs/2301.10241
+.. _`[3]`: https://arxiv.org/abs/2011.13961
\ No newline at end of file

docs/source/examples/dynamic/kplanes.rst

+K-Planes
+====================
+
+More to come...
\ No newline at end of file

docs/source/examples/dynamic/tineuvox.rst

+TiNeuVox
+====================
+
+More to come...
\ No newline at end of file

docs/source/examples/dnerf.rst → docs/source/examples/dynamic/tnerf.rst

-Dynamic Scene
-====================
+.. _`T-NeRF Example`:
 
+T-NeRF 
+====================
 See code `examples/train_mlp_dnerf.py` at our `github repository`_ for details.
 
-Benchmarks
-------------
-*updated on 2022-10-08*
-
-Here we trained a 8-layer-MLP for the radiance field and a 4-layer-MLP for the warping field,
-(similar to the T-Nerf model in the `D-Nerf`_ paper) on the `D-Nerf dataset`_. We used train 
-split for training and test split for evaluation. Our experiments are conducted on a 
-single NVIDIA TITAN RTX GPU. The training memory footprint is about 11GB.
+Radiance Field
+--------------
+Here we implement a very basic time-conditioned NeRF (T-NeRF) model (`examples/radiance_fields/mlp.py`)
+for dynamic scene reconstruction.
+The implementation is mostly follow the T-NeRF described in the `D-NeRF`_ paper, with a 8-layer-MLP 
+for the radiance field and a 4-layer-MLP for the warping field. The only major difference is that
+we reduce the max frequency of the positional encoding from 10 to 4, to respect the fact that the
+motion of the object is relatively smooth.
 
 .. note::
 
-    The :ref:`Occupancy Grid` used in this example is shared by all the frames. In other words, 
+    The :class:`nerfacc.OccGridEstimator` used in this example is shared by all the frames. In other words, 
     instead of using it to indicate the opacity of an area at a single timestamp, 
     Here we use it to indicate the `maximum` opacity at this area `over all the timestamps`.
     It is not optimal but still makes the rendering very efficient.
 
+
+Benchmarks: D-NeRF Dataset
+---------------------------
+*updated on 2022-10-08*
+
+Our experiments are conducted on a single NVIDIA TITAN RTX GPU. 
+The training memory footprint is about 11GB.
+
 +----------------------+----------+---------+-------+---------+-------+--------+---------+-------+-------+
 | PSNR                 | bouncing | hell    | hook  | jumping | lego  | mutant | standup | trex  | MEAN  |
 |                      | balls    | warrior |       | jacks   |       |        |         |       |       |
@@ -27,10 +36,7 @@ single NVIDIA TITAN RTX GPU. The training memory footprint is about 11GB.
 +----------------------+----------+---------+-------+---------+-------+--------+---------+-------+-------+
 | Ours  (~ 1 hr)       | 39.49    | 25.58   | 31.86 | 32.73   | 24.32 | 35.55  | 35.90   | 32.33 | 32.22 |
 +----------------------+----------+---------+-------+---------+-------+--------+---------+-------+-------+
-| Ours  (Training time)| 37min    | 52min   | 69min | 64min   | 44min | 79min  | 79min   | 39min | 58min |
-+----------------------+----------+---------+-------+---------+-------+--------+---------+-------+-------+
 
-.. _`D-Nerf`: https://arxiv.org/abs/2011.13961
-.. _`D-Nerf dataset`: https://www.dropbox.com/s/0bf6fl0ye2vz3vr/data.zip?dl=0
-.. _`github repository`: https://github.com/KAIR-BAIR/nerfacc/tree/76c0f9817da4c9c8b5ccf827eb069ee2ce854b75
+.. _`D-NeRF`: https://arxiv.org/abs/2011.13961
+.. _`github repository`: https://github.com/KAIR-BAIR/nerfacc/
 

docs/source/examples/static.rst

+Static NeRFs
+===================================
+
+Performance Overview
+--------------------
+
++----------------------+----------------+----------------------------------+-----------------------+--------------------------+
+| Methods              | Dataset        | Training Time :math:`\downarrow` | PSNR :math:`\uparrow` | LPIPS :math:`\downarrow` |
++======================+================+==================================+=======================+==========================+
+| TensoRF `[1]`_       | Tanks&Temple   | 18.3min                          | 28.13                 | 0.143                    |
++----------------------+----------------+----------------------------------+-----------------------+--------------------------+
+| *+nerfacc (occgrid)* |                | 12.6min                          | 28.10                 | 0.150                    |
++----------------------+----------------+----------------------------------+-----------------------+--------------------------+
+| TensoRF `[1]`_       | NeRF-Synthetic | 10.6min                          | 32.52                 | 0.047                    |
++----------------------+----------------+----------------------------------+-----------------------+--------------------------+
+| *+nerfacc (occgrid)* |                | 6.5min                           | 32.51                 | 0.044                    |
++----------------------+----------------+----------------------------------+-----------------------+--------------------------+
+| NeRF `[2]`_          | NeRF-Synthetic | 20hours                          | 31.00                 | 0.047                    |
++----------------------+----------------+----------------------------------+-----------------------+--------------------------+
+| *+nerfacc (occgrid)* |                | 52min                            | 31.55                 | 0.072                    |
++----------------------+----------------+----------------------------------+-----------------------+--------------------------+
+| Instant-NGP `[3]`_   | NeRF-Synthetic | 4.4min                           | 32.35                 |                          |
++----------------------+----------------+----------------------------------+-----------------------+--------------------------+
+| *+nerfacc (occgrid)* |                | 4.4min                           | 32.55                 | 0.056                    |
++----------------------+----------------+----------------------------------+-----------------------+--------------------------+
+| *+nerfacc (propnet)* |                | 5.2min                           | 31.40                 | 0.064                    |
++----------------------+----------------+----------------------------------+-----------------------+--------------------------+
+| Instant-NGP `[3]`_   | Mip-NeRF 360   | 5.3min                           | 25.93                 |                          |
++----------------------+----------------+----------------------------------+-----------------------+--------------------------+
+| *+nerfacc (occgrid)* |                | 5.5min                           | 26.38                 | 0.351                    |
++----------------------+----------------+----------------------------------+-----------------------+--------------------------+
+| *+nerfacc (propnet)* |                | 5.0min                           | 27.21                 | 0.300                    |
++----------------------+----------------+----------------------------------+-----------------------+--------------------------+
+
+Implementation Details
+----------------------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   static/*
+
+|
+
+3rd-Party Use Cases
+-------------------
+
+- `nerfstudio <https://docs.nerf.studio/>`_: A collaboration friendly studio for NeRFs.
+- `modelscope <https://github.com/modelscope/modelscope/blob/master/modelscope/models/cv/nerf_recon_acc/network/nerf.py>`_: A collection of deep-learning algorithms.
+- `sdfstudio <https://autonomousvision.github.io/sdfstudio/>`_: A Unified Framework for Surface Reconstruction.
+- `instant-nsr-pl <https://github.com/bennyguo/instant-nsr-pl>`_: Train NeuS in 10min.
+
+.. _`[1]`: https://arxiv.org/abs/2203.09517
+.. _`[2]`: https://arxiv.org/abs/2003.08934
+.. _`[3]`: https://arxiv.org/abs/2201.05989
\ No newline at end of file

docs/source/examples/vanilla.rst → docs/source/examples/static/nerf.rst

-Vanilla Nerf 
-====================
+.. _`Vanilla NeRF Example`:
 
+Vanilla NeRF
+====================
 See code `examples/train_mlp_nerf.py` at our `github repository`_ for details.
 
-Benchmarks
-------------
-*updated on 2022-10-08*
 
-Here we trained a 8-layer-MLP for the radiance field as in the `vanilla Nerf`_. We used the 
-train split for training and test split for evaluation as in the Nerf paper. Our experiments are 
-conducted on a single NVIDIA TITAN RTX GPU. The training memory footprint is about 10GB.
+Radiance Field
+--------------
+We follow the original `NeRF`_ paper to implement a 8-layer-MLP radiance field (`examples/radiance_fields/mlp.py`)
+with positional encoding. 
 
 .. note:: 
     The vanilla Nerf paper uses two MLPs for course-to-fine sampling. Instead here we only use a 
@@ -18,16 +17,22 @@ conducted on a single NVIDIA TITAN RTX GPU. The training memory footprint is abo
     so we can simplly increase the number of samples with a single MLP, to achieve the same goal 
     with the coarse-to-fine sampling, without runtime or memory issue.
 
+
+Benchmark: Nerf-Synthetic Dataset
+---------------------------------
+*updated on 2022-10-08*
+
+Our experiments are conducted on a single NVIDIA TITAN RTX GPU. 
+The training memory footprint is about 10GB.
+
 +----------------------+-------+-------+---------+-------+-------+-------+-------+-------+-------+
 | PSNR                 | Lego  | Mic   |Materials| Chair |Hotdog | Ficus | Drums | Ship  | MEAN  |
 |                      |       |       |         |       |       |       |       |       |       |
 +======================+=======+=======+=========+=======+=======+=======+=======+=======+=======+
 | NeRF  (~ days)       | 32.54 | 32.91 | 29.62   | 33.00 | 36.18 | 30.13 | 25.01 | 28.65 | 31.00 |
 +----------------------+-------+-------+---------+-------+-------+-------+-------+-------+-------+
-| Ours  (~ 50min)      | 33.69 | 33.76 | 29.73   | 33.32 | 35.80 | 32.52 | 25.39 | 28.18 | 31.55 |
-+----------------------+-------+-------+---------+-------+-------+-------+-------+-------+-------+
-| Ours  (Training time)| 58min | 53min | 46min   | 62min | 56min | 42min | 52min | 49min | 52min |
+| Ours  (~ 1 hr)       | 33.69 | 33.76 | 29.73   | 33.32 | 35.80 | 32.52 | 25.39 | 28.18 | 31.55 |
 +----------------------+-------+-------+---------+-------+-------+-------+-------+-------+-------+
 
-.. _`github repository`: https://github.com/KAIR-BAIR/nerfacc/blob/76c0f9817da4c9c8b5ccf827eb069ee2ce854b75/examples/train_mlp_nerf.py
-.. _`vanilla Nerf`: https://arxiv.org/abs/2003.08934
+.. _`github repository`: https://github.com/KAIR-BAIR/nerfacc/
+.. _`NeRF`: https://arxiv.org/abs/2003.08934

docs/source/examples/ngp.rst → docs/source/examples/static/ngp.rst

@@ -3,16 +3,23 @@
 Instant-NGP
 ====================
 
-See code `examples/train_ngp_nerf.py` at our `github repository`_ for details.
+See code `examples/train_ngp_nerf_occ.py` and `examples/train_ngp_nerf_prop.py` at our 
+`github repository`_ for details.
 
-Benchmarks
-------------
-*updated on 2023-03-14*
 
-Here we trained a `Instant-NGP Nerf`_ model on the `Nerf-Synthetic dataset`_. We follow the same
-settings with the Instant-NGP paper, which uses train split for training and test split for
-evaluation. All experiments are conducted on a single NVIDIA TITAN RTX GPU. The training
-memory footprint is about 3GB.
+Radiance Field
+--------------
+We follow the `Instant-NGP`_ paper to implement the radiance field (`examples/radiance_fields/ngp.py`),
+and aligns the hyperparameters (e.g., hashencoder, mlp) with the paper. It is build on top of the
+`tiny-cuda-nn`_ library.
+
+
+Benchmark: Nerf-Synthetic Dataset
+---------------------------------
+*updated on 2023-04-04 with nerfacc==0.5.0*
+
+Our experiments are conducted on a single NVIDIA TITAN RTX GPU. 
+The training memory footprint is about 3GB.
 
 .. note::
     
@@ -28,17 +35,54 @@ memory footprint is about 3GB.
 +=======================+=======+=======+=========+=======+=======+=======+=======+=======+=======+
 |Instant-NGP 35k steps  | 35.87 | 36.22 | 29.08   | 35.10 | 37.48 | 30.61 | 23.85 | 30.62 | 32.35 |
 +-----------------------+-------+-------+---------+-------+-------+-------+-------+-------+-------+
-|(training time)        | 309s  | 258s  | 256s    | 316s  | 292s  | 207s  | 218s  | 250s  | 263s  |
+| training time         | 309s  | 258s  | 256s    | 316s  | 292s  | 207s  | 218s  | 250s  | 263s  |
 +-----------------------+-------+-------+---------+-------+-------+-------+-------+-------+-------+
-|Ours (occ) 20k steps   | 35.81 | 36.87 | 29.59   | 35.70 | 37.45 | 33.63 | 24.98 | 30.64 | 33.08 |
+|Ours (occ) 20k steps   | 35.67 | 36.85 | 29.60   | 35.71 | 37.37 | 33.95 | 25.44 | 30.29 | 33.11 |
 +-----------------------+-------+-------+---------+-------+-------+-------+-------+-------+-------+
-|(training time)        | 288s  | 255s  | 247s    | 319s  | 274s  | 238s  | 247s  | 252s  | 265s  |
+| training time         | 288s  | 260s  | 253s    | 326s  | 272s  | 249s  | 252s  | 251s  | 269s  |
 +-----------------------+-------+-------+---------+-------+-------+-------+-------+-------+-------+
-|Ours (prop) 20k steps  | 34.06 | 34.32 | 27.93   | 34.27 | 36.47 | 31.39 | 24.39 | 30.57 | 31.68 |
+|Ours (prop) 20k steps  | 34.04 | 34.56 | 28.76   | 34.21 | 36.44 | 31.41 | 24.81 | 29.85 | 31.76 |
 +-----------------------+-------+-------+---------+-------+-------+-------+-------+-------+-------+
-|(training time)        | 238s  | 236s  | 250s    | 235s  | 235s  | 236s  | 236s  | 236s  | 240s  |
+| training time         | 225s  | 235s  | 235s    | 240s  | 239s  | 242s  | 258s  | 247s  | 240s  |
 +-----------------------+-------+-------+---------+-------+-------+-------+-------+-------+-------+
 
-.. _`Instant-NGP Nerf`: https://github.com/NVlabs/instant-ngp/tree/51e4107edf48338e9ab0316d56a222e0adf87143
+
+Benchmark: Mip-NeRF 360 Dataset
+---------------------------------
+*updated on 2023-04-04 with nerfacc==0.5.0*
+
+Our experiments are conducted on a single NVIDIA TITAN RTX GPU.
+
+.. note:: 
+    `Ours (prop)` combines the proposal network (:class:`nerfacc.PropNetEstimator`) with the 
+    Instant-NGP radiance field. This is exactly what the `Nerfacto`_ model is doing in the
+    `nerfstudio`_ project. In fact, the hyperparameters for `Ours (prop)` in this experiment
+    are aligned with the `Nerfacto`_ model.
+
++-----------------------+-------+-------+---------+-------+-------+-------+-------+-------+
+| PSNR                  |Bicycle| Garden|   Stump | Bonsai|Counter|Kitchen| Room  | MEAN  |
+|                       |       |       |         |       |       |       |       |       |
++=======================+=======+=======+=========+=======+=======+=======+=======+=======+
+|NeRF++ (~days)         | 22.64 | 24.32 | 23.34   | 29.15 | 26.38 | 27.80 | 28.87 | 26.21 |
++-----------------------+-------+-------+---------+-------+-------+-------+-------+-------+
+|Mip-NeRF 360 (~days)   | 24.37 | 26.98 | 26.40   | 33.46 | 29.55 | 32.23 | 31.63 | 29.23 |
++-----------------------+-------+-------+---------+-------+-------+-------+-------+-------+
+|Instant-NGP 35k steps  | 22.40 | 24.86 | 23.17   | 24.41 | 27.38 | 29.07 | 30.24 | 25.93 |
++-----------------------+-------+-------+---------+-------+-------+-------+-------+-------+
+| training time         | 301s  | 339s  | 295s    | 279s  | 339s  | 366s  | 317s  | 319s  |
++-----------------------+-------+-------+---------+-------+-------+-------+-------+-------+
+|Ours (occ) 20k steps   | 22.40 | 23.94 | 22.98   | 30.09 | 26.84 | 28.03 | 30.60 | 26.41 |
++-----------------------+-------+-------+---------+-------+-------+-------+-------+-------+
+| training time         | 277s  | 302s  | 299s    | 278s  | 315s  | 331s  | 301s  | 300s  |
++-----------------------+-------+-------+---------+-------+-------+-------+-------+-------+
+|Ours (prop) 20k steps  | 23.23 | 25.42 | 25.24   | 30.71 | 26.74 | 30.70 | 30.99 | 27.58 |
++-----------------------+-------+-------+---------+-------+-------+-------+-------+-------+
+| training time         | 276s  | 293s  | 291s    | 291s  | 291s  | 295s  | 287s  | 289s  |
++-----------------------+-------+-------+---------+-------+-------+-------+-------+-------+
+
+
 .. _`github repository`: https://github.com/KAIR-BAIR/nerfacc/
-.. _`Nerf-Synthetic dataset`: https://drive.google.com/drive/folders/1JDdLGDruGNXWnM1eqY1FNL9PlStjaKWi
+.. _`Instant-NGP`: https://arxiv.org/abs/2201.05989
+.. _`tiny-cuda-nn`: https://github.com/NVlabs/tiny-cuda-nn
+.. _`Nerfacto`: https://docs.nerf.studio/en/latest/nerfology/methods/nerfacto.html
+.. _`nerfstudio`: https://docs.nerf.studio/en/latest/
\ No newline at end of file

docs/source/examples/static/tensorf.rst

+TensoRF
+====================
+
+More to come...
\ No newline at end of file

docs/source/examples/unbounded.rst

-Unbounded Scene
-====================
-
-See code `examples/train_ngp_nerf.py` at our `github repository`_ for details.
-
-Benchmarks
-------------
-*updated on 2023-03-14*
-
-Here we trained a `Instant-NGP Nerf`_  on the `MipNerf360`_ dataset. We used train 
-split for training and test split for evaluation. Our experiments are conducted on a 
-single NVIDIA TITAN RTX GPU. The training memory footprint is about 6-9GB.
-
-The main difference between working with unbounded scenes and bounded scenes, is that
-a contraction method is needed to map the infinite space to a finite :ref:`Occupancy Grid`.
-We have difference options provided for this (see :ref:`Occupancy Grid`). The experiments
-here is basically the Instant-NGP experiments (see :ref:`Instant-NGP Example`) with a contraction method
-that takes from `MipNerf360`_.
-
-.. note:: 
-    Even though we are comparing with `Nerf++`_ and `MipNerf360`_, the model and everything are
-    totally different with them. There are plenty of ideas from those papers that would be very
-    helpful for the performance, but we didn't adopt them. As this is just a simple example to 
-    show how to use the library, we didn't want to make it too complicated.
-
-
-+----------------------+-------+-------+-------+-------+-------+-------+-------+-------+
-| PSNR                 |Garden |Bicycle|Bonsai |Counter|Kitchen| Room  | Stump | MEAN  |
-|                      |       |       |       |       |       |       |       |       |
-+======================+=======+=======+=======+=======+=======+=======+=======+=======+
-| Nerf++ (~days)       | 24.32 | 22.64 | 29.15 | 26.38 | 27.80 | 28.87 | 24.34 | 26.21 |
-+----------------------+-------+-------+-------+-------+-------+-------+-------+-------+
-| MipNerf360 (~days)   | 26.98 | 24.37 | 33.46 | 29.55 | 32.23 | 31.63 | 26.40 | 29.23 |
-+----------------------+-------+-------+-------+-------+-------+-------+-------+-------+
-| Ours (occ)           | 24.76 | 22.38 | 29.72 | 26.80 | 28.02 | 30.67 | 22.39 | 26.39 |
-+----------------------+-------+-------+-------+-------+-------+-------+-------+-------+
-| Ours (Training time) | 323s  | 302s  | 300s  | 337s  | 347s  | 320s  | 322s  | 322s  |
-+----------------------+-------+-------+-------+-------+-------+-------+-------+-------+
-| Ours (prop)          | 25.44 | 23.21 | 30.62 | 26.75 | 30.63 | 30.93 | 25.20 | 27.54 |
-+----------------------+-------+-------+-------+-------+-------+-------+-------+-------+
-| Ours (Training time) | 308s  | 304s  | 308s  | 306s  | 313s  | 301s  | 287s  | 304s  |
-+----------------------+-------+-------+-------+-------+-------+-------+-------+-------+
-
-Note `Ours (prop)` is basically a `Nerfacto_` model.
-
-.. _`Instant-NGP Nerf`: https://arxiv.org/abs/2201.05989
-.. _`MipNerf360`: https://arxiv.org/abs/2111.12077
-.. _`Nerf++`: https://arxiv.org/abs/2010.07492
-.. _`github repository`: https://github.com/KAIR-BAIR/nerfacc/
-.. _`Nerfacto`: https://github.com/nerfstudio-project/nerfstudio/blob/main/nerfstudio/models/nerfacto.py
\ No newline at end of file

docs/source/index.rst

@@ -2,23 +2,22 @@ NerfAcc Documentation
 ===================================
 
 NerfAcc is a PyTorch Nerf acceleration toolbox for both training and inference. It focus on
-efficient volumetric rendering of radiance fields, which is universal and plug-and-play for most of the NeRFs.
+efficient sampling in the volumetric rendering pipeline of radiance fields, which is 
+universal and plug-and-play for most of the NeRFs.
+With minimal modifications to the existing codebases, Nerfacc provides significant speedups 
+in training various recent NeRF papers.
+**And it is pure Python interface with flexible APIs!**
 
-Using NerfAcc, 
+|
 
-- The `vanilla Nerf`_ model with 8-layer MLPs can be trained to *better quality* (+~0.5 PNSR) \
-  in *1 hour* rather than *1~2 days* as in the paper.
-- The `Instant-NGP Nerf`_ model can be trained to *equal quality* in *4.5 minutes*, \
-  comparing to the official pure-CUDA implementation.
-- The `D-Nerf`_ model for *dynamic* objects can also be trained in *1 hour* \
-  rather than *2 days* as in the paper, and with *better quality* (+~2.5 PSNR).
-- Both *bounded* and *unbounded* scenes are supported.
+.. image:: _static/images/teaser.jpg
+  :align: center
 
-**And it is pure Python interface with flexible APIs!**
+|
 
 | Github: https://github.com/KAIR-BAIR/nerfacc
-| Paper: https://arxiv.org/pdf/2210.04847.pdf
-| Authors: `Ruilong Li`_, `Matthew Tancik`_, `Angjoo Kanazawa`_
+| Paper: https://arxiv.org/pdf/2210.04847.pdf (To Be Updated)
+| Authors: `Ruilong Li`_, `Hang Gao`_, `Matthew Tancik`_, `Angjoo Kanazawa`_
 
 .. note::
 
@@ -30,20 +29,41 @@ Using NerfAcc,
 Installation:
 -------------
 
-.. code-block:: console
+**Dependence**: Please install `Pytorch`_ first.
 
+The easist way is to install from PyPI. In this way it will build the CUDA code **on the first run** (JIT).
+
+.. code-block:: console
+   
    $ pip install nerfacc
 
+Or install from source. In this way it will build the CUDA code during installation.
+
+.. code-block:: console
+
+   $ pip install git+https://github.com/KAIR-BAIR/nerfacc.git
+
+We also provide pre-built wheels covering major combinations of Pytorch + CUDA versions. 
+See our `Github README`_ for what we support.
+
+.. code-block:: console
+   
+   // e.g. torch 1.13.0 + CUDA 11.7
+   $ pip install nerfacc -f https://nerfacc-bucket.s3.us-west-2.amazonaws.com/whl/torch-1.13.0_cu117.html
+
+
 Usage:
 -------------
 
-The idea of NerfAcc is to perform efficient ray marching and volumetric rendering. 
-So NerfAcc can work with any user-defined radiance field. To plug the NerfAcc rendering
-pipeline into your code and enjoy the acceleration, you only need to define two functions 
-with your radience field.
+The idea of NerfAcc is to perform efficient volumetric sampling with a computationally cheap estimator to discover surfaces.
+So NerfAcc can work with any user-defined radiance field. To plug the NerfAcc rendering pipeline into your code and enjoy 
+the acceleration, you only need to define two functions with your radience field.
 
-- `sigma_fn`: Compute density at each sample. It will be used by :func:`nerfacc.ray_marching` to skip the empty and occluded space during ray marching, which is where the major speedup comes from. 
-- `rgb_sigma_fn`: Compute color and density at each sample. It will be used by :func:`nerfacc.rendering` to conduct differentiable volumetric rendering. This function will receive gradients to update your network.
+- `sigma_fn`: Compute density at each sample. It will be used by the estimator
+  (e.g., :class:`nerfacc.OccGridEstimator`, :class:`nerfacc.PropNetEstimator`) to discover surfaces. 
+- `rgb_sigma_fn`: Compute color and density at each sample. It will be used by 
+  :func:`nerfacc.rendering` to conduct differentiable volumetric rendering. This function 
+  will receive gradients to update your radiance field.
 
 An simple example is like this:
 
@@ -58,48 +78,38 @@ An simple example is like this:
    rays_d: Tensor = ...  # ray normalized directions. (n_rays, 3)
    optimizer = ...  # optimizer
 
+   estimator = nerfacc.OccGridEstimator(...)
+
    def sigma_fn(
       t_starts: Tensor, t_ends:Tensor, ray_indices: Tensor
    ) -> Tensor:
-      """ Query density values from a user-defined radiance field.
-      :params t_starts: Start of the sample interval along the ray. (n_samples, 1).
-      :params t_ends: End of the sample interval along the ray. (n_samples, 1).
-      :params ray_indices: Ray indices that each sample belongs to. (n_samples,).
-      :returns The post-activation density values. (n_samples, 1).
-      """
+      """ Define how to query density for the estimator."""
       t_origins = rays_o[ray_indices]  # (n_samples, 3)
       t_dirs = rays_d[ray_indices]  # (n_samples, 3)
-      positions = t_origins + t_dirs * (t_starts + t_ends) / 2.0
+      positions = t_origins + t_dirs * (t_starts + t_ends)[:, None] / 2.0
       sigmas = radiance_field.query_density(positions) 
-      return sigmas  # (n_samples, 1)
+      return sigmas  # (n_samples,)
 
    def rgb_sigma_fn(
       t_starts: Tensor, t_ends: Tensor, ray_indices: Tensor
    ) -> Tuple[Tensor, Tensor]:
-      """ Query rgb and density values from a user-defined radiance field.
-      :params t_starts: Start of the sample interval along the ray. (n_samples, 1).
-      :params t_ends: End of the sample interval along the ray. (n_samples, 1).
-      :params ray_indices: Ray indices that each sample belongs to. (n_samples,).
-      :returns The post-activation rgb and density values. 
-         (n_samples, 3), (n_samples, 1).
-      """
+      """ Query rgb and density values from a user-defined radiance field. """
       t_origins = rays_o[ray_indices]  # (n_samples, 3)
       t_dirs = rays_d[ray_indices]  # (n_samples, 3)
-      positions = t_origins + t_dirs * (t_starts + t_ends) / 2.0
+      positions = t_origins + t_dirs * (t_starts + t_ends)[:, None] / 2.0
       rgbs, sigmas = radiance_field(positions, condition=t_dirs)  
-      return rgbs, sigmas  # (n_samples, 3), (n_samples, 1)
+      return rgbs, sigmas  # (n_samples, 3), (n_samples,)
 
-   # Efficient Raymarching: Skip empty and occluded space, pack samples from all rays.
-   # ray_indices: (n_samples,). t_starts: (n_samples, 1). t_ends: (n_samples, 1).
-   with torch.no_grad():
-      ray_indices, t_starts, t_ends = nerfacc.ray_marching(
-         rays_o, rays_d, sigma_fn=sigma_fn, near_plane=0.2, far_plane=1.0, 
-         early_stop_eps=1e-4, alpha_thre=1e-2, 
-      )
+   # Efficient Raymarching:
+   # ray_indices: (n_samples,). t_starts: (n_samples,). t_ends: (n_samples,).
+   ray_indices, t_starts, t_ends = estimator.sampling(
+      rays_o, rays_d, sigma_fn=sigma_fn, near_plane=0.2, far_plane=1.0, 
+      early_stop_eps=1e-4, alpha_thre=1e-2, 
+   )
 
    # Differentiable Volumetric Rendering.
    # colors: (n_rays, 3). opaicity: (n_rays, 1). depth: (n_rays, 1).
-   color, opacity, depth = nerfacc.rendering(
+   color, opacity, depth, extras = nerfacc.rendering(
       t_starts, t_ends, ray_indices, n_rays=rays_o.shape[0], rgb_sigma_fn=rgb_sigma_fn
    )
 
@@ -113,6 +123,13 @@ An simple example is like this:
 Links:
 -------------
 
+.. toctree::
+   :glob:
+   :maxdepth: 1
+   :caption: Methodology
+
+   methodology/*
+
 .. toctree::
    :glob:
    :maxdepth: 1
@@ -132,6 +149,7 @@ Links:
    :caption: Projects
 
    nerfstudio <https://docs.nerf.studio/>
+   instant-nsr-pl <https://github.com/bennyguo/instant-nsr-pl>
 
 
 .. _`vanilla Nerf`: https://arxiv.org/abs/2003.08934
@@ -142,5 +160,10 @@ Links:
 .. _`Nerf++`: https://arxiv.org/abs/2010.07492
 
 .. _`Ruilong Li`: https://www.liruilong.cn/
+.. _`Hang Gao`: https://hangg7.com/
 .. _`Matthew Tancik`: https://www.matthewtancik.com/
-.. _`Angjoo Kanazawa`: https://people.eecs.berkeley.edu/~kanazawa/
\ No newline at end of file
+.. _`Angjoo Kanazawa`: https://people.eecs.berkeley.edu/~kanazawa/
+
+.. _`Github README`: https://github.com/KAIR-BAIR/nerfacc#readme
+
+.. _`PyTorch`: https://pytorch.org/get-started/locally/