index.rst 5.44 KB
Newer Older
Ruilong Li's avatar
Ruilong Li committed
1
NerfAcc Documentation
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
2
===================================
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
3

4
NerfAcc is a PyTorch Nerf acceleration toolbox for both training and inference. It focus on
5
6
7
8
9
efficient sampling in the volumetric rendering pipeline of radiance fields, which is 
universal and plug-and-play for most of the NeRFs.
With minimal modifications to the existing codebases, Nerfacc provides significant speedups 
in training various recent NeRF papers.
**And it is pure Python interface with flexible APIs!**
Ruilong Li's avatar
docs  
Ruilong Li committed
10

11
|
Ruilong Li's avatar
docs  
Ruilong Li committed
12

13
14
.. image:: _static/images/teaser.jpg
  :align: center
Ruilong Li's avatar
Ruilong Li committed
15

16
|
17
18

| Github: https://github.com/KAIR-BAIR/nerfacc
Ruilong Li's avatar
Ruilong Li committed
19
| Paper: https://arxiv.org/pdf/2305.04966.pdf
20
| Authors: `Ruilong Li`_, `Hang Gao`_, `Matthew Tancik`_, `Angjoo Kanazawa`_
Ruilong Li's avatar
Ruilong Li committed
21
22

.. note::
Ruilong Li's avatar
Ruilong Li committed
23

Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
24
25
26
   Though this repo only contains examples for single scene optimization,
   we believe generalizable NeRFs across multiple scenes can also be accelerate with our 
   :class:`nerfacc.PropNetEstimator`. Examples will be added soon.
Ruilong Li's avatar
docs  
Ruilong Li committed
27

28

Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
29
30
31
Installation:
-------------

32
**Dependence**: Please install `Pytorch`_ first.
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
33

34
35
36
37
The easist way is to install from PyPI. In this way it will build the CUDA code **on the first run** (JIT).

.. code-block:: console
   
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
38
39
   $ pip install nerfacc

40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
Or install from source. In this way it will build the CUDA code during installation.

.. code-block:: console

   $ pip install git+https://github.com/KAIR-BAIR/nerfacc.git

We also provide pre-built wheels covering major combinations of Pytorch + CUDA versions. 
See our `Github README`_ for what we support.

.. code-block:: console
   
   // e.g. torch 1.13.0 + CUDA 11.7
   $ pip install nerfacc -f https://nerfacc-bucket.s3.us-west-2.amazonaws.com/whl/torch-1.13.0_cu117.html


55
56
57
Usage:
-------------

58
59
60
The idea of NerfAcc is to perform efficient volumetric sampling with a computationally cheap estimator to discover surfaces.
So NerfAcc can work with any user-defined radiance field. To plug the NerfAcc rendering pipeline into your code and enjoy 
the acceleration, you only need to define two functions with your radience field.
61

62
63
64
65
66
- `sigma_fn`: Compute density at each sample. It will be used by the estimator
  (e.g., :class:`nerfacc.OccGridEstimator`, :class:`nerfacc.PropNetEstimator`) to discover surfaces. 
- `rgb_sigma_fn`: Compute color and density at each sample. It will be used by 
  :func:`nerfacc.rendering` to conduct differentiable volumetric rendering. This function 
  will receive gradients to update your radiance field.
67
68
69
70
71
72
73
74
75
76
77
78

An simple example is like this:

.. code-block:: python

   import torch
   from torch import Tensor
   import nerfacc 

   radiance_field = ...  # network: a NeRF model
   rays_o: Tensor = ...  # ray origins. (n_rays, 3)
   rays_d: Tensor = ...  # ray normalized directions. (n_rays, 3)
79
   optimizer = ...  # optimizer
80

81
82
   estimator = nerfacc.OccGridEstimator(...)

83
84
85
   def sigma_fn(
      t_starts: Tensor, t_ends:Tensor, ray_indices: Tensor
   ) -> Tensor:
86
      """ Define how to query density for the estimator."""
87
88
      t_origins = rays_o[ray_indices]  # (n_samples, 3)
      t_dirs = rays_d[ray_indices]  # (n_samples, 3)
89
      positions = t_origins + t_dirs * (t_starts + t_ends)[:, None] / 2.0
90
      sigmas = radiance_field.query_density(positions) 
91
      return sigmas  # (n_samples,)
92
93
94
95

   def rgb_sigma_fn(
      t_starts: Tensor, t_ends: Tensor, ray_indices: Tensor
   ) -> Tuple[Tensor, Tensor]:
96
      """ Query rgb and density values from a user-defined radiance field. """
97
98
      t_origins = rays_o[ray_indices]  # (n_samples, 3)
      t_dirs = rays_d[ray_indices]  # (n_samples, 3)
99
      positions = t_origins + t_dirs * (t_starts + t_ends)[:, None] / 2.0
100
      rgbs, sigmas = radiance_field(positions, condition=t_dirs)  
101
      return rgbs, sigmas  # (n_samples, 3), (n_samples,)
102

103
104
105
106
107
108
   # Efficient Raymarching:
   # ray_indices: (n_samples,). t_starts: (n_samples,). t_ends: (n_samples,).
   ray_indices, t_starts, t_ends = estimator.sampling(
      rays_o, rays_d, sigma_fn=sigma_fn, near_plane=0.2, far_plane=1.0, 
      early_stop_eps=1e-4, alpha_thre=1e-2, 
   )
109
110
111

   # Differentiable Volumetric Rendering.
   # colors: (n_rays, 3). opaicity: (n_rays, 1). depth: (n_rays, 1).
112
   color, opacity, depth, extras = nerfacc.rendering(
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
113
114
      t_starts, t_ends, ray_indices, n_rays=rays_o.shape[0], rgb_sigma_fn=rgb_sigma_fn
   )
115

116
   # Optimize: Both the network and rays will receive gradients
117
118
119
120
121
122
123
124
125
   optimizer.zero_grad()
   loss = F.mse_loss(color, color_gt)
   loss.backward()
   optimizer.step()


Links:
-------------

126
127
128
129
130
131
132
.. toctree::
   :glob:
   :maxdepth: 1
   :caption: Methodology

   methodology/*

Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
133
134
135
.. toctree::
   :glob:
   :maxdepth: 1
136
   :caption: Python API
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
137

138
   apis/*
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
139
140
141
142

.. toctree::
   :glob:
   :maxdepth: 1
143
   :caption: Example Usages and Benchmarks
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
144

145
   examples/*
Ruilong Li's avatar
docs  
Ruilong Li committed
146
147

.. toctree::
Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
148
149
   :maxdepth: 1
   :caption: Projects
Ruilong Li's avatar
docs  
Ruilong Li committed
150

151
   nerfstudio <https://docs.nerf.studio/>
152
   sdfstudio <https://autonomousvision.github.io/sdfstudio/>
153
   instant-nsr-pl <https://github.com/bennyguo/instant-nsr-pl>
Ruilong Li's avatar
Ruilong Li committed
154

Ruilong Li(李瑞龙)'s avatar
Ruilong Li(李瑞龙) committed
155
.. _`vanilla Nerf`: https://arxiv.org/abs/2003.08934
Xiaoming Zhao's avatar
Xiaoming Zhao committed
156
157
.. _`Instant-NGP Nerf`: https://arxiv.org/abs/2201.05989
.. _`D-Nerf`: https://arxiv.org/abs/2011.13961
Ruilong Li's avatar
Ruilong Li committed
158
.. _`MipNerf360`: https://arxiv.org/abs/2111.12077
159
160
161
162
.. _`pixel-Nerf`: https://arxiv.org/abs/2012.02190
.. _`Nerf++`: https://arxiv.org/abs/2010.07492

.. _`Ruilong Li`: https://www.liruilong.cn/
163
.. _`Hang Gao`: https://hangg7.com/
164
.. _`Matthew Tancik`: https://www.matthewtancik.com/
165
166
167
168
169
.. _`Angjoo Kanazawa`: https://people.eecs.berkeley.edu/~kanazawa/

.. _`Github README`: https://github.com/KAIR-BAIR/nerfacc#readme

.. _`PyTorch`: https://pytorch.org/get-started/locally/