@@ -346,7 +352,7 @@ This documentation is intended to serve as a user's guide to nvdiffrast. For det
<blockquote>
<strong>Modular Primitives for High-Performance Differentiable Rendering</strong><br> Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, Timo Aila<br> ACM Transactions on Graphics 39(6) (proc. SIGGRAPH Asia 2020)
@@ -365,16 +371,16 @@ Examples of things we've done with nvdiffrast
<li>PyTorch 1.6 (recommended) or TensorFlow 1.14. TensorFlow 2.x is currently not supported.</li>
<li>A high-end NVIDIA GPU, NVIDIA drivers, CUDA 10.2 toolkit, and cuDNN 7.6.</li>
</ul>
<p>To download nvdiffrast, either download the repository at <ahref="https://github.com/NVlabs/nvdiffrast"class="uri">https://github.com/NVlabs/nvdiffrast</a> as a .zip file, or clone the repository using git:</p>
<p>To download nvdiffrast, either download the repository at <ahref="https://github.com/NVlabs/nvdiffrast">https://github.com/NVlabs/nvdiffrast</a> as a .zip file, or clone the repository using git:</p>
<p>We recommend running nvdiffrast on <ahref="https://www.docker.com/">Docker</a>. To build a Docker image with nvdiffrast and PyTorch 1.6 installed, run:</p>
<p>We recommend using Ubuntu, as some Linux distributions might not have all the required packages available — at least CentOS is reportedly problematic.</p>
<p>To try out some of the provided code examples, run:</p>
<p>Alternatively, if you have all the dependencies taken care of (consult the included Dockerfile for reference), you can install nvdiffrast in your local Python site-packages by running</p>
<p>at the root of the repository. You can also just add the repository root directory to your <code>PYTHONPATH</code>.</p>
<h3id="windows">Windows</h3>
<p>On Windows, nvdiffrast requires an external compiler for compiling the CUDA kernels. The development was done using Microsoft Visual Studio 2017 Professional Edition, and this version works with both PyTorch and TensorFlow versions of nvdiffrast. VS 2019 Professional Edition has also been confirmed to work with the PyTorch version of nvdiffrast. Other VS editions besides Professional Edition, including the Community Edition, should work but have not been tested.</p>
...
...
@@ -382,11 +388,11 @@ Examples of things we've done with nvdiffrast
<spanid="cb6-4"><ahref="#cb6-4"aria-hidden="true"tabindex="-1"></a><spanclass="co"># Run at the root of the repository to install nvdiffrast</span></span>
<p>Nvdiffrast offers four differentiable rendering primitives: <strong>rasterization</strong>, <strong>interpolation</strong>, <strong>texturing</strong>, and <strong>antialiasing</strong>. The operation of the primitives is described here in a platform-agnostic way. Platform-specific documentation can be found in the API reference section.</p>
...
...
@@ -455,7 +461,7 @@ Background replaced with white
</div>
</div>
<p>The middle image above shows the result of texture sampling using the interpolated texture coordinates from the previous step. Why is the background pink? The texture coordinates <spanclass="math inline">(<em>s</em>, <em>t</em>)</span> read as zero at those pixels, but that is a perfectly valid point to sample the texture. It happens that Spot's texture (left) has pink color at its <spanclass="math inline">(0, 0)</span> corner, and therefore all pixels in the background obtain that color as a result of the texture sampling operation. On the right, we have replaced the color of the <q>empty</q> pixels with a white color. Here's one way to do this in PyTorch:</p>
<p>where <code>rast_out</code> is the output of the rasterization operation. We simply test if the <spanclass="math inline"><em>t</em><em>r</em><em>i</em><em>a</em><em>n</em><em>g</em><em>l</em><em>e</em>_<em>i</em><em>d</em></span> field, i.e., channel 3 of the rasterizer output, is greater than zero, indicating that a triangle was rendered in that pixel. If so, we take the color from the textured image, and otherwise we take constant 1.0.</p>
<h3id="antialiasing">Antialiasing</h3>
<p>The last of the four primitive operations in nvdiffrast is antialiasing. Based on the geometry input (vertex positions and triangles), it will smooth out discontinuties at silhouette edges in a given image. The smoothing is based on a local approximation of coverage — an approximate integral over a pixel is calculated based on the exact location of relevant edges and the point-sampled colors at pixel centers.</p>
...
...
@@ -762,13 +768,13 @@ Third depth layer
</div>
</div>
<p>The API for depth peeling is based on <code>DepthPeeler</code> object that acts as a <ahref="https://docs.python.org/3/reference/datamodel.html#context-managers">context manager</a>, and its <code>rasterize_next_layer</code> method. The first call to <code>rasterize_next_layer</code> is equivalent to calling the traditional <code>rasterize</code> function, and subsequent calls report further depth layers. The arguments for rasterization are specified when instantiating the <code>DepthPeeler</code> object. Concretely, your code might look something like this:</p>
<spanid="cb8-2"><ahref="#cb8-2"aria-hidden="true"tabindex="-1"></a><spanclass="cf">for</span> i <spanclass="kw">in</span><spanclass="bu">range</span>(num_layers):</span>
<spanid="cb8-4"><ahref="#cb8-4"aria-hidden="true"tabindex="-1"></a> (process <spanclass="kw">or</span> store the results)</span></code></pre></div>
<p>There is no performance penalty compared to the basic rasterization op if you end up extracting only the first depth layer. In other words, the code above with <code>num_layers=1</code> runs exactly as fast as calling <code>rasterize</code> once.</p>
<p>Depth peeling is only supported in the PyTorch version of nvdiffrast. For implementation reasons, depth peeling reserves the OpenGL context so that other rasterization operations cannot be performed while the peeling is ongoing, i.e., inside the <code>with</code> block. Hence you cannot start a nested depth peeling operation or call <code>rasterize</code> inside the <code>with</code> block, unless you use a different OpenGL context.</p>
<p>For the sake of completeness, let us note the following small caveat: Depth peeling relies on depth values to distinguish surface points from each other. Therefore, culling "previously rendered surface points" actually means culling all surface points at the same or closer depth as those rendered into the pixel in previous passes. This matters only if you have multiple layers of geometry at matching depths — if your geometry consists of, say, nothing but two exactly overlapping triangles, you will see one of them in the first pass but never see the other one in subsequent passes, as it's at the exact depth that is already considered done.</p>
<p>For the sake of completeness, let us note the following small caveat: Depth peeling relies on depth values to distinguish surface points from each other. Therefore, culling "previously rendered surface points" actually means culling all surface points at the same or closer depth as those rendered into the pixel in previous passes. This matters only if you have multiple layers of geometry at matching depths — if your geometry consists of, say, nothing but two exactly overlapping triangles, you will see one of them in the first pass but never see the other one in subsequent passes, as it's at the exact depth that is already considered done.</p>
<h3id="differences-between-pytorch-and-tensorflow">Differences between PyTorch and TensorFlow</h3>
<p>Nvdiffrast can be used from PyTorch and from TensorFlow 1.x; the latter may change to TensorFlow 2.x if there is demand. These frameworks operate somewhat differently and that is reflected in the respective APIs. Simplifying a bit, in TensorFlow 1.x you construct a persistent graph out of persistent nodes, and run many batches of data through it. In PyTorch, there is no persistent graph or nodes, but a new, ephemeral graph is constructed for each batch of data and destroyed immediately afterwards. Therefore, there is also no persistent state for the operations. There is the <code>torch.nn.Module</code> abstraction for festooning operations with persistent state, but we do not use it.</p>
<p>As a consequence, things that would be part of persistent state of an nvdiffrast operation in TensorFlow must be stored by the user in PyTorch, and supplied to the operations as needed. In practice, this is a very small difference and amounts to just a couple of lines of code in most cases.</p>
...
...
@@ -912,9 +918,7 @@ device.</td></tr></table><div class="methods">Methods, only available if context
<pclass="shortdesc">Rasterize triangles.</p><pclass="longdesc">All input tensors must be contiguous and reside in GPU memory except for
the <code>ranges</code> tensor that, if specified, has to reside in CPU memory. The
output tensors will be contiguous and reside in GPU memory.</p><pclass="longdesc">Note: For an unknown reason, on Windows the very first rasterization call using
a newly created OpenGL context may *sometimes* output a blank buffer. This is a
known bug and has never been observed to affect subsequent calls.</p><divclass="arguments">Arguments:</div><tableclass="args"><trclass="arg"><tdclass="argname">glctx</td><tdclass="arg_short">OpenGL context of type <code>RasterizeGLContext</code>.</td></tr><trclass="arg"><tdclass="argname">pos</td><tdclass="arg_short">Vertex position tensor with dtype <code>torch.float32</code>. To enable range
output tensors will be contiguous and reside in GPU memory.</p><divclass="arguments">Arguments:</div><tableclass="args"><trclass="arg"><tdclass="argname">glctx</td><tdclass="arg_short">OpenGL context of type <code>RasterizeGLContext</code>.</td></tr><trclass="arg"><tdclass="argname">pos</td><tdclass="arg_short">Vertex position tensor with dtype <code>torch.float32</code>. To enable range
mode, this tensor should have a 2D shape [num_vertices, 4]. To enable
instanced mode, use a 3D shape [minibatch_size, num_vertices, 4].</td></tr><trclass="arg"><tdclass="argname">tri</td><tdclass="arg_short">Triangle tensor with shape [num_triangles, 3] and dtype <code>torch.int32</code>.</td></tr><trclass="arg"><tdclass="argname">resolution</td><tdclass="arg_short">Output resolution as integer tuple (height, width).</td></tr><trclass="arg"><tdclass="argname">ranges</td><tdclass="arg_short">In range mode, tensor with shape [minibatch_size, 2] and dtype
<code>torch.int32</code>, specifying start indices and counts into <code>tri</code>.
