initial commit

.gitignore

+SparseConvNetTorch/build/
+*.t7
+t7/
+*.o
+*.a
+*.so
+build
+__pycache__
+pickle
\ No newline at end of file

LICENSE

+Attribution-NonCommercial 4.0 International
+
+=======================================================================
+
+Creative Commons Corporation ("Creative Commons") is not a law firm and
+does not provide legal services or legal advice. Distribution of
+Creative Commons public licenses does not create a lawyer-client or
+other relationship. Creative Commons makes its licenses and related
+information available on an "as-is" basis. Creative Commons gives no
+warranties regarding its licenses, any material licensed under their
+terms and conditions, or any related information. Creative Commons
+disclaims all liability for damages resulting from their use to the
+fullest extent possible.
+
+Using Creative Commons Public Licenses
+
+Creative Commons public licenses provide a standard set of terms and
+conditions that creators and other rights holders may use to share
+original works of authorship and other material subject to copyright
+and certain other rights specified in the public license below. The
+following considerations are for informational purposes only, are not
+exhaustive, and do not form part of our licenses.
+
+     Considerations for licensors: Our public licenses are
+     intended for use by those authorized to give the public
+     permission to use material in ways otherwise restricted by
+     copyright and certain other rights. Our licenses are
+     irrevocable. Licensors should read and understand the terms
+     and conditions of the license they choose before applying it.
+     Licensors should also secure all rights necessary before
+     applying our licenses so that the public can reuse the
+     material as expected. Licensors should clearly mark any
+     material not subject to the license. This includes other CC-
+     licensed material, or material used under an exception or
+     limitation to copyright. More considerations for licensors:
+	wiki.creativecommons.org/Considerations_for_licensors
+
+     Considerations for the public: By using one of our public
+     licenses, a licensor grants the public permission to use the
+     licensed material under specified terms and conditions. If
+     the licensor's permission is not necessary for any reason--for
+     example, because of any applicable exception or limitation to
+     copyright--then that use is not regulated by the license. Our
+     licenses grant only permissions under copyright and certain
+     other rights that a licensor has authority to grant. Use of
+     the licensed material may still be restricted for other
+     reasons, including because others have copyright or other
+     rights in the material. A licensor may make special requests,
+     such as asking that all changes be marked or described.
+     Although not required by our licenses, you are encouraged to
+     respect those requests where reasonable. More_considerations
+     for the public: 
+	wiki.creativecommons.org/Considerations_for_licensees
+
+=======================================================================
+
+Creative Commons Attribution-NonCommercial 4.0 International Public
+License
+
+By exercising the Licensed Rights (defined below), You accept and agree
+to be bound by the terms and conditions of this Creative Commons
+Attribution-NonCommercial 4.0 International Public License ("Public
+License"). To the extent this Public License may be interpreted as a
+contract, You are granted the Licensed Rights in consideration of Your
+acceptance of these terms and conditions, and the Licensor grants You
+such rights in consideration of benefits the Licensor receives from
+making the Licensed Material available under these terms and
+conditions.
+
+
+Section 1 -- Definitions.
+
+  a. Adapted Material means material subject to Copyright and Similar
+     Rights that is derived from or based upon the Licensed Material
+     and in which the Licensed Material is translated, altered,
+     arranged, transformed, or otherwise modified in a manner requiring
+     permission under the Copyright and Similar Rights held by the
+     Licensor. For purposes of this Public License, where the Licensed
+     Material is a musical work, performance, or sound recording,
+     Adapted Material is always produced where the Licensed Material is
+     synched in timed relation with a moving image.
+
+  b. Adapter's License means the license You apply to Your Copyright
+     and Similar Rights in Your contributions to Adapted Material in
+     accordance with the terms and conditions of this Public License.
+
+  c. Copyright and Similar Rights means copyright and/or similar rights
+     closely related to copyright including, without limitation,
+     performance, broadcast, sound recording, and Sui Generis Database
+     Rights, without regard to how the rights are labeled or
+     categorized. For purposes of this Public License, the rights
+     specified in Section 2(b)(1)-(2) are not Copyright and Similar
+     Rights.
+  d. Effective Technological Measures means those measures that, in the
+     absence of proper authority, may not be circumvented under laws
+     fulfilling obligations under Article 11 of the WIPO Copyright
+     Treaty adopted on December 20, 1996, and/or similar international
+     agreements.
+
+  e. Exceptions and Limitations means fair use, fair dealing, and/or
+     any other exception or limitation to Copyright and Similar Rights
+     that applies to Your use of the Licensed Material.
+
+  f. Licensed Material means the artistic or literary work, database,
+     or other material to which the Licensor applied this Public
+     License.
+
+  g. Licensed Rights means the rights granted to You subject to the
+     terms and conditions of this Public License, which are limited to
+     all Copyright and Similar Rights that apply to Your use of the
+     Licensed Material and that the Licensor has authority to license.
+
+  h. Licensor means the individual(s) or entity(ies) granting rights
+     under this Public License.
+
+  i. NonCommercial means not primarily intended for or directed towards
+     commercial advantage or monetary compensation. For purposes of
+     this Public License, the exchange of the Licensed Material for
+     other material subject to Copyright and Similar Rights by digital
+     file-sharing or similar means is NonCommercial provided there is
+     no payment of monetary compensation in connection with the
+     exchange.
+
+  j. Share means to provide material to the public by any means or
+     process that requires permission under the Licensed Rights, such
+     as reproduction, public display, public performance, distribution,
+     dissemination, communication, or importation, and to make material
+     available to the public including in ways that members of the
+     public may access the material from a place and at a time
+     individually chosen by them.
+
+  k. Sui Generis Database Rights means rights other than copyright
+     resulting from Directive 96/9/EC of the European Parliament and of
+     the Council of 11 March 1996 on the legal protection of databases,
+     as amended and/or succeeded, as well as other essentially
+     equivalent rights anywhere in the world.
+
+  l. You means the individual or entity exercising the Licensed Rights
+     under this Public License. Your has a corresponding meaning.
+
+
+Section 2 -- Scope.
+
+  a. License grant.
+
+       1. Subject to the terms and conditions of this Public License,
+          the Licensor hereby grants You a worldwide, royalty-free,
+          non-sublicensable, non-exclusive, irrevocable license to
+          exercise the Licensed Rights in the Licensed Material to:
+
+            a. reproduce and Share the Licensed Material, in whole or
+               in part, for NonCommercial purposes only; and
+
+            b. produce, reproduce, and Share Adapted Material for
+               NonCommercial purposes only.
+
+       2. Exceptions and Limitations. For the avoidance of doubt, where
+          Exceptions and Limitations apply to Your use, this Public
+          License does not apply, and You do not need to comply with
+          its terms and conditions.
+
+       3. Term. The term of this Public License is specified in Section
+          6(a).
+
+       4. Media and formats; technical modifications allowed. The
+          Licensor authorizes You to exercise the Licensed Rights in
+          all media and formats whether now known or hereafter created,
+          and to make technical modifications necessary to do so. The
+          Licensor waives and/or agrees not to assert any right or
+          authority to forbid You from making technical modifications
+          necessary to exercise the Licensed Rights, including
+          technical modifications necessary to circumvent Effective
+          Technological Measures. For purposes of this Public License,
+          simply making modifications authorized by this Section 2(a)
+          (4) never produces Adapted Material.
+
+       5. Downstream recipients.
+
+            a. Offer from the Licensor -- Licensed Material. Every
+               recipient of the Licensed Material automatically
+               receives an offer from the Licensor to exercise the
+               Licensed Rights under the terms and conditions of this
+               Public License.
+
+            b. No downstream restrictions. You may not offer or impose
+               any additional or different terms or conditions on, or
+               apply any Effective Technological Measures to, the
+               Licensed Material if doing so restricts exercise of the
+               Licensed Rights by any recipient of the Licensed
+               Material.
+
+       6. No endorsement. Nothing in this Public License constitutes or
+          may be construed as permission to assert or imply that You
+          are, or that Your use of the Licensed Material is, connected
+          with, or sponsored, endorsed, or granted official status by,
+          the Licensor or others designated to receive attribution as
+          provided in Section 3(a)(1)(A)(i).
+
+  b. Other rights.
+
+       1. Moral rights, such as the right of integrity, are not
+          licensed under this Public License, nor are publicity,
+          privacy, and/or other similar personality rights; however, to
+          the extent possible, the Licensor waives and/or agrees not to
+          assert any such rights held by the Licensor to the limited
+          extent necessary to allow You to exercise the Licensed
+          Rights, but not otherwise.
+
+       2. Patent and trademark rights are not licensed under this
+          Public License.
+
+       3. To the extent possible, the Licensor waives any right to
+          collect royalties from You for the exercise of the Licensed
+          Rights, whether directly or through a collecting society
+          under any voluntary or waivable statutory or compulsory
+          licensing scheme. In all other cases the Licensor expressly
+          reserves any right to collect such royalties, including when
+          the Licensed Material is used other than for NonCommercial
+          purposes.
+
+
+Section 3 -- License Conditions.
+
+Your exercise of the Licensed Rights is expressly made subject to the
+following conditions.
+
+  a. Attribution.
+
+       1. If You Share the Licensed Material (including in modified
+          form), You must:
+
+            a. retain the following if it is supplied by the Licensor
+               with the Licensed Material:
+
+                 i. identification of the creator(s) of the Licensed
+                    Material and any others designated to receive
+                    attribution, in any reasonable manner requested by
+                    the Licensor (including by pseudonym if
+                    designated);
+
+                ii. a copyright notice;
+
+               iii. a notice that refers to this Public License;
+
+                iv. a notice that refers to the disclaimer of
+                    warranties;
+
+                 v. a URI or hyperlink to the Licensed Material to the
+                    extent reasonably practicable;
+
+            b. indicate if You modified the Licensed Material and
+               retain an indication of any previous modifications; and
+
+            c. indicate the Licensed Material is licensed under this
+               Public License, and include the text of, or the URI or
+               hyperlink to, this Public License.
+
+       2. You may satisfy the conditions in Section 3(a)(1) in any
+          reasonable manner based on the medium, means, and context in
+          which You Share the Licensed Material. For example, it may be
+          reasonable to satisfy the conditions by providing a URI or
+          hyperlink to a resource that includes the required
+          information.
+
+       3. If requested by the Licensor, You must remove any of the
+          information required by Section 3(a)(1)(A) to the extent
+          reasonably practicable.
+
+       4. If You Share Adapted Material You produce, the Adapter's
+          License You apply must not prevent recipients of the Adapted
+          Material from complying with this Public License.
+
+
+Section 4 -- Sui Generis Database Rights.
+
+Where the Licensed Rights include Sui Generis Database Rights that
+apply to Your use of the Licensed Material:
+
+  a. for the avoidance of doubt, Section 2(a)(1) grants You the right
+     to extract, reuse, reproduce, and Share all or a substantial
+     portion of the contents of the database for NonCommercial purposes
+     only;
+
+  b. if You include all or a substantial portion of the database
+     contents in a database in which You have Sui Generis Database
+     Rights, then the database in which You have Sui Generis Database
+     Rights (but not its individual contents) is Adapted Material; and
+
+  c. You must comply with the conditions in Section 3(a) if You Share
+     all or a substantial portion of the contents of the database.
+
+For the avoidance of doubt, this Section 4 supplements and does not
+replace Your obligations under this Public License where the Licensed
+Rights include other Copyright and Similar Rights.
+
+
+Section 5 -- Disclaimer of Warranties and Limitation of Liability.
+
+  a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
+     EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
+     AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
+     ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
+     IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
+     WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
+     PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
+     ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
+     KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT
+     ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.
+
+  b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
+     TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
+     NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
+     INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
+     COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
+     USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
+     ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
+     DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR
+     IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.
+
+  c. The disclaimer of warranties and limitation of liability provided
+     above shall be interpreted in a manner that, to the extent
+     possible, most closely approximates an absolute disclaimer and
+     waiver of all liability.
+
+
+Section 6 -- Term and Termination.
+
+  a. This Public License applies for the term of the Copyright and
+     Similar Rights licensed here. However, if You fail to comply with
+     this Public License, then Your rights under this Public License
+     terminate automatically.
+
+  b. Where Your right to use the Licensed Material has terminated under
+     Section 6(a), it reinstates:
+
+       1. automatically as of the date the violation is cured, provided
+          it is cured within 30 days of Your discovery of the
+          violation; or
+
+       2. upon express reinstatement by the Licensor.
+
+     For the avoidance of doubt, this Section 6(b) does not affect any
+     right the Licensor may have to seek remedies for Your violations
+     of this Public License.
+
+  c. For the avoidance of doubt, the Licensor may also offer the
+     Licensed Material under separate terms or conditions or stop
+     distributing the Licensed Material at any time; however, doing so
+     will not terminate this Public License.
+
+  d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
+     License.
+
+
+Section 7 -- Other Terms and Conditions.
+
+  a. The Licensor shall not be bound by any additional or different
+     terms or conditions communicated by You unless expressly agreed.
+
+  b. Any arrangements, understandings, or agreements regarding the
+     Licensed Material not stated herein are separate from and
+     independent of the terms and conditions of this Public License.
+
+
+Section 8 -- Interpretation.
+
+  a. For the avoidance of doubt, this Public License does not, and
+     shall not be interpreted to, reduce, limit, restrict, or impose
+     conditions on any use of the Licensed Material that could lawfully
+     be made without permission under this Public License.
+
+  b. To the extent possible, if any provision of this Public License is
+     deemed unenforceable, it shall be automatically reformed to the
+     minimum extent necessary to make it enforceable. If the provision
+     cannot be reformed, it shall be severed from this Public License
+     without affecting the enforceability of the remaining terms and
+     conditions.
+
+  c. No term or condition of this Public License will be waived and no
+     failure to comply consented to unless expressly agreed to by the
+     Licensor.
+
+  d. Nothing in this Public License constitutes or may be interpreted
+     as a limitation upon, or waiver of, any privileges and immunities
+     that apply to the Licensor or You, including from the legal
+     processes of any jurisdiction or authority.
+
+=======================================================================
+
+Creative Commons is not a party to its public
+licenses. Notwithstanding, Creative Commons may elect to apply one of
+its public licenses to material it publishes and in those instances
+will be considered the “Licensor.” The text of the Creative Commons
+public licenses is dedicated to the public domain under the CC0 Public
+Domain Dedication. Except for the limited purpose of indicating that
+material is shared under a Creative Commons public license or as
+otherwise permitted by the Creative Commons policies published at
+creativecommons.org/policies, Creative Commons does not authorize the
+use of the trademark "Creative Commons" or any other trademark or logo
+of Creative Commons without its prior written consent including,
+without limitation, in connection with any unauthorized modifications
+to any of its public licenses or any other arrangements,
+understandings, or agreements concerning use of licensed material. For
+the avoidance of doubt, this paragraph does not form part of the
+public licenses.
+
+Creative Commons may be contacted at creativecommons.org.
+

PyTorch/setup.py

+# Copyright 2016-present, Facebook, Inc.
+# All rights reserved.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+import os
+import torch
+from torch.utils.ffi import create_extension
+this_dir = os.path.dirname(os.path.realpath(__file__))
+torch_dir = os.path.dirname(torch.__file__)
+
+print('Building SCN module')
+if torch.cuda.is_available():
+    r = os.system(
+        'cd sparseconvnet/SCN; nvcc init.cu -c -o init.cu.o -ccbin /usr/bin/cc -m64 --std c++11 -Xcompiler ,\"-fopenmp\",\"-fPIC\",\"-O3\",\"-DNDEBUG\" -gencode arch=compute_52,code=sm_52 -gencode arch=compute_20,code=sm_21 -DNVCC -I/usr/local/cuda/include -I' +
+        torch_dir +
+        '/lib/include -I' +
+        torch_dir +
+        '/lib/include/TH -I' +
+        torch_dir +
+        '/lib/include/THC -I.')
+    assert r == 0
+    ffi = create_extension(
+        'sparseconvnet.SCN',
+        headers=[
+            'sparseconvnet/SCN/header_cpu.h',
+            'sparseconvnet/SCN/header_gpu.h'],
+        sources=[],
+        extra_objects=[
+            this_dir +
+            '/sparseconvnet/SCN/init.cu.o'],
+        relative_to=__file__,
+        with_cuda=True)
+else:
+    r = os.system(
+        'cd sparseconvnet/SCN; g++ -std=c++11 -fPIC -c init.cpp -o init.cpp.o -I' +
+        torch_dir +
+        '/lib/include -I' +
+        torch_dir +
+        '/lib/include/TH -I.')
+    assert r == 0
+    ffi = create_extension(
+        'sparseconvnet.SCN',
+        headers=['sparseconvnet/SCN/header_cpu.h'],
+        sources=[],
+        extra_objects=[
+            this_dir +
+            '/sparseconvnet/SCN/init.cpp.o'],
+        relative_to=__file__,
+        with_cuda=False)
+
+ffi.build()
+
+from setuptools import setup, find_packages
+setup(
+    name='sparseconvnet',
+    version='0.1',
+    description='Submanifold (Spatially) Sparse Convolutional Networks https://arxiv.org/abs/1706.01307',
+    author='Facebook AI Research',
+    author_email='benjamingraham@fb.com',
+    url='https://github.com/facebookresearch/SparseConvNet',
+    packages=find_packages(),
+)

PyTorch/sparseconvnet/SCN/drawCurve.cpp

+// Copyright 2016-present, Facebook, Inc.
+// All rights reserved.
+//
+// This source code is licensed under the license found in the
+// LICENSE file in the root directory of this source tree.
+
+// Helper function to draw pen strokes with
+// nPlanes = 3, feature vector = (1,dx,dy)
+extern "C" void scn_2_drawCurve(void **m, THFloatTensor *features,
+                                THFloatTensor *stroke) {
+  auto location = THLongTensor_newWithSize1d(2);
+  auto location_ = THLongTensor_data(location);
+
+  auto vec = THFloatTensor_newWithSize1d(3);
+  auto vec_ = THFloatTensor_data(vec);
+
+  int n = THFloatTensor_size(stroke, 0) - 1;
+  float *s = THFloatTensor_data(stroke); // stroke is a [n+1,2] array
+  long idx = 0;
+  float x1, y1, x2, y2; // n line segments (x1,y1) to (x2,y2)
+  x2 = s[idx++];
+  y2 = s[idx++];
+  for (int i = 0; i < n; ++i) {
+    x1 = x2;
+    y1 = y2;
+    x2 = s[idx++];
+    y2 = s[idx++];
+    float inverse_length =
+        powf(1e-10 + (x2 - x1) * (x2 - x1) + (y2 - y1) * (y2 - y1), -0.5);
+    vec_[0] = 1;
+    vec_[1] = (x2 - x1) * inverse_length;
+    vec_[2] = (y2 - y1) * inverse_length;
+    for (float a = 0; a < 1; a += inverse_length) {
+      location_[0] = x1 * a + x2 * (1 - a);
+      location_[1] = y1 * a + y2 * (1 - a);
+      scn_2_setInputSpatialLocation(m, features, location, vec, false);
+    }
+  }
+  THLongTensor_free(location);
+  THFloatTensor_free(vec);
+}

PyTorch/sparseconvnet/SCN/generic/32bits.h

+// Copyright 2016-present, Facebook, Inc.
+// All rights reserved.
+//
+// This source code is licensed under the license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include <array>
+#include <tuple>
+
+// Using 32 bit integers for coordinates and memory calculations.
+// They could be replaced with 64 bit integers.
+// Advantages of 64 bit:
+// - support for nFeatures * nActiveSites > 2^32 per hidden layer per batch
+// Disadvantages:
+// - larger, and therefore slower, data copies from CPU -> GPU
+// - more device memory needed to store sparseconvnet 'rulebooks'
+// - not really needed until GPUs have >> 32GB RAM
+
+using Int = int32_t;
+using uInt = uint32_t; // Max value = uInt_MAX used to denote 'non-existent'
+const uInt uInt_MAX = 4294967295; // 2^32-1
+const uInt Int_MAX = 2147483647;  // 2^31-1
+
+// Point<dimension> is a point in the d-dimensional integer lattice
+// (i.e. square-grid/cubic-grid, ...)
+template <uInt dimension> using Point = std::array<Int, dimension>;
+
+template <uInt dimension> Point<dimension> LongTensorToPoint(THLongTensor *t) {
+  Point<dimension> p;
+  long *td = THLongTensor_data(t);
+  for (int i = 0; i < dimension; i++)
+    p[i] = td[i];
+  return p;
+}
+template <uInt dimension>
+Point<2*dimension> TwoLongTensorsToPoint(THLongTensor *t0, THLongTensor *t1) {
+  Point<2 * dimension> p;
+  long *td;
+  td = THLongTensor_data(t0);
+  for (int i = 0; i < dimension; i++)
+    p[i] = td[i];
+  td = THLongTensor_data(t1);
+  for (int i = 0; i < dimension; i++)
+    p[i + dimension] = td[i];
+  return p;
+}
+template <uInt dimension>
+Point<3*dimension> ThreeLongTensorsToPoint(THLongTensor *t0, THLongTensor *t1,
+                                         THLongTensor *t2) {
+  Point<3 * dimension> p;
+  long *td;
+  td = THLongTensor_data(t0);
+  for (int i = 0; i < dimension; i++)
+    p[i] = td[i];
+  td = THLongTensor_data(t1);
+  for (int i = 0; i < dimension; i++)
+    p[i + dimension] = td[i];
+  td = THLongTensor_data(t2);
+  for (int i = 0; i < dimension; i++)
+    p[i + 2 * dimension] = td[i];
+  return p;
+}
+
+// FNV Hash function for Point<dimension>
+template <uInt dimension> struct IntArrayHash {
+  std::size_t operator()(Point<dimension> const &p) const {
+    uInt hash = 16777619;
+    for (auto x : p) {
+      hash *= 2166136261;
+      hash ^= x;
+    }
+    return hash;
+  }
+};
+
+#define THCITensor THCudaIntTensor
+#define THCITensor_nElement THCudaIntTensor_nElement
+#define THCITensor_resize1d THCudaIntTensor_resize1d
+#define THCITensor_data THCudaIntTensor_data

PyTorch/sparseconvnet/SCN/generic/64bits.h

+// Copyright 2016-present, Facebook, Inc.
+// All rights reserved.
+//
+// This source code is licensed under the license found in the
+// LICENSE file in the root directory of this source tree.
+
+#include <array>
+#include <tuple>
+
+// Using 32 bit integers for coordinates and memory calculations.
+// They could be replaced with 64 bit integers.
+// Advantages of 64 bit:
+// - support for nFeatures * nActiveSites > 2^32 per hidden layer per batch
+// Disadvantages:
+// - larger, and therefore slower, data copies from CPU -> GPU
+// - more device memory needed to store sparseconvnet 'rulebooks'
+// - not really needed until GPUs have >> 32GB RAM
+
+using Int = int64_t;
+using uInt = uint64_t; // Max value = uInt_MAX used to denote 'non-existent'
+const uInt uInt_MAX = 18446744073709551615; // 2^64-1
+const uInt Int_MAX = 9223372036854775807;   // 2^63-1
+
+// Point<dimension> is a point in the d-dimensional integer lattice
+// (i.e. square-grid/cubic-grid, ...)
+template <uInt dimension> using Point = std::array<Int, dimension>;
+
+template <uInt dimension> Point<dimension> LongTensorToPoint(THLongTensor *t) {
+  Point<dimension> p;
+  long *td = THLongTensor_data(t);
+  for (int i = 0; i < dimension; i++)
+    p[i] = td[i];
+  return p;
+}
+template <uInt dimension>
+Point<2*dimension> TwoLongTensorsToPoint(THLongTensor *t0, THLongTensor *t1) {
+  Point<2 * dimension> p;
+  long *td;
+  td = THLongTensor_data(t0);
+  for (int i = 0; i < dimension; i++)
+    p[i] = td[i];
+  td = THLongTensor_data(t1);
+  for (int i = 0; i < dimension; i++)
+    p[i + dimension] = td[i];
+  return p;
+}
+template <uInt dimension>
+Point<3*dimension> ThreeLongTensorsToPoint(THLongTensor *t0, THLongTensor *t1,
+                                         THLongTensor *t2) {
+  Point<3 * dimension> p;
+  long *td;
+  td = THLongTensor_data(t0);
+  for (int i = 0; i < dimension; i++)
+    p[i] = td[i];
+  td = THLongTensor_data(t1);
+  for (int i = 0; i < dimension; i++)
+    p[i + dimension] = td[i];
+  td = THLongTensor_data(t2);
+  for (int i = 0; i < dimension; i++)
+    p[i + 2 * dimension] = td[i];
+  return p;
+}
+
+// FNV Hash function for Point<dimension>
+template <uInt dimension> struct IntArrayHash {
+  std::size_t operator()(Point<dimension> const &p) const {
+    uInt hash = 14695981039346656037;
+    for (auto x : p) {
+      hash *= 1099511628211;
+      hash ^= x;
+    }
+    return hash;
+  }
+};
+
+#define THCITensor THCudaLongTensor
+#define THCITensor_nElement THCudaLongTensor_nElement
+#define THCITensor_resize1d THCudaLongTensor_resize1d
+#define THCITensor_data THCudaLongTensor_data

PyTorch/sparseconvnet/SCN/generic/CPU/ActivePooling.cpp

+// Copyright 2016-present, Facebook, Inc.
+// All rights reserved.
+//
+// This source code is licensed under the license found in the
+// LICENSE file in the root directory of this source tree.
+
+#ifndef TH_GENERIC_FILE_
+#define TH_GENERIC_FILE_ "generic/CPU/ActivePooling.cpp"
+#else
+#include "ActivePooling.h"
+
+extern "C" void scn_DR_(ActivePooling_updateOutput)(
+    THLongTensor *inputSize, void **m, THTensor *input_features,
+    THTensor *output_features, void *rulesBuffer, bool average) {
+
+  SCN_INITIALIZE_AND_REFERENCE(Metadata<Dimension>, m)
+  uInt nPlanes = input_features->size[1];
+  auto _rules = _m.getActivePoolingRuleBook(inputSize);
+  uInt batchSize = _rules[1][0];
+  uInt maxActive = _rules[1][1];
+  THTensor_(resize2d)(output_features, batchSize, nPlanes);
+  THTensor_(zero)(output_features);
+
+  ActivePooling_ForwardPass<real>(THTensor_(data)(input_features),
+                                  THTensor_(data)(output_features), batchSize,
+                                  maxActive, nPlanes, _rules, average);
+}
+extern "C" void scn_DR_(ActivePooling_updateGradInput)(
+    THLongTensor *inputSize, void **m, THTensor *input_features,
+    THTensor *d_input_features, THTensor *d_output_features, void *rulesBuffer,
+    bool average) {
+
+  SCN_INITIALIZE_AND_REFERENCE(Metadata<Dimension>, m)
+  uInt nPlanes = input_features->size[1];
+  auto _rules = _m.getActivePoolingRuleBook(inputSize);
+  uInt batchSize = _rules[1][0];
+  uInt maxActive = _rules[1][1];
+  THTensor_(resizeAs)(d_input_features, input_features);
+  THTensor_(zero)(d_input_features);
+
+  ActivePooling_BackwardPass<real>(
+      THTensor_(data)(d_input_features), THTensor_(data)(d_output_features),
+      batchSize, maxActive, nPlanes, _rules, average);
+}
+#endif

PyTorch/sparseconvnet/SCN/generic/CPU/ActivePooling.h

+// Copyright 2016-present, Facebook, Inc.
+// All rights reserved.
+//
+// This source code is licensed under the license found in the
+// LICENSE file in the root directory of this source tree.
+
+#ifndef CPU_ACTIVEPOOLING_H
+#define CPU_ACTIVEPOOLING_H
+
+// Assume output_features and d_input_features have been zero-ed
+
+template <typename T>
+void ActivePooling_ForwardPass(T *input_features, T *output_features,
+                               uInt batchSize, uInt maxActive, uInt nPlanes,
+                               RuleBook &rules, bool average) {
+  for (uInt outSite = 0; outSite < batchSize; outSite++) {
+    T *out = &output_features[outSite * nPlanes];
+    uInt *r = &rules[0][outSite * (maxActive + 1)];
+    uInt nActive = *r++;
+    T multiplier = (average and nActive > 0) ? 1.0f / nActive : 1.0f;
+    while (nActive-- > 0) {
+      T *inp = &input_features[(*r++) * nPlanes];
+      for (uInt plane = 0; plane < nPlanes; plane++)
+        out[plane] += inp[plane] * multiplier;
+    }
+  }
+}
+template <typename T>
+void ActivePooling_BackwardPass(T *d_input_features, T *d_output_features,
+                                uInt batchSize, uInt maxActive, uInt nPlanes,
+                                RuleBook &rules, bool average) {
+  for (uInt outSite = 0; outSite < batchSize; outSite++) {
+    T *out = &d_output_features[outSite * nPlanes];
+    uInt *r = &rules[0][outSite * (maxActive + 1)];
+    uInt nActive = *r++;
+    T multiplier = (average and nActive > 0) ? 1.0f / nActive : 1.0f;
+    while (nActive-- > 0) {
+      T *inp = &d_input_features[(*r++) * nPlanes];
+      for (uInt plane = 0; plane < nPlanes; plane++)
+        inp[plane] = out[plane] * multiplier;
+    }
+  }
+}
+#endif /* CPU_ACTIVEPOOLING_H */

PyTorch/sparseconvnet/SCN/generic/CPU/AffineReluTrivialConvolution.cpp

+// Copyright 2016-present, Facebook, Inc.
+// All rights reserved.
+//
+// This source code is licensed under the license found in the
+// LICENSE file in the root directory of this source tree.
+
+#ifndef TH_GENERIC_FILE
+#define TH_GENERIC_FILE "generic/CPU/AffineReluTrivialConvolution.cpp"
+#else
+#include "AffineReluTrivialConvolution.h"
+
+extern "C" void scn_R_(AffineReluTrivialConvolution_updateOutput)(
+    THTensor *input_features, THTensor *output_features, THTensor *affineWeight,
+    THTensor *affineBias, THTensor *convWeight) {
+  THTensor_(resize2d)(output_features, input_features->size[0],
+                      convWeight->size[1]);
+  AffineReluTrivialConvolution_ForwardPass(
+      THTensor_(data)(input_features), convWeight->size[0],
+      input_features->stride[0], THTensor_(data)(output_features),
+      convWeight->size[1], output_features->stride[0],
+      THTensor_(data)(affineWeight), THTensor_(data)(affineBias),
+      THTensor_(data)(convWeight), input_features->size[0]);
+}
+
+extern "C" void scn_R_(AffineReluTrivialConvolution_backward)(
+    THTensor *input_features, THTensor *d_input_features,
+    THTensor *d_output_features, THTensor *affineWeight,
+    THTensor *d_affineWeight, THTensor *affineBias, THTensor *d_affineBias,
+    THTensor *convWeight, THTensor *d_convWeight, bool additiveGrad) {
+
+  THTensor_(resizeAs)(d_input_features, input_features);
+  AffineReluTrivialConvolution_BackwardPass(
+      THTensor_(data)(input_features), THTensor_(data)(d_input_features),
+      convWeight->size[0], input_features->stride[0],
+      THTensor_(data)(d_output_features), convWeight->size[1],
+      d_output_features->stride[0], THTensor_(data)(affineWeight),
+      THTensor_(data)(d_affineWeight), THTensor_(data)(affineBias),
+      THTensor_(data)(d_affineBias), THTensor_(data)(convWeight),
+      THTensor_(data)(d_convWeight), input_features->size[0], additiveGrad);
+}
+
+#endif

PyTorch/sparseconvnet/SCN/generic/CPU/AffineReluTrivialConvolution.h

+// Copyright 2016-present, Facebook, Inc.
+// All rights reserved.
+//
+// This source code is licensed under the license found in the
+// LICENSE file in the root directory of this source tree.
+
+#ifndef CPU_AffineReluTrivialConvolution_H
+#define CPU_AffineReluTrivialConvolution_H
+#include "../SparseConvNet.h"
+#include <cstring>
+// buffer must have size >= nHot * (nIn+nOut)
+
+template <typename T>
+void AffineReluTrivialConvolution_ForwardPass(
+    T *input_features, uInt input_nPlanes, uInt input_stride,
+    T *output_features, uInt output_nPlanes, uInt output_stride,
+    T *affineWeight, T *affineBias, T *convWeight, uInt nActive) {
+
+  for (uInt row = 0; row < nActive; row++) {
+    for (uInt column = 0; column < output_nPlanes; column++) {
+      T sum = 0;
+      for (uInt j = 0; j < input_nPlanes; j++) {
+        T i = input_features[row * input_stride + j] * affineWeight[j] +
+              affineBias[j];
+        i = (i > 0) ? i : 0;
+        sum += i * convWeight[j * output_nPlanes + column];
+      }
+      output_features[row * output_stride + column] = sum;
+    }
+  }
+}
+
+template <typename T>
+void AffineReluTrivialConvolution_BackwardPass(
+    T *input_features, T *d_input_features, uInt input_nPlanes,
+    uInt input_stride, T *d_output_features, uInt output_nPlanes,
+    uInt output_stride, T *affineWeight, T *dAffineWeight, T *affineBias,
+    T *dAffineBias, T *convWeight, T *dConvWeight, uInt nActive,
+    bool additiveGrad) {
+
+  for (uInt row = 0; row < input_nPlanes; row++) {
+    for (uInt column = 0; column < output_nPlanes; column++) {
+      T sum = 0;
+      for (uInt j = 0; j < nActive; j++) {
+        T i = input_features[j * input_stride + row] * affineWeight[row] +
+              affineBias[row];
+        i = (i > 0) ? i : 0;
+        sum += i * d_output_features[j * output_stride + column];
+      }
+      dConvWeight[row * output_nPlanes + column] += sum;
+    }
+  }
+  for (uInt row = 0; row < nActive; row++) {
+    for (uInt column = 0; column < input_nPlanes; column++) {
+      T sum = 0;
+      for (uInt j = 0; j < output_nPlanes; j++) {
+        sum += d_output_features[row * output_stride + j] *
+               convWeight[column * output_nPlanes + j];
+      }
+      T i = input_features[row * input_stride + column] * affineWeight[column] +
+            affineBias[column];
+      if (i <= 0) // d_ReLU
+        sum = 0;
+      dAffineWeight[column] += sum * i;
+      dAffineBias[column] += sum;
+      sum *= affineWeight[column];
+      if (additiveGrad)
+        d_input_features[row * input_stride + column] += sum;
+      else
+        d_input_features[row * input_stride + column] = sum;
+    }
+  }
+}
+#endif /* CPU_AffineReluTrivialConvolution_H */

PyTorch/sparseconvnet/SCN/generic/CPU/AveragePooling.cpp

+// Copyright 2016-present, Facebook, Inc.
+// All rights reserved.
+//
+// This source code is licensed under the license found in the
+// LICENSE file in the root directory of this source tree.
+
+#ifndef TH_GENERIC_FILE_
+#define TH_GENERIC_FILE_ "generic/CPU/AveragePooling.cpp"
+#else
+#include "AveragePooling.h"
+
+extern "C" void scn_DR_(AveragePooling_updateOutput)(
+    THLongTensor *inputSize, THLongTensor *outputSize, THLongTensor *poolSize,
+    THLongTensor *poolStride, void **m, THTensor *input_features,
+    THTensor *output_features, long nFeaturesToDrop, void *rulesBuffer) {
+
+  SCN_INITIALIZE_AND_REFERENCE(Metadata<Dimension>, m)
+  uInt nPlanes = input_features->size[1] - nFeaturesToDrop;
+  auto _rules =
+      _m.getRuleBook(inputSize, outputSize, poolSize, poolStride, true);
+  uInt nActive = _m.getNActive(outputSize);
+  THTensor_(resize2d)(output_features, nActive,
+                      input_features->size[1] - nFeaturesToDrop);
+  THTensor_(zero)(output_features);
+
+  auto iF = THTensor_(data)(input_features) + nFeaturesToDrop;
+  auto oF = THTensor_(data)(output_features);
+
+  for (auto &r : _rules) {
+    uInt nHot = r.size() / 2;
+    AveragePooling_ForwardPass<real>(iF, oF, nPlanes, input_features->stride[0],
+                                     output_features->stride[0], &r[0], nHot,
+                                     _rules.size());
+  }
+}
+extern "C" void scn_DR_(AveragePooling_updateGradInput)(
+    THLongTensor *inputSize, THLongTensor *outputSize, THLongTensor *poolSize,
+    THLongTensor *poolStride, void **m, THTensor *input_features,
+    THTensor *d_input_features, THTensor *d_output_features,
+    long nFeaturesToDrop, void *rulesBuffer) {
+
+  SCN_INITIALIZE_AND_REFERENCE(Metadata<Dimension>, m)
+  uInt nPlanes = input_features->size[1] - nFeaturesToDrop;
+  auto _rules =
+      _m.getRuleBook(inputSize, outputSize, poolSize, poolStride, true);
+  uInt nActive = _m.getNActive(outputSize);
+  THTensor_(resizeAs)(d_input_features, input_features);
+  THTensor_(zero)(d_input_features);
+
+  auto diF = THTensor_(data)(d_input_features) + nFeaturesToDrop;
+  auto doF = THTensor_(data)(d_output_features);
+
+  for (auto &r : _rules) {
+    uInt nHot = r.size() / 2;
+    AveragePooling_BackwardPass<real>(
+        diF, doF, nPlanes, input_features->stride[0],
+        d_output_features->stride[0], &r[0], nHot, _rules.size());
+  }
+}
+#endif

PyTorch/sparseconvnet/SCN/generic/CPU/AveragePooling.h

+// Copyright 2016-present, Facebook, Inc.
+// All rights reserved.
+//
+// This source code is licensed under the license found in the
+// LICENSE file in the root directory of this source tree.
+
+#ifndef CPU_AVERAGEPOOLING_H
+#define CPU_AVERAGEPOOLING_H
+#include "../SparseConvNet.h"
+
+template <typename T>
+void AveragePooling_ForwardPass(T *input_features, T *output_features,
+                                uInt nPlanes, uInt input_stride,
+                                uInt output_stride, uInt *rules, uInt nHot,
+                                uInt filterVolume) {
+  for (uInt outSite = 0; outSite < nHot; outSite++) {
+    uInt i = rules[2 * outSite] * input_stride;
+    uInt o = rules[2 * outSite + 1] * output_stride;
+    for (uInt plane = 0; plane < nPlanes; plane++)
+      output_features[o + plane] += input_features[i + plane] / filterVolume;
+  }
+}
+template <typename T>
+void AveragePooling_BackwardPass(T *d_input_features, T *d_output_features,
+                                 uInt nPlanes, uInt input_stride,
+                                 uInt output_stride, uInt *rules, uInt nHot,
+                                 uInt filterVolume) {
+  for (uInt outSite = 0; outSite < nHot; outSite++) {
+    uInt i = rules[2 * outSite] * input_stride;
+    uInt o = rules[2 * outSite + 1] * output_stride;
+    for (uInt plane = 0; plane < nPlanes; plane++)
+      d_input_features[i + plane] +=
+          d_output_features[o + plane] / filterVolume;
+  }
+}
+#endif /* CPU_AVERAGEPOOLING_H */

PyTorch/sparseconvnet/SCN/generic/CPU/BatchNormalization.cpp

+// Copyright 2016-present, Facebook, Inc.
+// All rights reserved.
+//
+// This source code is licensed under the license found in the
+// LICENSE file in the root directory of this source tree.
+
+#ifndef TH_GENERIC_FILE
+#define TH_GENERIC_FILE "generic/CPU/BatchNormalization.cpp"
+#else
+#include "BatchNormalization.h"
+
+extern "C" void scn_R_(BatchNormalization_updateOutput)(
+    THTensor *input_features, THTensor *output_features, THTensor *saveMean,
+    THTensor *saveInvStd, THTensor *runningMean, THTensor *runningVar,
+    THTensor *weight, THTensor *bias, real eps, real momentum, bool train,
+    real leakiness) {
+
+  THTensor_(resizeAs)(output_features, input_features);
+  auto nActive = input_features->size[0];
+  auto nPlanes = input_features->size[1];
+  auto input_stride = input_features->stride[0];
+  auto output_stride = output_features->stride[0];
+  BatchNormalization_ForwardPass<real>(
+      THTensor_(data)(input_features), THTensor_(data)(output_features),
+      nPlanes, input_stride, output_stride, nActive, THTensor_(data)(saveMean),
+      THTensor_(data)(saveInvStd), THTensor_(data)(runningMean),
+      THTensor_(data)(runningVar), THOptionalTensorData(weight),
+      THOptionalTensorData(bias), eps, momentum, train, leakiness);
+}
+
+extern "C" void scn_R_(BatchNormalizationInTensor_updateOutput)(
+    THTensor *input_features, THTensor *output_features, THTensor *saveMean,
+    THTensor *saveInvStd, THTensor *runningMean, THTensor *runningVar,
+    THTensor *weight, THTensor *bias, real eps, real momentum, bool train,
+    real leakiness) {
+
+  auto nActive = input_features->size[0];
+  auto nPlanes = input_features->size[1];
+  auto input_stride = input_features->stride[0];
+  auto output_stride = output_features->stride[0];
+
+  BatchNormalization_ForwardPass<real>(
+      THTensor_(data)(input_features), THTensor_(data)(output_features),
+      nPlanes, input_stride, output_stride, nActive, THTensor_(data)(saveMean),
+      THTensor_(data)(saveInvStd), THTensor_(data)(runningMean),
+      THTensor_(data)(runningVar), THOptionalTensorData(weight),
+      THOptionalTensorData(bias), eps, momentum, train, leakiness);
+}
+
+extern "C" void scn_R_(BatchNormalization_backward)(
+    THTensor *input_features, THTensor *d_input_features,
+    THTensor *output_features, THTensor *d_output_features, THTensor *saveMean,
+    THTensor *saveInvStd, THTensor *runningMean, THTensor *runningVar,
+    THTensor *weight, THTensor *bias, THTensor *d_weight, THTensor *d_bias,
+    real leakiness) {
+
+  THTensor_(resizeAs)(d_input_features, input_features);
+  auto nActive = input_features->size[0];
+  auto nPlanes = input_features->size[1];
+  auto input_stride = input_features->stride[0];
+  auto output_stride = output_features->stride[0];
+  BatchNormalization_BackwardPass<real>(
+      THTensor_(data)(input_features), THTensor_(data)(d_input_features),
+      THTensor_(data)(output_features), THTensor_(data)(d_output_features),
+      nPlanes, input_stride, output_stride, nActive, THTensor_(data)(saveMean),
+      THTensor_(data)(saveInvStd), THTensor_(data)(runningMean),
+      THTensor_(data)(runningVar), THOptionalTensorData(weight),
+      THOptionalTensorData(bias), THOptionalTensorData(d_weight),
+      THOptionalTensorData(d_bias), leakiness);
+}
+#endif

PyTorch/sparseconvnet/SCN/generic/CPU/BatchNormalization.h

+// Copyright 2016-present, Facebook, Inc.
+// All rights reserved.
+//
+// This source code is licensed under the license found in the
+// LICENSE file in the root directory of this source tree.
+
+#ifndef CPU_BATCHNORMALIZATION_H
+#define CPU_BATCHNORMALIZATION_H
+#include "../SparseConvNet.h"
+#include <vector>
+
+// in/output_stride is normally the same as nPlanes; allow other values to act
+// on a subset of columns, i.e. an inplace DenseNet blocks
+
+template <typename T>
+void BatchNormalization_ForwardPass(T *input_features, T *output_features,
+                                    uInt nPlanes, uInt input_stride,
+                                    uInt output_stride, uInt nActive,
+                                    T *saveMean, T *saveInvStd, T *runningMean,
+                                    T *runningVar, T *weight, T *bias, T eps,
+                                    T momentum, bool train, T leakiness) {
+  if (train) {
+    std::memset(saveMean, 0, nPlanes * sizeof(T));
+    std::memset(saveInvStd, 0, nPlanes * sizeof(T));
+    for (uInt row = 0, ci = 0; row < nActive;
+         row++, ci += input_stride - nPlanes) {
+      for (uInt plane = 0; plane < nPlanes; plane++, ci++) {
+        saveMean[plane] += input_features[ci];
+      }
+    }
+    for (uInt plane = 0; plane < nPlanes; plane++) {
+      saveMean[plane] /= nActive;
+      runningMean[plane] =
+          momentum * runningMean[plane] + (1 - momentum) * saveMean[plane];
+    }
+    for (uInt row = 0, ci = 0; row < nActive;
+         row++, ci += input_stride - nPlanes) {
+      for (uInt plane = 0; plane < nPlanes; plane++, ci++) {
+        saveInvStd[plane] +=
+            (input_features[ci] - saveMean[plane]) *
+            (input_features[ci] - saveMean[plane]); // accumulate sum-squares
+        // before inverse square
+        // rooting
+      }
+    }
+    for (uInt plane = 0; plane < nPlanes; plane++) {
+      runningVar[plane] = momentum * runningVar[plane] +
+                          (1 - momentum) * saveInvStd[plane] / (nActive - 1);
+      saveInvStd[plane] = powf(saveInvStd[plane] / nActive + eps, -0.5);
+    }
+  } else {
+    for (uInt plane = 0; plane < nPlanes; plane++) {
+      saveMean[plane] = runningMean[plane];
+      saveInvStd[plane] = powf(runningVar[plane] + eps, -0.5);
+    }
+  }
+  std::vector<T> w(nPlanes);
+  std::vector<T> b(nPlanes);
+  for (uInt plane = 0; plane < nPlanes; plane++) {
+    w[plane] = saveInvStd[plane] * (weight ? weight[plane] : 1);
+    b[plane] = -saveMean[plane] * w[plane] + (bias ? bias[plane] : 0);
+  }
+  for (uInt row = 0, ci = 0, co = 0; row < nActive;
+       row++, ci += input_stride - nPlanes, co += output_stride - nPlanes) {
+    for (uInt plane = 0; plane < nPlanes; plane++, ci++, co++) {
+      T out = input_features[ci] * w[plane] + b[plane];
+      out = (out > 0) ? out : (out * leakiness);
+      output_features[co] = out;
+    }
+  }
+}
+
+template <typename T>
+void BatchNormalization_BackwardPass(T *input_features, T *d_input_features,
+                                     T *output_features, T *d_output_features,
+                                     uInt nPlanes, uInt input_stride,
+                                     uInt output_stride, uInt nActive,
+                                     T *saveMean, T *saveInvStd, T *runningMean,
+                                     T *runningVar, T *weight, T *bias,
+                                     T *d_weight, T *d_bias, T leakiness) {
+  std::vector<T> gradMean(nPlanes);
+  std::vector<T> dotp(nPlanes);
+  std::vector<T> k(nPlanes);
+  for (uInt row = 0, ci = 0, co = 0; row < nActive;
+       row++, ci += input_stride - nPlanes, co += output_stride - nPlanes) {
+    for (uInt plane = 0; plane < nPlanes; plane++, ci++, co++) {
+      T d = d_output_features[co];
+      d = (output_features[co] > 0) ? d : (d * leakiness);
+      d_output_features[co] = d;
+      gradMean[plane] += d;
+      dotp[plane] += (input_features[ci] - saveMean[plane]) * d;
+    }
+  }
+  for (uInt plane = 0; plane < nPlanes; plane++) {
+    if (d_bias)
+      d_bias[plane] = gradMean[plane]; // sum of grads, really, until ...
+    gradMean[plane] /= nActive;        // ...now
+    k[plane] = dotp[plane] * saveInvStd[plane] * saveInvStd[plane] / nActive;
+  }
+  for (uInt row = 0, ci = 0, co = 0; row < nActive;
+       row++, ci += input_stride - nPlanes, co += output_stride - nPlanes) {
+    for (uInt plane = 0; plane < nPlanes; plane++, ci++, co++) {
+      d_input_features[ci] =
+          (d_output_features[co] - gradMean[plane] -
+           (input_features[ci] - saveMean[plane]) * k[plane]) *
+          saveInvStd[plane] * (weight ? weight[plane] : 1);
+    }
+  }
+  if (d_weight)
+    for (uInt plane = 0; plane < nPlanes; plane++) {
+      d_weight[plane] = dotp[plane] * saveInvStd[plane];
+    }
+}
+#endif /* CPU_BATCHNORMALIZATION_H */

PyTorch/sparseconvnet/SCN/generic/CPU/BatchwiseMultiplicativeDropout.cpp

+// Copyright 2016-present, Facebook, Inc.
+// All rights reserved.
+//
+// This source code is licensed under the license found in the
+// LICENSE file in the root directory of this source tree.
+
+#ifndef TH_GENERIC_FILE
+#define TH_GENERIC_FILE "generic/CPU/BatchwiseMultiplicativeDropout.cpp"
+#else
+
+extern "C" void scn_R_(BatchwiseMultiplicativeDropout_updateOutput)(
+    THTensor *input_features, THTensor *output_features, THTensor *noise,
+    float alpha) {
+  if (input_features != output_features)
+    THTensor_(resizeAs)(output_features, input_features);
+  auto nActive = input_features->size[0];
+  auto nPlanes = input_features->size[1];
+  auto iF = THTensor_(data)(input_features);
+  auto oF = THTensor_(data)(output_features);
+  auto nz = THTensor_(data)(noise);
+  for (uInt row = 0; row < nActive; row++)
+    for (uInt plane = 0, o = row * nPlanes, i = row * nPlanes; plane < nPlanes;
+         plane++, o++, i++)
+      oF[o] = (iF[i] > 0) ? iF[i] * nz[plane] : iF[i] * nz[plane] * alpha;
+}
+extern "C" void scn_R_(BatchwiseMultiplicativeDropout_updateGradInput)(
+    THTensor *input_features, THTensor *d_input_features,
+    THTensor *d_output_features, THTensor *noise, float alpha) {
+  if (d_input_features != d_output_features)
+    THTensor_(resizeAs)(d_input_features, d_output_features);
+  auto nActive = input_features->size[0];
+  auto nPlanes = input_features->size[1];
+  auto iF = THTensor_(data)(input_features);
+  auto diF = THTensor_(data)(d_input_features);
+  auto doF = THTensor_(data)(d_output_features);
+  auto nz = THTensor_(data)(noise);
+  for (uInt row = 0; row < nActive; row++)
+    for (uInt plane = 0, o = row * nPlanes, i = row * nPlanes; plane < nPlanes;
+         plane++, o++, i++)
+      diF[i] = (iF[i] > 0) ? doF[o] * nz[plane] : doF[o] * nz[plane] * alpha;
+}
+#endif

PyTorch/sparseconvnet/SCN/generic/CPU/Convolution.cpp

+// Copyright 2016-present, Facebook, Inc.
+// All rights reserved.
+//
+// This source code is licensed under the license found in the
+// LICENSE file in the root directory of this source tree.
+
+#ifndef TH_GENERIC_FILE_
+#define TH_GENERIC_FILE_ "generic/CPU/Convolution.cpp"
+#else
+#include "Convolution.h"
+
+extern "C" double scn_DR_(Convolution_updateOutput)(
+    THLongTensor *inputSize, THLongTensor *outputSize, THLongTensor *filterSize,
+    THLongTensor *filterStride, void **m, THTensor *input_features,
+    THTensor *output_features, THTensor *weight, THTensor *bias,
+    long filterVolume, void *rulesBuffer) {
+
+  SCN_INITIALIZE_AND_REFERENCE(Metadata<Dimension>, m)
+  auto _rules =
+      _m.getRuleBook(inputSize, outputSize, filterSize, filterStride, true);
+  uInt nActive = _m.getNActive(outputSize);
+  THTensor_(resize2d)(output_features, nActive, weight->size[1]);
+  if (not bias)
+    THTensor_(zero)(output_features);
+
+  auto iF = THTensor_(data)(input_features);
+  auto oF = THTensor_(data)(output_features);
+  auto ip = input_features->size[1];
+  auto op = output_features->size[1];
+  auto w = THTensor_(data)(weight);
+  auto b = THOptionalTensorData(bias);
+  Convolution_ForwardPass(iF, ip, ip, oF, op, op, w, b, _rules, nActive,
+                          THBlas_(gemm));
+  double flops = 0;
+  for (auto &r : _rules)
+    flops += r.size() / 2 * ip * op;
+  return flops;
+}
+
+extern "C" void scn_DR_(Convolution_backward)(
+    THLongTensor *inputSize, THLongTensor *outputSize, THLongTensor *filterSize,
+    THLongTensor *filterStride, void **m, THTensor *input_features,
+    THTensor *d_input_features, THTensor *d_output_features, THTensor *weight,
+    THTensor *d_weight, THTensor *d_bias, long filterVolume,
+    void *rulesBuffer) {
+
+  SCN_INITIALIZE_AND_REFERENCE(Metadata<Dimension>, m)
+  auto _rules =
+      _m.getRuleBook(inputSize, outputSize, filterSize, filterStride, true);
+  uInt nActive = _m.getNActive(outputSize);
+  THTensor_(resizeAs)(d_input_features, input_features);
+  THTensor_(zero)(d_input_features);
+
+  auto iF = THTensor_(data)(input_features);
+  auto diF = THTensor_(data)(d_input_features);
+  auto doF = THTensor_(data)(d_output_features);
+  auto ip = input_features->size[1];
+  auto op = d_output_features->size[1];
+  auto w = THTensor_(data)(weight);
+  auto dw = THTensor_(data)(d_weight);
+  auto db = THOptionalTensorData(d_bias);
+
+  Convolution_BackwardPass(iF, diF, ip, ip, doF, op, op, w, dw, db, _rules,
+                           nActive, THBlas_(gemm));
+}
+
+extern "C" double scn_DR_(ValidConvolution_updateOutput)(
+    THLongTensor *inputSize, THLongTensor *filterSize, void **m,
+    THTensor *input_features, THTensor *output_features, THTensor *weight,
+    THTensor *bias, long filterVolume, void *rulesBuffer) {
+
+  SCN_INITIALIZE_AND_REFERENCE(Metadata<Dimension>, m)
+  auto _rules = _m.getValidRuleBook(inputSize, filterSize, true);
+  uInt nActive = input_features->size[0];
+  THTensor_(resize2d)(output_features, nActive, weight->size[1]);
+  if (not bias)
+    THTensor_(zero)(output_features);
+
+  auto iF = THTensor_(data)(input_features);
+  auto oF = THTensor_(data)(output_features);
+  auto ip = input_features->size[1];
+  auto op = output_features->size[1];
+  auto w = THTensor_(data)(weight);
+  auto b = THOptionalTensorData(bias);
+
+  Convolution_ForwardPass(iF, ip, ip, oF, op, op, w, b, _rules, nActive,
+                          THBlas_(gemm));
+  double flops = 0;
+  for (auto &r : _rules)
+    flops += r.size() / 2 * ip * op;
+
+  return flops;
+}
+
+extern "C" void scn_DR_(ValidConvolution_backward)(
+    THLongTensor *inputSize, THLongTensor *filterSize, void **m,
+    THTensor *input_features, THTensor *d_input_features,
+    THTensor *d_output_features, THTensor *weight, THTensor *d_weight,
+    THTensor *d_bias, long filterVolume, void *rulesBuffer) {
+
+  SCN_INITIALIZE_AND_REFERENCE(Metadata<Dimension>, m)
+  auto _rules = _m.getValidRuleBook(inputSize, filterSize, true);
+  uInt nActive = input_features->size[0];
+  THTensor_(resizeAs)(d_input_features, input_features);
+  THTensor_(zero)(d_input_features);
+
+  auto iF = THTensor_(data)(input_features);
+  auto diF = THTensor_(data)(d_input_features);
+  auto doF = THTensor_(data)(d_output_features);
+  auto ip = input_features->size[1];
+  auto op = d_output_features->size[1];
+  auto w = THTensor_(data)(weight);
+  auto dw = THTensor_(data)(d_weight);
+  auto db = THOptionalTensorData(d_bias);
+
+  Convolution_BackwardPass(iF, diF, ip, ip, doF, op, op, w, dw, db, _rules,
+                           nActive, THBlas_(gemm));
+}
+
+#endif

PyTorch/sparseconvnet/SCN/generic/CPU/Convolution.h

+// Copyright 2016-present, Facebook, Inc.
+// All rights reserved.
+//
+// This source code is licensed under the license found in the
+// LICENSE file in the root directory of this source tree.
+
+#ifndef CPU_CONVOLUTION_H
+#define CPU_CONVOLUTION_H
+#include "../SparseConvNet.h"
+#include <cstring>
+// buffer must have size >= nHot * (nIn+nOut)
+
+template <typename T>
+void Convolution_ForwardPass(
+    T *input_features, uInt input_nPlanes, uInt input_nPLANES, T *output_features,
+    uInt output_nPlanes, uInt output_nPLANES, T *weight, T *bias, RuleBook &rules,
+    uInt output_nActive,
+    void (*gemm)(char transa, char transb, long m, long n, long k, T alpha,
+                 T *a, long lda, T *b, long ldb, T beta, T *c, long ldc)) {
+
+  if (bias != nullptr) // Set bias
+    for (uInt row = 0; row < output_nActive; row++)
+      for (uInt column = 0; column < output_nPlanes; column++)
+        output_features[row * output_nPLANES + column] = bias[column];
+
+  std::vector<T> input_buffer, output_buffer;
+  for (auto &r : rules) {
+    uInt nHot = r.size() / 2;
+    input_buffer.resize(nHot * input_nPlanes);
+    output_buffer.resize(nHot * output_nPlanes);
+    for (uInt row = 0; row < nHot; row++)
+      std::memcpy(&input_buffer[row * input_nPlanes],
+                  input_features + r[2 * row] * input_nPLANES,
+                  sizeof(T) * input_nPlanes);
+    // Do GEMM (note: gemm assumes column-major matrices)
+    // input_buffer    is l*m (row-major)
+    // weight          is m*r (row-major)
+    // output_buffer   is l*r (row-major)
+    // buffer * weights -> output_buffers
+    (*gemm)('n', 'n',
+            output_nPlanes,                   // r
+            nHot,                             // l
+            input_nPlanes,                    // m
+            1,                                // alpha
+            weight, output_nPlanes,           // r
+            &input_buffer[0], input_nPlanes,  // m
+            0,                                // beta
+            &output_buffer[0], output_nPlanes // r
+            );
+    weight += input_nPlanes * output_nPlanes;
+    for (uInt row = 0; row < nHot; row++) {
+      T *b = &output_buffer[row * output_nPlanes];
+      T *o = &output_features[r[2 * row + 1] * output_nPLANES];
+      for (uInt k = 0; k < output_nPlanes; k++)
+        o[k] += b[k];
+    }
+  }
+}
+
+template <typename T>
+void Convolution_BackwardPass(
+    T *input_features, T *d_input_features, uInt input_nPlanes,uInt input_nPLANES,
+    T *d_output_features, uInt output_nPlanes,uInt output_nPLANES, T *weight, T *d_weight,
+    T *d_bias, RuleBook &rules, uInt output_nActive,
+    void (*gemm)(char transa, char transb, long m, long n, long k, T alpha,
+                 T *a, long lda, T *b, long ldb, T beta, T *c, long ldc)) {
+
+  if (d_bias)
+    for (uInt row = 0; row < output_nActive; row++)
+      for (uInt i = 0; i < output_nPlanes; i++)
+        d_bias[i] += d_output_features[row * output_nPLANES + i];
+
+  std::vector<T> input_buffer, output_buffer;
+  for (auto &r : rules) {
+    uInt nHot = r.size() / 2;
+    input_buffer.resize(nHot * input_nPlanes);
+    output_buffer.resize(nHot * output_nPlanes);
+    for (uInt row = 0; row < nHot; row++)
+      std::memcpy(&output_buffer[row * output_nPlanes],
+                  &d_output_features[r[2 * row + 1] * output_nPLANES],
+                  sizeof(T) * output_nPlanes);
+    // Do GEMM (note: gemm assumes column-major matrices)
+    // output_buffer is l*m (row-major)
+    // weights           is r*m (row-major)
+    // input_buffer          is l*r (row-major)
+    // output_buffer * T(weight) -> input_buffer
+    (*gemm)('t', 'n',
+            input_nPlanes,                     // r
+            nHot,                              // l
+            output_nPlanes,                    // m
+            1,                                 // alpha
+            weight, output_nPlanes,            // m
+            &output_buffer[0], output_nPlanes, // m
+            0,                                 // beta
+            &input_buffer[0], input_nPlanes    // r
+            );
+    weight += input_nPlanes * output_nPlanes;
+    for (uInt row = 0; row < nHot; row++) {
+      T *b = &input_buffer[row * input_nPlanes];
+      T *i = &d_input_features[r[2 * row] * input_nPLANES];
+      for (uInt k = 0; k < input_nPlanes; k++)
+        i[k] += b[k];
+    }
+
+    for (uInt row = 0; row < nHot; row++)
+      std::memcpy(&input_buffer[row * input_nPlanes],
+                  input_features + r[2 * row] * input_nPLANES,
+                  sizeof(T) * input_nPlanes);
+    // Do GEMM (note: gemm assumes column-major matrices)
+    // input_buffer          is m*l (row-major)
+    // output_buffer          is m*r   (row-major)
+    // d_weights        is l*r (row-major)
+    // T(input_buffer) * output_buffer -> d_weight
+    (*gemm)('n', 't',
+            output_nPlanes,                    // r
+            input_nPlanes,                     // l
+            nHot,                              // m
+            1,                                 // alpha
+            &output_buffer[0], output_nPlanes, // r
+            &input_buffer[0], input_nPlanes,   // l
+            1,                                 // beta
+            d_weight, output_nPlanes           // r
+            );
+    d_weight += input_nPlanes * output_nPlanes;
+  }
+}
+#endif /* CPU_CONVOLUTION_H */

PyTorch/sparseconvnet/SCN/generic/CPU/Deconvolution.cpp

+// Copyright 2016-present, Facebook, Inc.
+// All rights reserved.
+//
+// This source code is licensed under the license found in the
+// LICENSE file in the root directory of this source tree.
+
+#ifndef TH_GENERIC_FILE_
+#define TH_GENERIC_FILE_ "generic/CPU/Deconvolution.cpp"
+#else
+#include "Deconvolution.h"
+
+extern "C" double scn_DR_(Deconvolution_updateOutput)(
+    THLongTensor *inputSize, THLongTensor *outputSize, THLongTensor *filterSize,
+    THLongTensor *filterStride, void **m, THTensor *input_features,
+    THTensor *output_features, THTensor *weight, THTensor *bias,
+    long filterVolume, void *rulesBuffer) {
+
+  SCN_INITIALIZE_AND_REFERENCE(Metadata<Dimension>, m)
+  auto _rules =
+      _m.getRuleBook(outputSize, inputSize, filterSize, filterStride, true);
+  uInt nActive = _m.getNActive(outputSize);
+  THTensor_(resize2d)(output_features, nActive, weight->size[1]);
+  if (not bias)
+    THTensor_(zero)(output_features);
+
+  auto iF = THTensor_(data)(input_features);
+  auto oF = THTensor_(data)(output_features);
+  auto ip = input_features->size[1];
+  auto op = output_features->size[1];
+  auto w = THTensor_(data)(weight);
+  auto b = THOptionalTensorData(bias);
+
+  Deconvolution_ForwardPass(iF, ip, ip, oF, op, op, w, b, _rules, nActive,
+                            THBlas_(gemm));
+  double flops = 0;
+  for (auto &r : _rules)
+    flops += r.size() / 2 * ip * op;
+  return flops;
+}
+
+extern "C" void scn_DR_(Deconvolution_backward)(
+    THLongTensor *inputSize, THLongTensor *outputSize, THLongTensor *filterSize,
+    THLongTensor *filterStride, void **m, THTensor *input_features,
+    THTensor *d_input_features, THTensor *d_output_features, THTensor *weight,
+    THTensor *d_weight, THTensor *d_bias, long filterVolume,
+    void *rulesBuffer) {
+
+  SCN_INITIALIZE_AND_REFERENCE(Metadata<Dimension>, m)
+  auto _rules =
+      _m.getRuleBook(outputSize, inputSize, filterSize, filterStride, true);
+  uInt nActive = _m.getNActive(outputSize);
+  THTensor_(resizeAs)(d_input_features, input_features);
+  THTensor_(zero)(d_input_features);
+
+  auto iF = THTensor_(data)(input_features);
+  auto diF = THTensor_(data)(d_input_features);
+  auto doF = THTensor_(data)(d_output_features);
+  auto ip = input_features->size[1];
+  auto op = d_output_features->size[1];
+  auto w = THTensor_(data)(weight);
+  auto dw = THTensor_(data)(d_weight);
+  auto db = THOptionalTensorData(d_bias);
+
+  Deconvolution_BackwardPass(iF, diF, ip, ip, doF, op, op, w, dw, db, _rules,
+                             nActive, THBlas_(gemm));
+}
+
+#endif

PyTorch/sparseconvnet/SCN/generic/CPU/Deconvolution.h

+// Copyright 2016-present, Facebook, Inc.
+// All rights reserved.
+//
+// This source code is licensed under the license found in the
+// LICENSE file in the root directory of this source tree.
+
+#ifndef CPU_Deconvolution_H
+#define CPU_Deconvolution_H
+#include "../SparseConvNet.h"
+#include <cstring>
+// buffer must have size >= nHot * (nIn+nOut)
+
+template <typename T>
+void Deconvolution_ForwardPass(
+    T *input_features, uInt input_nPlanes, uInt input_nPLANES,
+    T *output_features, uInt output_nPlanes, uInt output_nPLANES, T *weight,
+    T *bias, RuleBook &rules, uInt output_nActive,
+    void (*gemm)(char transa, char transb, long m, long n, long k, T alpha,
+                 T *a, long lda, T *b, long ldb, T beta, T *c, long ldc)) {
+
+  if (bias != nullptr) // Set bias
+    for (uInt row = 0; row < output_nActive; row++)
+      for (uInt column = 0; column < output_nPlanes; column++)
+        output_features[row * output_nPLANES + column] = bias[column];
+
+  std::vector<T> input_buffer, output_buffer;
+  for (auto &r : rules) {
+    uInt nHot = r.size() / 2;
+    input_buffer.resize(nHot * input_nPlanes);
+    output_buffer.resize(nHot * output_nPlanes);
+    for (uInt row = 0; row < nHot; row++)
+      std::memcpy(&input_buffer[row * input_nPlanes],
+                  input_features + r[2 * row + 1] * input_nPLANES,
+                  sizeof(T) * input_nPlanes);
+    // Do GEMM (note: gemm assumes column-major matrices)
+    // input_buffer    is l*m (row-major)
+    // weight          is m*r (row-major)
+    // output_buffer   is l*r (row-major)
+    // buffer * weights -> output_buffers
+    (*gemm)('n', 'n',
+            output_nPlanes,                   // r
+            nHot,                             // l
+            input_nPlanes,                    // m
+            1,                                // alpha
+            weight, output_nPlanes,           // r
+            &input_buffer[0], input_nPlanes,  // m
+            0,                                // beta
+            &output_buffer[0], output_nPlanes // r
+            );
+    weight += input_nPlanes * output_nPlanes;
+    for (uInt row = 0; row < nHot; row++) {
+      T *b = &output_buffer[row * output_nPlanes];
+      T *o = &output_features[r[2 * row] * output_nPLANES];
+      for (uInt k = 0; k < output_nPlanes; k++)
+        o[k] += b[k];
+    }
+  }
+}
+
+template <typename T>
+void Deconvolution_BackwardPass(
+    T *input_features, T *d_input_features, uInt input_nPlanes,
+    uInt input_nPLANES, T *d_output_features, uInt output_nPlanes,
+    uInt output_nPLANES, T *weight, T *d_weight, T *d_bias, RuleBook &rules,
+    uInt output_nActive,
+    void (*gemm)(char transa, char transb, long m, long n, long k, T alpha,
+                 T *a, long lda, T *b, long ldb, T beta, T *c, long ldc)) {
+
+  if (d_bias)
+    for (uInt row = 0; row < output_nActive; row++)
+      for (uInt i = 0; i < output_nPlanes; i++)
+        d_bias[i] += d_output_features[row * output_nPLANES + i];
+
+  std::vector<T> input_buffer, output_buffer;
+  for (auto &r : rules) {
+    uInt nHot = r.size() / 2;
+    input_buffer.resize(nHot * input_nPlanes);
+    output_buffer.resize(nHot * output_nPlanes);
+    for (uInt row = 0; row < nHot; row++)
+      std::memcpy(&output_buffer[row * output_nPlanes],
+                  &d_output_features[r[2 * row] * output_nPLANES],
+                  sizeof(T) * output_nPlanes);
+    // Do GEMM (note: gemm assumes column-major matrices)
+    // output_buffer is l*m (row-major)
+    // weights           is r*m (row-major)
+    // input_buffer          is l*r (row-major)
+    // output_buffer * T(weight) -> input_buffer
+    (*gemm)('t', 'n',
+            input_nPlanes,                     // r
+            nHot,                              // l
+            output_nPlanes,                    // m
+            1,                                 // alpha
+            weight, output_nPlanes,            // m
+            &output_buffer[0], output_nPlanes, // m
+            0,                                 // beta
+            &input_buffer[0], input_nPlanes    // r
+            );
+    weight += input_nPlanes * output_nPlanes;
+    for (uInt row = 0; row < nHot; row++) {
+      T *b = &input_buffer[row * input_nPlanes];
+      T *i = &d_input_features[r[2 * row + 1] * input_nPLANES];
+      for (uInt k = 0; k < input_nPlanes; k++)
+        i[k] += b[k];
+    }
+
+    for (uInt row = 0; row < nHot; row++)
+      std::memcpy(&input_buffer[row * input_nPlanes],
+                  input_features + r[2 * row + 1] * input_nPLANES,
+                  sizeof(T) * input_nPlanes);
+    // Do GEMM (note: gemm assumes column-major matrices)
+    // input_buffer          is m*l (row-major)
+    // output_buffer          is m*r   (row-major)
+    // d_weights        is l*r (row-major)
+    // T(input_buffer) * output_buffer -> d_weight
+    (*gemm)('n', 't',
+            output_nPlanes,                    // r
+            input_nPlanes,                     // l
+            nHot,                              // m
+            1,                                 // alpha
+            &output_buffer[0], output_nPlanes, // r
+            &input_buffer[0], input_nPlanes,   // l
+            1,                                 // beta
+            d_weight, output_nPlanes           // r
+            );
+    d_weight += input_nPlanes * output_nPlanes;
+  }
+}
+#endif /* CPU_Deconvolution_H */

PyTorch/sparseconvnet/SCN/generic/CPU/LeakyReLU.cpp

+// Copyright 2016-present, Facebook, Inc.
+// All rights reserved.
+//
+// This source code is licensed under the license found in the
+// LICENSE file in the root directory of this source tree.
+
+#ifndef TH_GENERIC_FILE
+#define TH_GENERIC_FILE "generic/CPU/LeakyReLU.cpp"
+#else
+
+extern "C" void scn_R_(LeakyReLU_updateOutput)(THTensor *input_features,
+                                               THTensor *output_features,
+                                               float alpha) {
+  if (input_features != output_features)
+    THTensor_(resizeAs)(output_features, input_features);
+  auto iF = THTensor_(data)(input_features);
+  auto oF = THTensor_(data)(output_features);
+  auto n = THTensor_(nElement)(input_features);
+
+  for (uInt i = 0; i < n; i++)
+    oF[i] = (iF[i] > 0) ? iF[i] : iF[i] * alpha;
+}
+extern "C" void scn_R_(LeakyReLU_updateGradInput)(THTensor *input_features,
+                                                  THTensor *d_input_features,
+                                                  THTensor *d_output_features,
+                                                  float alpha) {
+  if (d_input_features != d_output_features)
+    THTensor_(resizeAs)(d_input_features, d_output_features);
+  auto iF = THTensor_(data)(input_features);
+  auto diF = THTensor_(data)(d_input_features);
+  auto doF = THTensor_(data)(d_output_features);
+  auto n = THTensor_(nElement)(d_input_features);
+
+  for (uInt i = 0; i < n; i++)
+    diF[i] = (iF[i] > 0) ? doF[i] : doF[i] * alpha;
+}
+#endif