GitLab

* wip: layer normalization on cpu

* wip: add cuda implementation, nor working yet

* wip: try to fix cuda implementation

* swap grid_strid_range and grid_strid_range_y: does not work yet

* fix CUDA implementation

* implement cuda gradient

* add documentation, move layer_norm, update bn_visitor

* add tests

* use stddev instead of variance in test (they are both 1, anyway)

* add test for means and invstds on CPU and CUDA

* rename visitor to disable_duplicative_bias

* handle more cases in the visitor_disable_input_bias

* Add tests for visitor_disable_input_bias

* Add GELU activation layer

* fix some copy-paste leftovers

* fix comment

* use exact faster implementation

* do not use cmath constants

* add cuda test for loss_binary_log_per_pixel and some needed refactoring

* add cuda test for loss_multiclass_log_per_pixel

* forgot to add cpu version in loss

* remove a line I added by mistake

* fix typos

* declare label_to_ignore as static

* use tensor_index function instead of index method

* test cuda and cpu gradients values

* use DLIB_TEST instead of DLIB_CASSERT

* add cuda implementation for loss_multiclass_log_per_pixel_weighted

* add test for cuda and cpu implementations

* fix comment

* move weighted label to its own file

* Update path in doc
Co-authored-by: Davis E. King <davis685@gmail.com>

* wip: attempt to use cuda for loss mse channel

* wip: maybe this is a step in the right direction

* Try to fix dereferencing the truth data (#1)

* Try to fix dereferencing the truth data

* Fix memory layout

* fix loss scaling and update tests

* rename temp1 to temp

* readd lambda captures for output_width and output_height

clangd was complaining about this, and suggested me to remove them
in the first, place:

```
Lambda capture 'output_height' is not required to be captured for this use (fix available)
Lambda capture 'output_width' is not required to be captured for this use (fix available)
```

* add a weighted_loss typedef to loss_multiclass_log_weighted_ for consistency

* update docs for weighted losses

* refactor multi channel loss and add cpu-cuda tests

* make operator() const

* make error relative to the loss value
Co-authored-by: Juha Reunanen <juha.reunanen@tomaattinen.com>

* add leaky_relu activation layer

* add inplace case for leaky_relu and test_layer

* make clear that alpha is not learned by leaky_relu

* remove branch from cuda kernel

stride values.  This lets you run the tensor resizing routine on subwindows in
a tensor.

concat layer's backward() method.  It was assigning the gradient to previous
layers instead of adding the gradient, as required by the layer interface
specification.  This change also noticeably speeds up concat layers since only
one CUDA kernel launch now happens per concat operation, rather than one
kernel launch for each sample in a tensor.

reallocation and copying inside conv_'s backward pass.  Doing this
required adding an add_to_output boolean option to the methods of
tensor_conv.

refactored interface to reduce complexity so conv and convt layers forward passes have to call setup explicit now and there is only one ()-operator

rather than the entire tensor.

fixed backward pass in cont layer to accumulate gradients this will pass the layer test now also removed compile warnings and changed some comments

layers.  Updated the solvers to support this.

skip layers and add_prev style layers.  In particular, now in-place layers only
overwrite the gradient information in their child layer if they are operating
in in-place mode.  Otherwise, they add their gradients to their child layers.

It should also be noted that it's safe for in-place layers to overwrite
gradients when in in-place mode since their child layers are inaccessible when
in-place layers operate in in-place mode.  This prevents any other layers from
trying to add to the child layer, thereby avoiding the potability of layer
interference.  So the bug this change fixes is that, when not in in-place mode
the child layers are still accessible but in-place layers were *still*
overwriting child gradients.

to expose it in the final layer interface.

running variance output.

to automatically select the right mode.  The serialization format for bn_
layers has also changed, but the code will still be able to deserialize older
bn_ objects.

implementation of assign_conv_bias_gradient().

tensors with different sizes and it will zero pad them as needed.

since that's a little different in cuDNN.  I also removed my CUDA code for
doing batch normalization and replaced it with cuDNN's new batch normalization
methods.

Finally, I forgot to add a convolutional option to the bn_ object.  Now it has
one so you can set the mode however you like, either BATCH_NORM_FC or
BATCH_NORM_CONV.