This is a (Cython-based) Python wrapper for [Philipp Krähenbühl's Fully-Connected CRFs](http://web.archive.org/web/20161023180357/http://www.philkr.net/home/densecrf)(version 2, [new, incomplete page](http://www.philkr.net/2011/12/01/nips/)).
## 1 简介
If you use this code for your reasearch, please cite:
PyDenseCRF 是一个基于Python的库，它提供了对Philipp Krähenbühl的全连接条件随机场（Fully-Connected Conditional Random Fields, FCRF）的封装。这个工具最初用于提升图像语义分割任务的性能，尤其在处理细节和边缘检测上表现出色。
```
Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
Philipp Krähenbühl and Vladlen Koltun
NIPS 2011
```
and provide a link to this repository as a footnote or a citation.
Installation
============
The package is on PyPI, so simply run `pip install pydensecrf` to install it.
If you want the newest and freshest version, you can install it by executing:
Since this library needs to compile C++ code, installation can be a little more problematic than pure Python packages.
Make sure to [have Cython installed](https://github.com/lucasb-eyer/pydensecrf/issues/62#issuecomment-400563257) or try [installing via conda instead](https://github.com/lucasb-eyer/pydensecrf/issues/69#issuecomment-400639881) if you are getting problems.
The parameters map to those in the paper as follows: `sxy` in the `Gaussian` case is `$\theta_{\gamma}$`,
and in the `Bilateral` case, `sxy` and `srgb` map to `$\theta_{\alpha}$` and `$\theta_{\beta}$`, respectively.
The names are shorthand for "x/y standard-deviation" and "rgb standard-deviation" and for reference, the formula is:
### Non-RGB bilateral
An important caveat is that `addPairwiseBilateral` only works for RGB images, i.e. three channels.
If your data is of different type than this simple but common case, you'll need to compute your
own pairwise energy using `utils.create_pairwise_bilateral`; see the [generic non-2D case](https://github.com/lucasb-eyer/pydensecrf#generic-non-2d) for details.
A good [example of working with Non-RGB data](https://github.com/lucasb-eyer/pydensecrf/blob/master/examples/Non%20RGB%20Example.ipynb) is provided as a notebook in the examples folder.
### Compatibilities
The `compat` argument can be any of the following:
- A number, then a `PottsCompatibility` is being used.
- A 1D array, then a `DiagonalCompatibility` is being used.
- A 2D array, then a `MatrixCompatibility` is being used.
These are label-compatibilites `µ(xi, xj)` whose parameters could possibly be [learned](https://github.com/lucasb-eyer/pydensecrf#learning).
For example, they could indicate that mistaking `bird` pixels for `sky` is not as bad as mistaking `cat` for `sky`.
The arrays should have `nlabels` or `(nlabels,nlabels)` as shape and a `float32` datatype.
### Kernels
Possible values for the `kernel` argument are:
-`CONST_KERNEL`
-`DIAG_KERNEL` (the default)
-`FULL_KERNEL`
This specifies the kernel's precision-matrix `Λ(m)`, which could possibly be learned.
These indicate correlations between feature types, the default implying no correlation.
Again, this could possiblty be [learned](https://github.com/lucasb-eyer/pydensecrf#learning).
### Normalizations
Possible values for the `normalization` argument are:
-`NO_NORMALIZATION`
-`NORMALIZE_BEFORE`
-`NORMALIZE_AFTER`
-`NORMALIZE_SYMMETRIC` (the default)
### Kernel weight
I have so far not found a way to set the kernel weights `w(m)`.
According to the paper, `w(2)` was set to 1 and `w(1)` was cross-validated, but never specified.
Looking through Philip's code (included in [pydensecrf/densecrf](https://github.com/lucasb-eyer/pydensecrf/tree/master/pydensecrf/densecrf)),
I couldn't find such explicit weights, and my guess is they are thus hard-coded to 1.
If anyone knows otherwise, please open an issue or, better yet, a pull-request.
Update: user @waldol1 has an idea in [this issue](https://github.com/lucasb-eyer/pydensecrf/issues/37). Feel free to try it out!
Inference
---------
The easiest way to do inference with 5 iterations is to simply call:
```python
Q=d.inference(5)
```
And the MAP prediction is then:
```python
map=np.argmax(Q,axis=0).reshape((640,480))
```
If you're interested in the class-probabilities `Q`, you'll notice `Q` is a
wrapped Eigen matrix. The Eigen wrappers of this project implement the buffer
interface and can be simply cast to numpy arrays like so:
```python
proba=np.array(Q)
```
Step-by-step inference
----------------------
If for some reason you want to run the inference loop manually, you can do so:
```python
Q,tmp1,tmp2=d.startInference()
foriinrange(5):
print("KL-divergence at {}: {}".format(i,d.klDivergence(Q)))
d.stepInference(Q,tmp1,tmp2)
```
Generic non-2D
--------------
The `DenseCRF` class can be used for generic (non-2D) dense CRFs.
Its usage is exactly the same as above, except that the 2D-specific pairwise
potentials `addPairwiseGaussian` and `addPairwiseBilateral` are missing.
Instead, you need to use the generic `addPairwiseEnergy` method like this:
```python
d=dcrf.DenseCRF(100,5)# npoints, nlabels
feats=np.array(...)# Get the pairwise features from somewhere.
In addition, you can pass `compatibility`, `kernel` and `normalization`
arguments just like in the 2D gaussian and bilateral cases.
The potential will be computed as `w*exp(-0.5 * |f_i - f_j|^2)`.
### Pairwise potentials for N-D
User @markusnagel has written a couple numpy-functions generalizing the two
classic 2-D image pairwise potentials (gaussian and bilateral) to an arbitrary
number of dimensions: `create_pairwise_gaussian` and `create_pairwise_bilateral`.
You can access them as `from pydensecrf.utils import create_pairwise_gaussian`
and then have a look at their docstring to see how to use them.
Learning
--------
The learning has not been fully wrapped. If you need it, get in touch or better
yet, wrap it and submit a pull-request!
Here's a pointer for starters: issue#24. We need to wrap the gradients and getting/setting parameters.
But then, we also need to do something with these, most likely call [minimizeLBFGS from optimization.cpp](https://github.com/lucasb-eyer/pydensecrf/blob/d824b89ee3867bca3e90b9f04c448f1b41821524/pydensecrf/densecrf/src/optimization.cpp).
It should be relatively straightforward to just follow the learning examples included in the [original code](http://graphics.stanford.edu/projects/drf/densecrf_v_2_2.zip).
Common Problems
===============
`undefined symbol` when importing
---------------------------------
If while importing pydensecrf you get an error about some undefined symbols (for example `.../pydensecrf/densecrf.so: undefined symbol: _ZTINSt8ios_base7failureB5cxx11E`), you most likely are inadvertently mixing different compilers or toolchains. Try to see what's going on using tools like `ldd`. If you're using Anaconda, [running `conda install libgcc` might be a solution](https://github.com/lucasb-eyer/pydensecrf/issues/28).
ValueError: Buffer dtype mismatch, expected 'float' but got 'double'
This is a pretty [co](https://github.com/lucasb-eyer/pydensecrf/issues/52)mm[on](https://github.com/lucasb-eyer/pydensecrf/issues/49) user error.
It means exactly what it says: you are passing a `double` but it wants a `float`.
Solve it by, for example, calling `d.setUnaryEnergy(U.astype(np.float32))` instead of just `d.setUnaryEnergy(U)`, or using `float32` in your code in the first place.
My results are all pixelated like [MS Paint's airbrush tool](http://lmgtfy.com/?q=MS+Paint+Airbrush+tool)!
This is a (Cython-based) Python wrapper for [Philipp Krähenbühl's Fully-Connected CRFs](http://web.archive.org/web/20161023180357/http://www.philkr.net/home/densecrf)(version 2, [new, incomplete page](http://www.philkr.net/2011/12/01/nips/)).
If you use this code for your reasearch, please cite:
```
Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
Philipp Krähenbühl and Vladlen Koltun
NIPS 2011
```
and provide a link to this repository as a footnote or a citation.
Installation
============
The package is on PyPI, so simply run `pip install pydensecrf` to install it.
If you want the newest and freshest version, you can install it by executing:
Note that you need a relatively recent version of Cython (at least version 0.22) for this wrapper,
the one shipped with Ubuntu 14.04 is too old. (Thanks to Scott Wehrwein for pointing this out.)
I suggest you use a [virtual environment](https://virtualenv.readthedocs.org/en/latest/) and install
the newest version of Cython there (`pip install cython`), but you may update the system version by
```
sudo apt-get remove cython
sudo pip install -U cython
```
### Problems on Windows/VS
Since this library needs to compile C++ code, installation can be a little more problematic than pure Python packages.
Make sure to [have Cython installed](https://github.com/lucasb-eyer/pydensecrf/issues/62#issuecomment-400563257) or try [installing via conda instead](https://github.com/lucasb-eyer/pydensecrf/issues/69#issuecomment-400639881) if you are getting problems.
PRs that improve Windows support are welcome.
### Problems on Colab/Kaggle Kernel
`pydensecrf` does not come pre-installed in Colab or Kaggle Kernel. Running `pip install pydensecrf` will result into
build failures. Follow these steps instead for Colab/Kaggle Kernel:
The parameters map to those in the paper as follows: `sxy` in the `Gaussian` case is `$\theta_{\gamma}$`,
and in the `Bilateral` case, `sxy` and `srgb` map to `$\theta_{\alpha}$` and `$\theta_{\beta}$`, respectively.
The names are shorthand for "x/y standard-deviation" and "rgb standard-deviation" and for reference, the formula is:
### Non-RGB bilateral
An important caveat is that `addPairwiseBilateral` only works for RGB images, i.e. three channels.
If your data is of different type than this simple but common case, you'll need to compute your
own pairwise energy using `utils.create_pairwise_bilateral`; see the [generic non-2D case](https://github.com/lucasb-eyer/pydensecrf#generic-non-2d) for details.
A good [example of working with Non-RGB data](https://github.com/lucasb-eyer/pydensecrf/blob/master/examples/Non%20RGB%20Example.ipynb) is provided as a notebook in the examples folder.
### Compatibilities
The `compat` argument can be any of the following:
- A number, then a `PottsCompatibility` is being used.
- A 1D array, then a `DiagonalCompatibility` is being used.
- A 2D array, then a `MatrixCompatibility` is being used.
These are label-compatibilites `µ(xi, xj)` whose parameters could possibly be [learned](https://github.com/lucasb-eyer/pydensecrf#learning).
For example, they could indicate that mistaking `bird` pixels for `sky` is not as bad as mistaking `cat` for `sky`.
The arrays should have `nlabels` or `(nlabels,nlabels)` as shape and a `float32` datatype.
### Kernels
Possible values for the `kernel` argument are:
-`CONST_KERNEL`
-`DIAG_KERNEL` (the default)
-`FULL_KERNEL`
This specifies the kernel's precision-matrix `Λ(m)`, which could possibly be learned.
These indicate correlations between feature types, the default implying no correlation.
Again, this could possiblty be [learned](https://github.com/lucasb-eyer/pydensecrf#learning).
### Normalizations
Possible values for the `normalization` argument are:
-`NO_NORMALIZATION`
-`NORMALIZE_BEFORE`
-`NORMALIZE_AFTER`
-`NORMALIZE_SYMMETRIC` (the default)
### Kernel weight
I have so far not found a way to set the kernel weights `w(m)`.
According to the paper, `w(2)` was set to 1 and `w(1)` was cross-validated, but never specified.
Looking through Philip's code (included in [pydensecrf/densecrf](https://github.com/lucasb-eyer/pydensecrf/tree/master/pydensecrf/densecrf)),
I couldn't find such explicit weights, and my guess is they are thus hard-coded to 1.
If anyone knows otherwise, please open an issue or, better yet, a pull-request.
Update: user @waldol1 has an idea in [this issue](https://github.com/lucasb-eyer/pydensecrf/issues/37). Feel free to try it out!
Inference
---------
The easiest way to do inference with 5 iterations is to simply call:
```python
Q=d.inference(5)
```
And the MAP prediction is then:
```python
map=np.argmax(Q,axis=0).reshape((640,480))
```
If you're interested in the class-probabilities `Q`, you'll notice `Q` is a
wrapped Eigen matrix. The Eigen wrappers of this project implement the buffer
interface and can be simply cast to numpy arrays like so:
```python
proba=np.array(Q)
```
Step-by-step inference
----------------------
If for some reason you want to run the inference loop manually, you can do so:
```python
Q,tmp1,tmp2=d.startInference()
foriinrange(5):
print("KL-divergence at {}: {}".format(i,d.klDivergence(Q)))
d.stepInference(Q,tmp1,tmp2)
```
Generic non-2D
--------------
The `DenseCRF` class can be used for generic (non-2D) dense CRFs.
Its usage is exactly the same as above, except that the 2D-specific pairwise
potentials `addPairwiseGaussian` and `addPairwiseBilateral` are missing.
Instead, you need to use the generic `addPairwiseEnergy` method like this:
```python
d=dcrf.DenseCRF(100,5)# npoints, nlabels
feats=np.array(...)# Get the pairwise features from somewhere.
In addition, you can pass `compatibility`, `kernel` and `normalization`
arguments just like in the 2D gaussian and bilateral cases.
The potential will be computed as `w*exp(-0.5 * |f_i - f_j|^2)`.
### Pairwise potentials for N-D
User @markusnagel has written a couple numpy-functions generalizing the two
classic 2-D image pairwise potentials (gaussian and bilateral) to an arbitrary
number of dimensions: `create_pairwise_gaussian` and `create_pairwise_bilateral`.
You can access them as `from pydensecrf.utils import create_pairwise_gaussian`
and then have a look at their docstring to see how to use them.
Learning
--------
The learning has not been fully wrapped. If you need it, get in touch or better
yet, wrap it and submit a pull-request!
Here's a pointer for starters: issue#24. We need to wrap the gradients and getting/setting parameters.
But then, we also need to do something with these, most likely call [minimizeLBFGS from optimization.cpp](https://github.com/lucasb-eyer/pydensecrf/blob/d824b89ee3867bca3e90b9f04c448f1b41821524/pydensecrf/densecrf/src/optimization.cpp).
It should be relatively straightforward to just follow the learning examples included in the [original code](http://graphics.stanford.edu/projects/drf/densecrf_v_2_2.zip).
Common Problems
===============
`undefined symbol` when importing
---------------------------------
If while importing pydensecrf you get an error about some undefined symbols (for example `.../pydensecrf/densecrf.so: undefined symbol: _ZTINSt8ios_base7failureB5cxx11E`), you most likely are inadvertently mixing different compilers or toolchains. Try to see what's going on using tools like `ldd`. If you're using Anaconda, [running `conda install libgcc` might be a solution](https://github.com/lucasb-eyer/pydensecrf/issues/28).
ValueError: Buffer dtype mismatch, expected 'float' but got 'double'
This is a pretty [co](https://github.com/lucasb-eyer/pydensecrf/issues/52)mm[on](https://github.com/lucasb-eyer/pydensecrf/issues/49) user error.
It means exactly what it says: you are passing a `double` but it wants a `float`.
Solve it by, for example, calling `d.setUnaryEnergy(U.astype(np.float32))` instead of just `d.setUnaryEnergy(U)`, or using `float32` in your code in the first place.
My results are all pixelated like [MS Paint's airbrush tool](http://lmgtfy.com/?q=MS+Paint+Airbrush+tool)!
