README.md 10 KB
Newer Older
lucasb-eyer's avatar
lucasb-eyer committed
1
2
3
PyDenseCRF
==========

4
This is a (Cython-based) Python wrapper for [Philipp Krähenbühl's Fully-Connected CRFs](http://web.archive.org/web/20161023180357/http://www.philkr.net/home/densecrf) (version 2, [new, incomplete page](http://www.philkr.net/2011/12/01/nips/)).
lucasb-eyer's avatar
lucasb-eyer committed
5
6
7
8
9
10
11
12
13
14
15
16
17
18

If you use this code for your reasearch, please cite:

```
Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
Philipp Krähenbühl and Vladlen Koltun
NIPS 2011
```

and provide a link to this repository as a footnote or a citation.

Installation
============

lucasb-eyer's avatar
lucasb-eyer committed
19
20
21
The package is on PyPI, so simply run `pip install pydensecrf` to install it.

If you want the newest and freshest version, you can install it by executing:
lucasb-eyer's avatar
lucasb-eyer committed
22
23

```
lucasb-eyer's avatar
lucasb-eyer committed
24
pip install git+https://github.com/lucasb-eyer/pydensecrf.git
lucasb-eyer's avatar
lucasb-eyer committed
25
26
```

lucasb-eyer's avatar
lucasb-eyer committed
27
28
and ignoring all the warnings coming from Eigen.

29
Note that you need a relatively recent version of Cython (at least version 0.22) for this wrapper,
30
the one shipped with Ubuntu 14.04 is too old. (Thanks to Scott Wehrwein for pointing this out.)
31
32
I suggest you use a [virtual environment](https://virtualenv.readthedocs.org/en/latest/) and install
the newest version of Cython there (`pip install cython`), but you may update the system version by
33
34
35
36
37
38

```
sudo apt-get remove cython
sudo pip install -U cython
```

lucasb-eyer's avatar
lucasb-eyer committed
39
40
41
42
43
Usage
=====

For images, the easiest way to use this library is using the `DenseCRF2D` class:

44
```python
lucasb-eyer's avatar
lucasb-eyer committed
45
import numpy as np
46
import pydensecrf.densecrf as dcrf
lucasb-eyer's avatar
lucasb-eyer committed
47

48
d = dcrf.DenseCRF2D(640, 480, 5)  # width, height, nlabels
lucasb-eyer's avatar
lucasb-eyer committed
49
50
51
52
53
54
55
```

Unary potential
---------------

You can then set a fixed unary potential in the following way:

56
```python
lucasb-eyer's avatar
lucasb-eyer committed
57
U = np.array(...)     # Get the unary in some way.
58
print(U.shape)        # -> (5, 640, 480)
lucasb-eyer's avatar
lucasb-eyer committed
59
print(U.dtype)        # -> dtype('float32')
60
U = U.reshape((5,-1)) # Needs to be flat.
lucasb-eyer's avatar
lucasb-eyer committed
61
62
63
64
65
66
67
68
69
70
71
d.setUnaryEnergy(U)

# Or alternatively: d.setUnary(ConstUnary(U))
```

Remember that `U` should be negative log-probabilities, so if you're using
probabilities `py`, don't forget to `U = -np.log(py)` them.

Requiring the `reshape` on the unary is an API wart that I'd like to fix, but
don't know how to without introducing an explicit dependency on numpy.

72
73
74
75
76
### Getting a Unary

There's two common ways of getting unary potentials:

1. From a hard labeling generated by a human or some other processing.
77
   This case is covered by `from pydensecrf.utils import unary_from_labels`.
78
79

2. From a probability distribution computed by, e.g. the softmax output of a
80
   deep network. For this, see `from pydensecrf.utils import unary_from_softmax`.
81

82
For usage of both of these, please refer to their docstrings or have a look at [the example](examples/inference.py).
83

lucasb-eyer's avatar
lucasb-eyer committed
84
85
86
87
88
Pairwise potentials
-------------------

The two-dimensional case has two utility methods for adding the most-common pairwise potentials:

89
```python
lucasb-eyer's avatar
lucasb-eyer committed
90
91
92
93
94
95
96
97
98
99
100
# This adds the color-independent term, features are the locations only.
d.addPairwiseGaussian(sxy=(3,3), compat=3, kernel=dcrf.DIAG_KERNEL, normalization=dcrf.NORMALIZE_SYMMETRIC)

# This adds the color-dependent term, i.e. features are (x,y,r,g,b).
# im is an image-array, e.g. im.dtype == np.uint8 and im.shape == (640,480,3)
d.addPairwiseBilateral(sxy=(80,80), srgb=(13,13,13), rgbim=im, compat=10, kernel=dcrf.DIAG_KERNEL, normalization=dcrf.NORMALIZE_SYMMETRIC)
```

Both of these methods have shortcuts and default-arguments such that the most
common use-case can be simplified to:

101
```python
lucasb-eyer's avatar
lucasb-eyer committed
102
103
104
105
d.addPairwiseGaussian(sxy=3, compat=3)
d.addPairwiseBilateral(sxy=80, srgb=13, rgbim=im, compat=10)
```

106
107
108
109
110
111
The parameters map to those in the paper as follows: `sxy` in the `Gaussian` case is `$\theta_{\gamma}$`,
and in the `Bilateral` case, `sxy` and `srgb` map to `$\theta_{\alpha}$` and `$\theta_{\beta}$`, respectively.
The names are shorthand for "x/y standard-deviation" and "rgb standard-deviation" and for reference, the formula is:

![Equation 3 in the original paper](https://user-images.githubusercontent.com/10962198/36150757-01122bf2-10c5-11e8-97d2-2e833c1c9461.png)

112
113
### Non-RGB bilateral

114
115
116
117
An important caveat is that `addPairwiseBilateral` only works for RGB images, i.e. three channels.
If your data is of different type than this simple but common case, you'll need to compute your
own pairwise energy using `utils.create_pairwise_bilateral`; see the [generic non-2D case](https://github.com/lucasb-eyer/pydensecrf#generic-non-2d) for details.

118
119
A good [example of working with Non-RGB data](https://github.com/lucasb-eyer/pydensecrf/blob/master/examples/Non%20RGB%20Example.ipynb) is provided as a notebook in the examples folder.

lucasb-eyer's avatar
lucasb-eyer committed
120
121
### Compatibilities

122
The `compat` argument can be any of the following:
lucasb-eyer's avatar
lucasb-eyer committed
123
124
125
126
127

- A number, then a `PottsCompatibility` is being used.
- A 1D array, then a `DiagonalCompatibility` is being used.
- A 2D array, then a `MatrixCompatibility` is being used.

128
129
130
These are label-compatibilites `µ(xi, xj)` whose parameters could possibly be [learned](https://github.com/lucasb-eyer/pydensecrf#learning).
For example, they could indicate that mistaking `bird` pixels for `sky` is not as bad as mistaking `cat` for `sky`.
The arrays should have `nlabels` or `(nlabels,nlabels)` as shape and a `float32` datatype.
131

lucasb-eyer's avatar
lucasb-eyer committed
132
133
134
135
136
137
138
139
### Kernels

Possible values for the `kernel` argument are:

- `CONST_KERNEL`
- `DIAG_KERNEL` (the default)
- `FULL_KERNEL`

Lucas Beyer's avatar
Lucas Beyer committed
140
141
142
143
This specifies the kernel's precision-matrix `Λ(m)`, which could possibly be learned.
These indicate correlations between feature types, the default implying no correlation.
Again, this could possiblty be [learned](https://github.com/lucasb-eyer/pydensecrf#learning).

lucasb-eyer's avatar
lucasb-eyer committed
144
145
146
147
148
149
150
151
152
### Normalizations

Possible values for the `normalization` argument are:

- `NO_NORMALIZATION`
- `NORMALIZE_BEFORE`
- `NORMALIZE_AFTER`
- `NORMALIZE_SYMMETRIC` (the default)

Lucas Beyer's avatar
Lucas Beyer committed
153
154
155
156
157
158
159
### Kernel weight

I have so far not found a way to set the kernel weights `w(m)`.
According to the paper, `w(2)` was set to 1 and `w(1)` was cross-validated, but never specified.
Looking through Philip's code (included in [pydensecrf/densecrf](https://github.com/lucasb-eyer/pydensecrf/tree/master/pydensecrf/densecrf)),
I couldn't find such explicit weights, and my guess is they are thus hard-coded to 1.
If anyone knows otherwise, please open an issue or, better yet, a pull-request.
160
Update: user @waldol1 has an idea in [this issue](https://github.com/lucasb-eyer/pydensecrf/issues/37). Feel free to try it out!
Lucas Beyer's avatar
Lucas Beyer committed
161

lucasb-eyer's avatar
lucasb-eyer committed
162
163
164
Inference
---------

165
The easiest way to do inference with 5 iterations is to simply call:
lucasb-eyer's avatar
lucasb-eyer committed
166

167
```python
168
Q = d.inference(5)
lucasb-eyer's avatar
lucasb-eyer committed
169
170
171
172
```

And the MAP prediction is then:

173
```python
lucasb-eyer's avatar
lucasb-eyer committed
174
175
176
map = np.argmax(Q, axis=0).reshape((640,480))
```

177
178
179
180
181
182
183
184
If you're interested in the class-probabilities `Q`, you'll notice `Q` is a
wrapped Eigen matrix. The Eigen wrappers of this project implement the buffer
interface and can be simply cast to numpy arrays like so:

```python
proba = np.array(Q)
```

lucasb-eyer's avatar
lucasb-eyer committed
185
186
187
188
189
Step-by-step inference
----------------------

If for some reason you want to run the inference loop manually, you can do so:

190
```python
lucasb-eyer's avatar
lucasb-eyer committed
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
Q, tmp1, tmp2 = d.startInference()
for i in range(5):
    print("KL-divergence at {}: {}".format(i, d.klDivergence(Q)))
    d.stepInference(Q, tmp1, tmp2)
```

Generic non-2D
--------------

The `DenseCRF` class can be used for generic (non-2D) dense CRFs.
Its usage is exactly the same as above, except that the 2D-specific pairwise
potentials `addPairwiseGaussian` and `addPairwiseBilateral` are missing.

Instead, you need to use the generic `addPairwiseEnergy` method like this:

206
```python
207
d = dcrf.DenseCRF(100, 5)  # npoints, nlabels
lucasb-eyer's avatar
lucasb-eyer committed
208
209

feats = np.array(...)  # Get the pairwise features from somewhere.
210
print(feats.shape)     # -> (7, 100) = (feature dimensionality, npoints)
lucasb-eyer's avatar
lucasb-eyer committed
211
212
213
214
215
216
217
218
219
220
print(feats.dtype)     # -> dtype('float32')

dcrf.addPairwiseEnergy(feats)
```

In addition, you can pass `compatibility`, `kernel` and `normalization`
arguments just like in the 2D gaussian and bilateral cases.

The potential will be computed as `w*exp(-0.5 * |f_i - f_j|^2)`.

221
222
223
224
225
226
227
228
### Pairwise potentials for N-D

User @markusnagel has written a couple numpy-functions generalizing the two
classic 2-D image pairwise potentials (gaussian and bilateral) to an arbitrary
number of dimensions: `create_pairwise_gaussian` and `create_pairwise_bilateral`.
You can access them as `from pydensecrf.utils import create_pairwise_gaussian`
and then have a look at their docstring to see how to use them.

lucasb-eyer's avatar
lucasb-eyer committed
229
230
231
232
233
Learning
--------

The learning has not been fully wrapped. If you need it, get in touch or better
yet, wrap it and submit a pull-request!
Lucas Beyer's avatar
Lucas Beyer committed
234
235
236
237

Here's a pointer for starters: issue#24. We need to wrap the gradients and getting/setting parameters.
But then, we also need to do something with these, most likely call [minimizeLBFGS from optimization.cpp](https://github.com/lucasb-eyer/pydensecrf/blob/d824b89ee3867bca3e90b9f04c448f1b41821524/pydensecrf/densecrf/src/optimization.cpp).
It should be relatively straightforward to just follow the learning examples included in the [original code](http://graphics.stanford.edu/projects/drf/densecrf_v_2_2.zip).
Lucas Beyer's avatar
Lucas Beyer committed
238
239
240
241
242
243
244
245

Common Problems
===============

`undefined symbol` when importing
---------------------------------

If while importing pydensecrf you get an error about some undefined symbols (for example `.../pydensecrf/densecrf.so: undefined symbol: _ZTINSt8ios_base7failureB5cxx11E`), you most likely are inadvertently mixing different compilers or toolchains. Try to see what's going on using tools like `ldd`. If you're using Anaconda, [running `conda install libgcc` might be a solution](https://github.com/lucasb-eyer/pydensecrf/issues/28).
lucasb-eyer's avatar
lucasb-eyer committed
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260

Maintaining
===========

These are instructions for maintainers about how to release new versions. (Mainly instructions for myself.)

```
# Go increment the version in setup.py
> python setup.py build_ext
> python setup.py sdist
> twine upload dist/pydensecrf-VERSION_NUM.tar.gz
```

And that's it. At some point, it would be cool to automate this on [TravisCI](https://docs.travis-ci.com/user/deployment/pypi/), but not worth it yet.
At that point, looking into [creating "manylinux" wheels](https://github.com/pypa/python-manylinux-demo) might be nice, too.
lucasb-eyer's avatar
lucasb-eyer committed
261
262
263
264
265
266

Testing
=======

Thanks to @MarvinTeichmann we now have proper tests, install the package and run `py.test`.
Maybe there's a better way to run them, but both of us don't know :smile: