README.md 11.3 KB
Newer Older
lucasb-eyer's avatar
lucasb-eyer committed
1
2
3
PyDenseCRF
==========

4
This is a (Cython-based) Python wrapper for [Philipp Krähenbühl's Fully-Connected CRFs](http://web.archive.org/web/20161023180357/http://www.philkr.net/home/densecrf) (version 2, [new, incomplete page](http://www.philkr.net/2011/12/01/nips/)).
lucasb-eyer's avatar
lucasb-eyer committed
5
6
7
8
9
10
11
12
13
14
15
16
17
18

If you use this code for your reasearch, please cite:

```
Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
Philipp Krähenbühl and Vladlen Koltun
NIPS 2011
```

and provide a link to this repository as a footnote or a citation.

Installation
============

lucasb-eyer's avatar
lucasb-eyer committed
19
20
21
The package is on PyPI, so simply run `pip install pydensecrf` to install it.

If you want the newest and freshest version, you can install it by executing:
lucasb-eyer's avatar
lucasb-eyer committed
22
23

```
lucasb-eyer's avatar
lucasb-eyer committed
24
pip install git+https://github.com/lucasb-eyer/pydensecrf.git
lucasb-eyer's avatar
lucasb-eyer committed
25
26
```

lucasb-eyer's avatar
lucasb-eyer committed
27
28
and ignoring all the warnings coming from Eigen.

29
Note that you need a relatively recent version of Cython (at least version 0.22) for this wrapper,
30
the one shipped with Ubuntu 14.04 is too old. (Thanks to Scott Wehrwein for pointing this out.)
31
32
I suggest you use a [virtual environment](https://virtualenv.readthedocs.org/en/latest/) and install
the newest version of Cython there (`pip install cython`), but you may update the system version by
33
34
35
36
37
38

```
sudo apt-get remove cython
sudo pip install -U cython
```

lucasb-eyer's avatar
lucasb-eyer committed
39
40
41
42
43
Usage
=====

For images, the easiest way to use this library is using the `DenseCRF2D` class:

44
```python
lucasb-eyer's avatar
lucasb-eyer committed
45
import numpy as np
46
import pydensecrf.densecrf as dcrf
lucasb-eyer's avatar
lucasb-eyer committed
47

48
d = dcrf.DenseCRF2D(640, 480, 5)  # width, height, nlabels
lucasb-eyer's avatar
lucasb-eyer committed
49
50
51
52
53
54
55
```

Unary potential
---------------

You can then set a fixed unary potential in the following way:

56
```python
lucasb-eyer's avatar
lucasb-eyer committed
57
U = np.array(...)     # Get the unary in some way.
58
print(U.shape)        # -> (5, 480, 640)
lucasb-eyer's avatar
lucasb-eyer committed
59
print(U.dtype)        # -> dtype('float32')
60
U = U.reshape((5,-1)) # Needs to be flat.
lucasb-eyer's avatar
lucasb-eyer committed
61
62
63
64
65
66
67
68
69
70
71
d.setUnaryEnergy(U)

# Or alternatively: d.setUnary(ConstUnary(U))
```

Remember that `U` should be negative log-probabilities, so if you're using
probabilities `py`, don't forget to `U = -np.log(py)` them.

Requiring the `reshape` on the unary is an API wart that I'd like to fix, but
don't know how to without introducing an explicit dependency on numpy.

72
73
74
75
76
77
78
79
80
**Note** that the `nlabels` dimension is the first here before the reshape;
you may need to move it there before reshaping if that's not already the case,
like so:

```python
print(U.shape)  # -> (480, 640, 5)
U = U.transpose(2, 0, 1).reshape((5,-1))
```

81
82
83
84
85
### Getting a Unary

There's two common ways of getting unary potentials:

1. From a hard labeling generated by a human or some other processing.
86
   This case is covered by `from pydensecrf.utils import unary_from_labels`.
87
88

2. From a probability distribution computed by, e.g. the softmax output of a
89
   deep network. For this, see `from pydensecrf.utils import unary_from_softmax`.
90

91
For usage of both of these, please refer to their docstrings or have a look at [the example](examples/inference.py).
92

lucasb-eyer's avatar
lucasb-eyer committed
93
94
95
96
97
Pairwise potentials
-------------------

The two-dimensional case has two utility methods for adding the most-common pairwise potentials:

98
```python
lucasb-eyer's avatar
lucasb-eyer committed
99
100
101
102
103
104
105
106
107
108
109
# This adds the color-independent term, features are the locations only.
d.addPairwiseGaussian(sxy=(3,3), compat=3, kernel=dcrf.DIAG_KERNEL, normalization=dcrf.NORMALIZE_SYMMETRIC)

# This adds the color-dependent term, i.e. features are (x,y,r,g,b).
# im is an image-array, e.g. im.dtype == np.uint8 and im.shape == (640,480,3)
d.addPairwiseBilateral(sxy=(80,80), srgb=(13,13,13), rgbim=im, compat=10, kernel=dcrf.DIAG_KERNEL, normalization=dcrf.NORMALIZE_SYMMETRIC)
```

Both of these methods have shortcuts and default-arguments such that the most
common use-case can be simplified to:

110
```python
lucasb-eyer's avatar
lucasb-eyer committed
111
112
113
114
d.addPairwiseGaussian(sxy=3, compat=3)
d.addPairwiseBilateral(sxy=80, srgb=13, rgbim=im, compat=10)
```

115
116
117
118
119
120
The parameters map to those in the paper as follows: `sxy` in the `Gaussian` case is `$\theta_{\gamma}$`,
and in the `Bilateral` case, `sxy` and `srgb` map to `$\theta_{\alpha}$` and `$\theta_{\beta}$`, respectively.
The names are shorthand for "x/y standard-deviation" and "rgb standard-deviation" and for reference, the formula is:

![Equation 3 in the original paper](https://user-images.githubusercontent.com/10962198/36150757-01122bf2-10c5-11e8-97d2-2e833c1c9461.png)

121
122
### Non-RGB bilateral

123
124
125
126
An important caveat is that `addPairwiseBilateral` only works for RGB images, i.e. three channels.
If your data is of different type than this simple but common case, you'll need to compute your
own pairwise energy using `utils.create_pairwise_bilateral`; see the [generic non-2D case](https://github.com/lucasb-eyer/pydensecrf#generic-non-2d) for details.

127
128
A good [example of working with Non-RGB data](https://github.com/lucasb-eyer/pydensecrf/blob/master/examples/Non%20RGB%20Example.ipynb) is provided as a notebook in the examples folder.

lucasb-eyer's avatar
lucasb-eyer committed
129
130
### Compatibilities

131
The `compat` argument can be any of the following:
lucasb-eyer's avatar
lucasb-eyer committed
132
133
134
135
136

- A number, then a `PottsCompatibility` is being used.
- A 1D array, then a `DiagonalCompatibility` is being used.
- A 2D array, then a `MatrixCompatibility` is being used.

137
138
139
These are label-compatibilites `µ(xi, xj)` whose parameters could possibly be [learned](https://github.com/lucasb-eyer/pydensecrf#learning).
For example, they could indicate that mistaking `bird` pixels for `sky` is not as bad as mistaking `cat` for `sky`.
The arrays should have `nlabels` or `(nlabels,nlabels)` as shape and a `float32` datatype.
140

lucasb-eyer's avatar
lucasb-eyer committed
141
142
143
144
145
146
147
148
### Kernels

Possible values for the `kernel` argument are:

- `CONST_KERNEL`
- `DIAG_KERNEL` (the default)
- `FULL_KERNEL`

Lucas Beyer's avatar
Lucas Beyer committed
149
150
151
152
This specifies the kernel's precision-matrix `Λ(m)`, which could possibly be learned.
These indicate correlations between feature types, the default implying no correlation.
Again, this could possiblty be [learned](https://github.com/lucasb-eyer/pydensecrf#learning).

lucasb-eyer's avatar
lucasb-eyer committed
153
154
155
156
157
158
159
160
161
### Normalizations

Possible values for the `normalization` argument are:

- `NO_NORMALIZATION`
- `NORMALIZE_BEFORE`
- `NORMALIZE_AFTER`
- `NORMALIZE_SYMMETRIC` (the default)

Lucas Beyer's avatar
Lucas Beyer committed
162
163
164
165
166
167
168
### Kernel weight

I have so far not found a way to set the kernel weights `w(m)`.
According to the paper, `w(2)` was set to 1 and `w(1)` was cross-validated, but never specified.
Looking through Philip's code (included in [pydensecrf/densecrf](https://github.com/lucasb-eyer/pydensecrf/tree/master/pydensecrf/densecrf)),
I couldn't find such explicit weights, and my guess is they are thus hard-coded to 1.
If anyone knows otherwise, please open an issue or, better yet, a pull-request.
169
Update: user @waldol1 has an idea in [this issue](https://github.com/lucasb-eyer/pydensecrf/issues/37). Feel free to try it out!
Lucas Beyer's avatar
Lucas Beyer committed
170

lucasb-eyer's avatar
lucasb-eyer committed
171
172
173
Inference
---------

174
The easiest way to do inference with 5 iterations is to simply call:
lucasb-eyer's avatar
lucasb-eyer committed
175

176
```python
177
Q = d.inference(5)
lucasb-eyer's avatar
lucasb-eyer committed
178
179
180
181
```

And the MAP prediction is then:

182
```python
lucasb-eyer's avatar
lucasb-eyer committed
183
184
185
map = np.argmax(Q, axis=0).reshape((640,480))
```

186
187
188
189
190
191
192
193
If you're interested in the class-probabilities `Q`, you'll notice `Q` is a
wrapped Eigen matrix. The Eigen wrappers of this project implement the buffer
interface and can be simply cast to numpy arrays like so:

```python
proba = np.array(Q)
```

lucasb-eyer's avatar
lucasb-eyer committed
194
195
196
197
198
Step-by-step inference
----------------------

If for some reason you want to run the inference loop manually, you can do so:

199
```python
lucasb-eyer's avatar
lucasb-eyer committed
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
Q, tmp1, tmp2 = d.startInference()
for i in range(5):
    print("KL-divergence at {}: {}".format(i, d.klDivergence(Q)))
    d.stepInference(Q, tmp1, tmp2)
```

Generic non-2D
--------------

The `DenseCRF` class can be used for generic (non-2D) dense CRFs.
Its usage is exactly the same as above, except that the 2D-specific pairwise
potentials `addPairwiseGaussian` and `addPairwiseBilateral` are missing.

Instead, you need to use the generic `addPairwiseEnergy` method like this:

215
```python
216
d = dcrf.DenseCRF(100, 5)  # npoints, nlabels
lucasb-eyer's avatar
lucasb-eyer committed
217
218

feats = np.array(...)  # Get the pairwise features from somewhere.
219
print(feats.shape)     # -> (7, 100) = (feature dimensionality, npoints)
lucasb-eyer's avatar
lucasb-eyer committed
220
221
222
223
224
225
226
227
228
229
print(feats.dtype)     # -> dtype('float32')

dcrf.addPairwiseEnergy(feats)
```

In addition, you can pass `compatibility`, `kernel` and `normalization`
arguments just like in the 2D gaussian and bilateral cases.

The potential will be computed as `w*exp(-0.5 * |f_i - f_j|^2)`.

230
231
232
233
234
235
236
237
### Pairwise potentials for N-D

User @markusnagel has written a couple numpy-functions generalizing the two
classic 2-D image pairwise potentials (gaussian and bilateral) to an arbitrary
number of dimensions: `create_pairwise_gaussian` and `create_pairwise_bilateral`.
You can access them as `from pydensecrf.utils import create_pairwise_gaussian`
and then have a look at their docstring to see how to use them.

lucasb-eyer's avatar
lucasb-eyer committed
238
239
240
241
242
Learning
--------

The learning has not been fully wrapped. If you need it, get in touch or better
yet, wrap it and submit a pull-request!
Lucas Beyer's avatar
Lucas Beyer committed
243
244
245
246

Here's a pointer for starters: issue#24. We need to wrap the gradients and getting/setting parameters.
But then, we also need to do something with these, most likely call [minimizeLBFGS from optimization.cpp](https://github.com/lucasb-eyer/pydensecrf/blob/d824b89ee3867bca3e90b9f04c448f1b41821524/pydensecrf/densecrf/src/optimization.cpp).
It should be relatively straightforward to just follow the learning examples included in the [original code](http://graphics.stanford.edu/projects/drf/densecrf_v_2_2.zip).
Lucas Beyer's avatar
Lucas Beyer committed
247
248
249
250
251
252
253
254

Common Problems
===============

`undefined symbol` when importing
---------------------------------

If while importing pydensecrf you get an error about some undefined symbols (for example `.../pydensecrf/densecrf.so: undefined symbol: _ZTINSt8ios_base7failureB5cxx11E`), you most likely are inadvertently mixing different compilers or toolchains. Try to see what's going on using tools like `ldd`. If you're using Anaconda, [running `conda install libgcc` might be a solution](https://github.com/lucasb-eyer/pydensecrf/issues/28).
lucasb-eyer's avatar
lucasb-eyer committed
255

256
257
258
259
260
261
262
ValueError: Buffer dtype mismatch, expected 'float' but got 'double'
--------------------------------------------------------------------

This is a pretty [co](https://github.com/lucasb-eyer/pydensecrf/issues/52)mm[on](https://github.com/lucasb-eyer/pydensecrf/issues/49) user error.
It means exactly what it says: you are passing a `double` but it wants a `float`.
Solve it by, for example, calling `d.setUnaryEnergy(U.astype(np.float32))` instead of just `d.setUnaryEnergy(U)`, or using `float32` in your code in the first place.

263
264
265
266
267
268
269
270
271
My results are all pixelated like [MS Paint's airbrush tool](http://lmgtfy.com/?q=MS+Paint+Airbrush+tool)!
----------------------------------------------------------

You screwed up reshaping somewhere and treated the class/label dimension as spatial dimension.
This is you misunderstanding NumPy's memory layout and nothing that PyDenseCRF can detect or help with.

This mistake often happens for the Unary, see the [**Note** in that section of the README](https://github.com/lucasb-eyer/pydensecrf#unary-potential).


lucasb-eyer's avatar
lucasb-eyer committed
272
273
274
275
276
277
278
279
280
281
282
283
284
285
Maintaining
===========

These are instructions for maintainers about how to release new versions. (Mainly instructions for myself.)

```
# Go increment the version in setup.py
> python setup.py build_ext
> python setup.py sdist
> twine upload dist/pydensecrf-VERSION_NUM.tar.gz
```

And that's it. At some point, it would be cool to automate this on [TravisCI](https://docs.travis-ci.com/user/deployment/pypi/), but not worth it yet.
At that point, looking into [creating "manylinux" wheels](https://github.com/pypa/python-manylinux-demo) might be nice, too.
lucasb-eyer's avatar
lucasb-eyer committed
286
287
288
289
290
291

Testing
=======

Thanks to @MarvinTeichmann we now have proper tests, install the package and run `py.test`.
Maybe there's a better way to run them, but both of us don't know :smile: