README.md 8.33 KB
Newer Older
lucasb-eyer's avatar
lucasb-eyer committed
1
2
3
PyDenseCRF
==========

4
This is a (Cython-based) Python wrapper for [Philipp Krähenbühl's Fully-Connected CRFs](http://www.philkr.net/home/densecrf) (version 2).
lucasb-eyer's avatar
lucasb-eyer committed
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

If you use this code for your reasearch, please cite:

```
Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
Philipp Krähenbühl and Vladlen Koltun
NIPS 2011
```

and provide a link to this repository as a footnote or a citation.

Installation
============

You can install this using `pip` by executing:

```
lucasb-eyer's avatar
lucasb-eyer committed
22
pip install git+https://github.com/lucasb-eyer/pydensecrf.git
lucasb-eyer's avatar
lucasb-eyer committed
23
24
```

lucasb-eyer's avatar
lucasb-eyer committed
25
26
and ignoring all the warnings coming from Eigen.

27
Note that you need a relatively recent version of Cython (at least version 0.22) for this wrapper,
28
the one shipped with Ubuntu 14.04 is too old. (Thanks to Scott Wehrwein for pointing this out.)
29
30
I suggest you use a [virtual environment](https://virtualenv.readthedocs.org/en/latest/) and install
the newest version of Cython there (`pip install cython`), but you may update the system version by
31
32
33
34
35
36

```
sudo apt-get remove cython
sudo pip install -U cython
```

lucasb-eyer's avatar
lucasb-eyer committed
37
38
39
40
41
Usage
=====

For images, the easiest way to use this library is using the `DenseCRF2D` class:

42
```python
lucasb-eyer's avatar
lucasb-eyer committed
43
import numpy as np
44
import pydensecrf.densecrf as dcrf
lucasb-eyer's avatar
lucasb-eyer committed
45

46
d = dcrf.DenseCRF2D(640, 480, 5)  # width, height, nlabels
lucasb-eyer's avatar
lucasb-eyer committed
47
48
49
50
51
52
53
```

Unary potential
---------------

You can then set a fixed unary potential in the following way:

54
```python
lucasb-eyer's avatar
lucasb-eyer committed
55
U = np.array(...)     # Get the unary in some way.
56
print(U.shape)        # -> (5, 640, 480)
lucasb-eyer's avatar
lucasb-eyer committed
57
print(U.dtype)        # -> dtype('float32')
58
U = U.reshape((5,-1)) # Needs to be flat.
lucasb-eyer's avatar
lucasb-eyer committed
59
60
61
62
63
64
65
66
67
68
69
d.setUnaryEnergy(U)

# Or alternatively: d.setUnary(ConstUnary(U))
```

Remember that `U` should be negative log-probabilities, so if you're using
probabilities `py`, don't forget to `U = -np.log(py)` them.

Requiring the `reshape` on the unary is an API wart that I'd like to fix, but
don't know how to without introducing an explicit dependency on numpy.

70
71
72
73
74
### Getting a Unary

There's two common ways of getting unary potentials:

1. From a hard labeling generated by a human or some other processing.
75
   This case is covered by `from pydensecrf.utils import unary_from_labels`.
76
77

2. From a probability distribution computed by, e.g. the softmax output of a
78
   deep network. For this, see `from pydensecrf.utils import unary_from_softmax`.
79

80
For usage of both of these, please refer to their docstrings or have a look at [the example](examples/inference.py).
81

lucasb-eyer's avatar
lucasb-eyer committed
82
83
84
85
86
Pairwise potentials
-------------------

The two-dimensional case has two utility methods for adding the most-common pairwise potentials:

87
```python
lucasb-eyer's avatar
lucasb-eyer committed
88
89
90
91
92
93
94
95
96
97
98
# This adds the color-independent term, features are the locations only.
d.addPairwiseGaussian(sxy=(3,3), compat=3, kernel=dcrf.DIAG_KERNEL, normalization=dcrf.NORMALIZE_SYMMETRIC)

# This adds the color-dependent term, i.e. features are (x,y,r,g,b).
# im is an image-array, e.g. im.dtype == np.uint8 and im.shape == (640,480,3)
d.addPairwiseBilateral(sxy=(80,80), srgb=(13,13,13), rgbim=im, compat=10, kernel=dcrf.DIAG_KERNEL, normalization=dcrf.NORMALIZE_SYMMETRIC)
```

Both of these methods have shortcuts and default-arguments such that the most
common use-case can be simplified to:

99
```python
lucasb-eyer's avatar
lucasb-eyer committed
100
101
102
103
d.addPairwiseGaussian(sxy=3, compat=3)
d.addPairwiseBilateral(sxy=80, srgb=13, rgbim=im, compat=10)
```

104
105
106
107
An important caveat is that `addPairwiseBilateral` only works for RGB images, i.e. three channels.
If your data is of different type than this simple but common case, you'll need to compute your
own pairwise energy using `utils.create_pairwise_bilateral`; see the [generic non-2D case](https://github.com/lucasb-eyer/pydensecrf#generic-non-2d) for details.

lucasb-eyer's avatar
lucasb-eyer committed
108
109
### Compatibilities

110
The `compat` argument can be any of the following:
lucasb-eyer's avatar
lucasb-eyer committed
111
112
113
114
115

- A number, then a `PottsCompatibility` is being used.
- A 1D array, then a `DiagonalCompatibility` is being used.
- A 2D array, then a `MatrixCompatibility` is being used.

Lucas Beyer's avatar
Lucas Beyer committed
116
These are label-compatibilites `µ(xi, xj)` whose parameters could possibly be [learned](https://github.com/lucasb-eyer/pydensecrf#learning). For example, they could indicate that mistaking `bird` pixels for `sky` is not as bad as mistaking `cat` for `sky`. (Wrong, old interpretation: <s>[ways to weight contributions](https://github.com/lucasb-eyer/pydensecrf/issues/8#issuecomment-188478006)</s>).
117

lucasb-eyer's avatar
lucasb-eyer committed
118
119
120
121
122
123
124
125
### Kernels

Possible values for the `kernel` argument are:

- `CONST_KERNEL`
- `DIAG_KERNEL` (the default)
- `FULL_KERNEL`

Lucas Beyer's avatar
Lucas Beyer committed
126
127
128
129
This specifies the kernel's precision-matrix `Λ(m)`, which could possibly be learned.
These indicate correlations between feature types, the default implying no correlation.
Again, this could possiblty be [learned](https://github.com/lucasb-eyer/pydensecrf#learning).

lucasb-eyer's avatar
lucasb-eyer committed
130
131
132
133
134
135
136
137
138
### Normalizations

Possible values for the `normalization` argument are:

- `NO_NORMALIZATION`
- `NORMALIZE_BEFORE`
- `NORMALIZE_AFTER`
- `NORMALIZE_SYMMETRIC` (the default)

Lucas Beyer's avatar
Lucas Beyer committed
139
140
141
142
143
144
145
146
### Kernel weight

I have so far not found a way to set the kernel weights `w(m)`.
According to the paper, `w(2)` was set to 1 and `w(1)` was cross-validated, but never specified.
Looking through Philip's code (included in [pydensecrf/densecrf](https://github.com/lucasb-eyer/pydensecrf/tree/master/pydensecrf/densecrf)),
I couldn't find such explicit weights, and my guess is they are thus hard-coded to 1.
If anyone knows otherwise, please open an issue or, better yet, a pull-request.

lucasb-eyer's avatar
lucasb-eyer committed
147
148
149
Inference
---------

150
The easiest way to do inference with 5 iterations is to simply call:
lucasb-eyer's avatar
lucasb-eyer committed
151

152
```python
153
Q = d.inference(5)
lucasb-eyer's avatar
lucasb-eyer committed
154
155
156
157
```

And the MAP prediction is then:

158
```python
lucasb-eyer's avatar
lucasb-eyer committed
159
160
161
map = np.argmax(Q, axis=0).reshape((640,480))
```

162
163
164
165
166
167
168
169
If you're interested in the class-probabilities `Q`, you'll notice `Q` is a
wrapped Eigen matrix. The Eigen wrappers of this project implement the buffer
interface and can be simply cast to numpy arrays like so:

```python
proba = np.array(Q)
```

lucasb-eyer's avatar
lucasb-eyer committed
170
171
172
173
174
Step-by-step inference
----------------------

If for some reason you want to run the inference loop manually, you can do so:

175
```python
lucasb-eyer's avatar
lucasb-eyer committed
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
Q, tmp1, tmp2 = d.startInference()
for i in range(5):
    print("KL-divergence at {}: {}".format(i, d.klDivergence(Q)))
    d.stepInference(Q, tmp1, tmp2)
```

Generic non-2D
--------------

The `DenseCRF` class can be used for generic (non-2D) dense CRFs.
Its usage is exactly the same as above, except that the 2D-specific pairwise
potentials `addPairwiseGaussian` and `addPairwiseBilateral` are missing.

Instead, you need to use the generic `addPairwiseEnergy` method like this:

191
```python
192
d = dcrf.DenseCRF(100, 5)  # npoints, nlabels
lucasb-eyer's avatar
lucasb-eyer committed
193
194

feats = np.array(...)  # Get the pairwise features from somewhere.
195
print(feats.shape)     # -> (7, 100) = (feature dimensionality, npoints)
lucasb-eyer's avatar
lucasb-eyer committed
196
197
198
199
200
201
202
203
204
205
print(feats.dtype)     # -> dtype('float32')

dcrf.addPairwiseEnergy(feats)
```

In addition, you can pass `compatibility`, `kernel` and `normalization`
arguments just like in the 2D gaussian and bilateral cases.

The potential will be computed as `w*exp(-0.5 * |f_i - f_j|^2)`.

206
207
208
209
210
211
212
213
### Pairwise potentials for N-D

User @markusnagel has written a couple numpy-functions generalizing the two
classic 2-D image pairwise potentials (gaussian and bilateral) to an arbitrary
number of dimensions: `create_pairwise_gaussian` and `create_pairwise_bilateral`.
You can access them as `from pydensecrf.utils import create_pairwise_gaussian`
and then have a look at their docstring to see how to use them.

lucasb-eyer's avatar
lucasb-eyer committed
214
215
216
217
218
Learning
--------

The learning has not been fully wrapped. If you need it, get in touch or better
yet, wrap it and submit a pull-request!
Lucas Beyer's avatar
Lucas Beyer committed
219
220
221
222

Here's a pointer for starters: issue#24. We need to wrap the gradients and getting/setting parameters.
But then, we also need to do something with these, most likely call [minimizeLBFGS from optimization.cpp](https://github.com/lucasb-eyer/pydensecrf/blob/d824b89ee3867bca3e90b9f04c448f1b41821524/pydensecrf/densecrf/src/optimization.cpp).
It should be relatively straightforward to just follow the learning examples included in the [original code](http://graphics.stanford.edu/projects/drf/densecrf_v_2_2.zip).
Lucas Beyer's avatar
Lucas Beyer committed
223
224
225
226
227
228
229
230

Common Problems
===============

`undefined symbol` when importing
---------------------------------

If while importing pydensecrf you get an error about some undefined symbols (for example `.../pydensecrf/densecrf.so: undefined symbol: _ZTINSt8ios_base7failureB5cxx11E`), you most likely are inadvertently mixing different compilers or toolchains. Try to see what's going on using tools like `ldd`. If you're using Anaconda, [running `conda install libgcc` might be a solution](https://github.com/lucasb-eyer/pydensecrf/issues/28).