Image Processing

This page documents the functionality present in this library that deals with the management and manipulation of images. One thing to note is that there is no explicit image object. Instead, everything deals with array2d objects that contain various kinds of pixels or user defined generic image objects.

Pixel Types

Most image handling routines in dlib will accept images containing any pixel type. This is made possible by defining a traits class, pixel_traits, for each possible pixel type. This traits class enables image processing routines to determine how to handle each kind of pixel and therefore only pixels which have a pixel_traits definition may be used. The following list defines all the pixel types which come with pixel_traits definitions.

Object Detection

If you want to create object detectors then try the scan_fhog_pyramid tool first. It is quite easy to use and train and will, in many cases, give excellent results. If that doesn't give good results then try the more powerful convolutional neural network based detector.

Pixels rgb_pixel bgr_pixel rgb_alpha_pixel bgr_alpha_pixel hsi_pixel lab_pixel pixel_traits get_pixel_intensity assign_pixel assign_pixel_intensity
Image I/O jpeg_loader load_bmp load_dng load_image load_jpeg load_png png_loader save_bmp save_dng save_png save_jpeg webp_loader load_webp save_webp
Object Detection get_frontal_face_detector object_detector evaluate_detectors full_object_detection mmod_rect yolo_rect scan_image scan_image_movable_parts find_points_above_thresh find_peaks scan_image_pyramid scan_image_boxes scan_fhog_pyramid scan_image_custom find_candidate_object_locations min_barrier_distance test_box_overlap remove_unobtainable_rectangles setup_hashed_features correlation_tracker Scan Image Pyramid Tools compute_box_dimensions create_single_box_detection_template create_overlapped_2x2_detection_template create_grid_detection_template determine_object_boxes setup_grid_detection_templates setup_grid_detection_templates_verbose
Feature Extraction get_surf_points shape_predictor SURF Tools hessian_pyramid compute_surf_descriptor haar_x haar_y get_interest_points interest_point surf_point compute_dominant_angle draw_surf_points find_bright_keypoints find_dark_keypoints hog_image extract_fhog_features fine_hog_image poly_image hashed_feature_image binned_vector_feature_image nearest_neighbor_feature_image randomly_sample_image_features make_uniform_lbp_image extract_histogram_descriptors extract_uniform_lbp_descriptors extract_highdim_face_lbp_descriptors
Edges and Thresholds edge_orientation hysteresis_threshold sobel_edge_detector suppress_non_maximum_edges normalize_image_gradients remove_incoherent_edge_pixels threshold_image partition_pixels hough_transform image_gradients find_bright_lines find_dark_lines
Morphology label_connected_blobs label_connected_blobs_watershed segment_image binary_dilation binary_erosion binary_open binary_close binary_intersection binary_union binary_difference binary_complement skeleton encode_8_pixel_neighbors find_line_endpoints
Filtering gaussian_blur spatially_filter_image spatially_filter_image_separable float_spatially_filter_image_separable separable_3x3_filter_block_grayscale separable_3x3_filter_block_rgb sum_filter sum_filter_assign max_filter spatially_filter_image_separable_down
Scaling and Rotating pyramid_up pyramid_down pyramid_disable create_tiled_pyramid interpolate_nearest_neighbor interpolate_bilinear interpolate_quadratic transform_image rotate_image resize_image letterbox_image flip_image_left_right flip_image_up_down add_image_left_right_flips add_image_rotations upsample_image_dataset flip_image_dataset_left_right rotate_image_dataset extract_image_chips extract_image_4points random_cropper jitter_image sub_image
Visualization randomly_color_image heatmap jet render_face_detections draw_line draw_solid_circle draw_surf_points draw_rectangle tile_images draw_fhog fill_rect
Miscellaneous cv_image toMat assign_image assign_image_scaled assign_all_pixels assign_border_pixels equalize_histogram get_histogram zero_border_pixels integral_image integral_image_generic disturb_colors random_color_transform
hough_transform dlib/image_transforms.h dlib/image_transforms/hough_transform_abstract.h This object is a tool for computing the line finding version of the Hough transform given some kind of edge detection image as input. It also allows the edge pixels to be weighted such that higher weighted edge pixels contribute correspondingly more to the output of the Hough transform, allowing stronger edges to create correspondingly stronger line detections in the final Hough transform. hough_transform_ex.cpp.html skeleton dlib/image_transforms.h dlib/image_transforms/morphological_operations_abstract.h This function computes the skeletonization of an image. That is, given a binary image, we progressively thin the binary blobs until only a single pixel wide skeleton of the original blobs remains. encode_8_pixel_neighbors dlib/image_transforms.h dlib/image_transforms/morphological_operations_abstract.h This routine looks at the 8 pixels immediately surrounding a pixel and encodes their on/off pattern into the 8 bits of an unsigned char. Needless to say, this routine only makes sense for binary images. find_line_endpoints dlib/image_transforms.h dlib/image_transforms/morphological_operations_abstract.h This routine finds endpoints of lines in a thinned binary image. For example, if the image was produced by skeleton() or something like a Canny edge detector then you can use find_line_endpoints() to find the pixels sitting on the ends of lines. disturb_colors dlib/image_transforms.h dlib/image_transforms/random_color_transform_abstract.h Applies a random color transform an image. This is done by creating a random_color_transform with the given parameters and then transforming each pixel in the image with the resulting transform. random_color_transform dlib/image_transforms.h dlib/image_transforms/random_color_transform_abstract.h This object generates a random color balancing and gamma correction transform. It then allows you to apply that specific transform to as many rgb_pixel objects as you like. integral_image_generic dlib/image_transforms.h dlib/image_transforms/integral_image_abstract.h This object is an alternate way of representing image data that allows for very fast computations of sums of pixels in rectangular regions. To use this object you load it with a normal image and then you can use the get_sum_of_area() member function to compute sums of pixels in a given area in constant time. integral_image dlib/image_transforms.h dlib/image_transforms/integral_image_abstract.h This is a specialization of the integral_image_generic template for the case where sums of pixel values should be represented with longs. E.g. if you use 8bit pixels in your original images then this is the appropriate kind of integral image to use with them. haar_x dlib/image_transforms.h dlib/image_transforms/integral_image_abstract.h This is a function that operates on an integral_image and allows you to compute the response of a Haar wavelet oriented along the X axis. haar_y dlib/image_transforms.h dlib/image_transforms/integral_image_abstract.h This is a function that operates on an integral_image and allows you to compute the response of a Haar wavelet oriented along the Y axis. draw_surf_points dlib/image_keypoint/draw_surf_points.h dlib/image_keypoint/draw_surf_points_abstract.h This routine adds a bunch of surf_point objects onto an image_window object so they can be visualized. surf_ex.cpp.html compute_dominant_angle dlib/image_keypoint.h dlib/image_keypoint/surf_abstract.h Computes and returns the dominant angle (i.e. the angle of the dominant gradient) at a given point and scale in an image. This function is part of the main processing of the SURF algorithm. get_surf_points dlib/image_keypoint.h dlib/image_keypoint/surf_abstract.h This function runs the complete SURF algorithm on an input image and returns the points it found. For a description of what exactly the SURF algorithm does you should read the following paper:
SURF: Speeded Up Robust Features By Herbert Bay, Tinne Tuytelaars, and Luc Van Gool

Also note that there are numerous flavors of the SURF algorithm you can put together using the functions in dlib. The get_surf_points() function is just an example of one way you might do so.

surf_ex.cpp.html shape_predictor dlib/image_processing.h dlib/image_processing/shape_predictor_abstract.h This object is a tool that takes in an image region containing some object and outputs a set of point locations that define the pose of the object. The classic example of this is human face pose prediction, where you take an image of a human face as input and are expected to identify the locations of important facial landmarks such as the corners of the mouth and eyes, tip of the nose, and so forth. For example, here is the output of dlib's 68-face-landmark shape_predictor on an image from the HELEN dataset:



To create useful instantiations of this object you need to use the shape_predictor_trainer object to train a shape_predictor using a set of training images, each annotated with shapes you want to predict. To do this, the shape_predictor_trainer uses the state-of-the-art method from the paper:
One Millisecond Face Alignment with an Ensemble of Regression Trees by Vahid Kazemi and Josephine Sullivan, CVPR 2014
face_landmark_detection_ex.cpp.html train_shape_predictor_ex.cpp.html webcam_face_pose_ex.cpp.html train_shape_predictor.py.html face_landmark_detection.py.html face_alignment.py.html compute_surf_descriptor dlib/image_keypoint.h dlib/image_keypoint/surf_abstract.h Computes the 64 dimensional SURF descriptor vector of a box centered at a given center point, tilted at a given angle, and sized according to a given scale. find_bright_keypoints dlib/image_transforms.h dlib/image_transforms/edge_detector_abstract.h This routine finds bright "keypoints" in an image. In general, these are bright/white localized blobs. It does this by computing the determinant of the image Hessian at each location and storing this value into an output saliency image if both eigenvalues of the Hessian are negative. If either eigenvalue is positive then the saliency for that pixel is 0. find_dark_keypoints dlib/image_transforms.h dlib/image_transforms/edge_detector_abstract.h This routine finds dark "keypoints" in an image. In general, these are dark localized blobs. It does this by computing the determinant of the image Hessian at each location and storing this value into an output saliency image if both eigenvalues of the Hessian are positive. If either eigenvalue is negative then the saliency for that pixel is 0. hog_image dlib/image_keypoint.h dlib/image_keypoint/hog_abstract.h This object is a tool for performing the image feature extraction algorithm described in the following paper:
Histograms of Oriented Gradients for Human Detection by Navneet Dalal and Bill Triggs
object_detector_ex.cpp.html train_object_detector.cpp.html extract_fhog_features dlib/image_transforms.h dlib/image_transforms/fhog_abstract.h This function implements the HOG feature extraction method described in the paper:
Object Detection with Discriminatively Trained Part Based Models by P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan in IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010
This means that it takes an input image and outputs Felzenszwalb's 31 dimensional version of HOG features. fhog_ex.cpp.html fine_hog_image dlib/image_keypoint.h dlib/image_keypoint/fine_hog_image_abstract.h This object is a version of the hog_image that allows you to extract HOG features at a finer resolution. poly_image dlib/image_keypoint.h dlib/image_keypoint/poly_image_abstract.h This object is a tool for extracting local feature descriptors from an image. In particular, it fits polynomials to local pixel patches and allows you to query the coefficients of these polynomials. hashed_feature_image dlib/image_keypoint.h dlib/image_keypoint/hashed_feature_image_abstract.h This object is a tool for performing image feature extraction. In particular, it wraps another image feature extractor and converts the wrapped image feature vectors into sparse indicator vectors. It does this by hashing each feature vector and then returns a new vector which is zero everywhere except for the position determined by the hash.

The following feature extractors can be wrapped by the hashed_feature_image: object_detector_ex.cpp.html train_object_detector.cpp.html binned_vector_feature_image dlib/image_keypoint.h dlib/image_keypoint/binned_vector_feature_image_abstract.h This object is a tool for performing image feature extraction. In particular, it wraps another image feature extractor and converts the wrapped image feature vectors into a high dimensional sparse vector. For example, if the lower level feature extractor outputs the vector [3,4,5] and this vector is hashed into the second bin of four bins then the output sparse vector is:
[0,0,0,0, 3,4,5,1, 0,0,0,0, 0,0,0,0].
That is, the output vector has a dimensionality that is equal to the number of hash bins times the dimensionality of the lower level vector plus one. The value in the extra dimension concatenated onto the end of the vector is always a constant value of 1 and serves as a bias value. This means that, if there are N hash bins, these vectors are capable of representing N different linear functions, each operating on the vectors that fall into their corresponding hash bin.

The following feature extractors can be wrapped by the binned_vector_feature_image: nearest_neighbor_feature_image dlib/image_keypoint.h dlib/image_keypoint/nearest_neighbor_feature_image_abstract.h This object is a tool for performing image feature extraction. In particular, it wraps another image feature extractor and converts the wrapped image feature vectors into sparse indicator vectors. It does this by finding the nearest neighbor for each feature vector and returning an indicator vector that is zero everywhere except for the position indicated by the nearest neighbor.

The following feature extractors can be wrapped by the nearest_neighbor_feature_image: hessian_pyramid dlib/image_keypoint.h dlib/image_keypoint/hessian_pyramid_abstract.h This object represents an image pyramid where each level in the pyramid holds determinants of Hessian matrices for the original input image. This object can be used to find stable interest points in an image.

This object is an implementation of the fast Hessian pyramid as described in the paper:
SURF: Speeded Up Robust Features By Herbert Bay, Tinne Tuytelaars, and Luc Van Gool
This implementation was also influenced by the very well documented OpenSURF library and its corresponding description of how the fast Hessian algorithm functions:
Notes on the OpenSURF Library by Christopher Evans
 get_interest_points dlib/image_keypoint.h dlib/image_keypoint/hessian_pyramid_abstract.h This function extracts interest points from a hessian_pyramid. interest_point dlib/image_keypoint.h dlib/image_keypoint/hessian_pyramid_abstract.h This is a simple struct used to represent the interest points returned by the get_interest_points function. surf_point dlib/image_keypoint.h dlib/image_keypoint/surf_abstract.h This is a simple struct used to represent the SURF points returned by the get_surf_points function. surf_ex.cpp.html pixel_traits dlib/pixel.h dlib/pixel.h As the name implies, this is a traits class for pixel types. It allows you to determine what sort of pixel type you are dealing with. hsi_pixel dlib/pixel.h dlib/pixel.h This is a simple struct that represents a HSI colored graphical pixel. lab_pixel dlib/pixel.h dlib/pixel.h This is a simple struct that represents a CIELab colored graphical pixel. rgb_alpha_pixel dlib/pixel.h dlib/pixel.h This is a simple struct that represents an RGB colored graphical pixel with an alpha channel. bgr_alpha_pixel dlib/pixel.h dlib/pixel.h This is a simple struct that represents an BGR colored graphical pixel with an alpha channel.

The difference between this object and the rgb_alpha_pixel is just that this struct lays its pixels down in memory in BGR order rather than RGB order. You only care about this if you are doing something like using the cv_image object to map an OpenCV image into a more object oriented form.

rgb_pixel dlib/pixel.h dlib/pixel.h This is a simple struct that represents an RGB colored graphical pixel. bgr_pixel dlib/pixel.h dlib/pixel.h This is a simple struct that represents a BGR colored graphical pixel.

The difference between this object and the rgb_pixel is just that this struct lays its pixels down in memory in BGR order rather than RGB order. You only care about this if you are doing something like using the cv_image object to map an OpenCV image into a more object oriented form.

cv_image dlib/opencv.h dlib/opencv/cv_image_abstract.h This object is meant to be used as a simple wrapper around the OpenCV IplImage struct or Mat object. Using this class template you can turn an OpenCV image into something that looks like a normal dlib style image object.

So you should be able to use cv_image objects with many of the image processing functions in dlib as well as the GUI tools for displaying images on the screen.

Note that you can do the reverse conversion, from dlib to OpenCV, using the toMat routine.

webcam_face_pose_ex.cpp.html toMat dlib/opencv.h dlib/opencv/to_open_cv_abstract.h This routine converts a dlib style image into an instance of OpenCV's cv::Mat object. This is done by setting up the Mat object to point to the same memory as the dlib image.

Note that you can do the reverse conversion, from OpenCV to dlib, using the cv_image object.

heatmap dlib/image_transforms.h dlib/image_transforms/colormaps_abstract.h Converts a grayscale image into a heatmap. This is useful if you want to display a grayscale image with more than 256 values. In particular, this function uses the following color mapping:
image_ex.cpp.html jet dlib/image_transforms.h dlib/image_transforms/colormaps_abstract.h Converts a grayscale image into an image using the jet color scheme. This is useful if you want to display a grayscale image with more than 256 values. In particular, this function uses the following color mapping:
randomly_color_image dlib/image_transforms.h dlib/image_transforms/colormaps_abstract.h Randomly generates a mapping from gray level pixel values to the RGB pixel space and then uses this mapping to create a colored version an image.

This function is useful for displaying the results of some image segmentation. For example, the output of label_connected_blobs, label_connected_blobs_watershed, or segment_image.

assign_pixel dlib/pixel.h dlib/pixel.h assign_pixel() is a templated function that can assign any pixel type to another pixel type. It will perform whatever conversion is necessary to make the assignment work. (E.g. color to grayscale conversion) assign_pixel_intensity dlib/pixel.h dlib/pixel.h assign_pixel_intensity() is a templated function that can change the intensity of a pixel. So if the pixel in question is a grayscale pixel then it simply assigns that pixel the given value. However, if the pixel is not a grayscale pixel then it converts the pixel to the HSI color space and sets the I channel to the given intensity and then converts this HSI value back to the original pixel's color space. get_pixel_intensity dlib/pixel.h dlib/pixel.h get_pixel_intensity() is a templated function that returns the grayscale intensity of a pixel. If the pixel isn't a grayscale pixel then it converts the pixel to grayscale and returns that value. png_loader dlib/image_io.h dlib/image_loader/png_loader_abstract.h This object loads a Portable Network Graphics (PNG) image file into an array2d of pixels.

Note that you must define DLIB_PNG_SUPPORT if you want to use this object. You must also set your build environment to link to the libpng library. However, if you use CMake and dlib's default CMakeLists.txt file then it will get setup automatically.

jpeg_loader dlib/image_io.h dlib/image_loader/jpeg_loader_abstract.h This object loads a JPEG image file into an array2d of pixels.

Note that you must define DLIB_JPEG_SUPPORT if you want to use this object. You must also set your build environment to link to the libjpeg library. However, if you use CMake and dlib's default CMakeLists.txt file then it will get setup automatically.

webp_loader dlib/image_io.h dlib/image_loader/webp_loader_abstract.h This object loads a WebP image file into an array2d of pixels.

Note that you must define DLIB_WEBP_SUPPORT if you want to use this object. You must also set your build environment to link to the libwebp library. However, if you use CMake and dlib's default CMakeLists.txt file then it will get setup automatically.

load_jpeg dlib/image_io.h dlib/image_loader/jpeg_loader_abstract.h This function loads a JPEG image file into an array2d of pixels.

Note that you must define DLIB_JPEG_SUPPORT if you want to use this function. You must also set your build environment to link to the libjpeg library. However, if you use CMake and dlib's default CMakeLists.txt file then it will get setup automatically.

load_dng dlib/image_io.h dlib/image_loader/image_loader_abstract.h This global function loads a dlib DNG file (a lossless compressed image format) into an array2d of pixels. load_webp dlib/image_io.h dlib/image_loader/image_loader_abstract.h This function loads a WebP image file into an array2d of pixels.

Note that you must define DLIB_WEBP_SUPPORT if you want to use this function. You must also set your build environment to link to the libwebp library. However, if you use CMake and dlib's default CMakeLists.txt file then it will get setup automatically.

save_dng dlib/image_io.h dlib/image_saver/image_saver_abstract.h This global function saves an image as a dlib DNG file (a lossless compressed image format).

This routine can save images containing any type of pixel. However, the DNG format can natively store only the following pixel types: rgb_pixel, hsi_pixel, rgb_alpha_pixel, uint8, uint16, float, and double. All other pixel types will be converted into one of these types as appropriate before being saved to disk.

save_png dlib/image_io.h dlib/image_saver/save_png_abstract.h This global function writes an image to disk as a PNG (Portable Network Graphics) file.

Note that you must define DLIB_PNG_SUPPORT if you want to use this function. You must also set your build environment to link to the libpng library. However, if you use CMake and dlib's default CMakeLists.txt file then it will get setup automatically.

This routine can save images containing any type of pixel. However, save_png() can only natively store the following pixel types: rgb_pixel, rgb_alpha_pixel, uint8, and uint16. All other pixel types will be converted into one of these types as appropriate before being saved to disk.

save_jpeg dlib/image_io.h dlib/image_saver/save_jpeg_abstract.h This global function writes an image to disk as a JPEG file.

Note that you must define DLIB_JPEG_SUPPORT if you want to use this function. You must also set your build environment to link to the libjpeg library. However, if you use CMake and dlib's default CMakeLists.txt file then it will get setup automatically.

This routine can save images containing any type of pixel. However, save_jpeg() can only natively store the following pixel types: rgb_pixel and uint8. All other pixel types will be converted into one of these types as appropriate before being saved to disk.

save_webp dlib/image_io.h dlib/image_saver/save_webp_abstract.h This global function writes an image to disk as a WebP file.

Note that you must define DLIB_WEBP_SUPPORT if you want to use this function. You must also set your build environment to link to the libwebp library. However, if you use CMake and dlib's default CMakeLists.txt file then it will get setup automatically.

This routine can save images containing any type of pixel. However, save_webp() can only natively store the following pixel types: rgb_pixel, rgb_alpha_pixel, bgr_pixel, and bgr_alpha_pixel. All other pixel types will be converted into one of these types as appropriate before being saved to disk.

load_image dlib/image_io.h dlib/image_loader/load_image_abstract.h This global function takes a file name and loads it into an array2d of pixels using the appropriate image loading routine. The supported types are BMP, PNG, JPEG, GIF, and the dlib DNG file format.

Note that you can only load PNG, JPEG, and GIF files if you link against libpng, libjpeg, and libgif respectively. You will also need to #define DLIB_PNG_SUPPORT, DLIB_JPEG_SUPPORT, and DLIB_GIF_SUPPORT. Or use CMake and it will do all this for you.

image_ex.cpp.html load_bmp dlib/image_io.h dlib/image_loader/image_loader_abstract.h This global function loads a MS Windows BMP file into an array2d of pixels. load_png dlib/image_io.h dlib/image_loader/png_loader_abstract.h This function loads a Portable Network Graphics (PNG) image file into an array2d of pixels.

Note that you must define DLIB_PNG_SUPPORT if you want to use this function. You must also set your build environment to link to the libpng library. However, if you use CMake and dlib's default CMakeLists.txt file then it will get setup automatically.

save_bmp dlib/image_io.h dlib/image_saver/image_saver_abstract.h This global function saves an image as a MS Windows BMP file.

This routine can save images containing any type of pixel. However, it will convert all color pixels into rgb_pixel and grayscale pixels into uint8 type before saving to disk.

draw_line dlib/image_transforms.h dlib/image_transforms/draw_abstract.h This global function draws a line on an image. draw_solid_circle dlib/image_transforms.h dlib/image_transforms/draw_abstract.h This global function draws a solid circle on an image. render_face_detections dlib/image_processing/render_face_detections.h dlib/image_processing/render_face_detections_abstract.h This function takes a set of full_object_detections which represent human faces annotated with 68 facial landmarks (according to the iBUG 300-W scheme) and converts them into a form suitable for display on an image_window.

For example, it will take the output of a shape_predictor that uses this facial landmarking scheme and will produce visualizations like this:

face_landmark_detection_ex.cpp.html webcam_face_pose_ex.cpp.html draw_rectangle dlib/image_transforms.h dlib/image_transforms/draw_abstract.h This global function draws a rectangle on an image. tile_images dlib/image_transforms.h dlib/image_transforms/draw_abstract.h This function takes an array of images and tiles them into a single large square image and returns this new big tiled image. Therefore, it is a useful method to visualize many small images at once. draw_fhog dlib/image_transforms.h dlib/image_transforms/fhog_abstract.h This function takes a FHOG feature map which was created by extract_fhog_features and converts it into an image suitable for display on the screen. In particular, we draw all the hog cells into a grayscale image in a way that shows the magnitude and orientation of the gradient energy in each cell. fhog_ex.cpp.html fhog_object_detector_ex.cpp.html fill_rect dlib/image_transforms.h dlib/image_transforms/draw_abstract.h This global function draws a solid rectangle on an image. assign_border_pixels dlib/image_transforms.h dlib/image_transforms/assign_image_abstract.h This global function assigns all the pixels in the border of an image to a specific value. assign_all_pixels dlib/image_transforms.h dlib/image_transforms/assign_image_abstract.h This global function assigns all the pixels in an image a specific value. assign_image dlib/image_transforms.h dlib/image_transforms/assign_image_abstract.h This global function copies one image into another and performs any necessary color space conversions to make it work right. assign_image_scaled dlib/image_transforms.h dlib/image_transforms/assign_image_abstract.h This global function copies one image into another and performs any necessary color space conversions to make it work right. Additionally, if the dynamic range of the source image is too big to fit into the destination image then it will attempt to perform the appropriate scaling. get_histogram dlib/image_transforms.h dlib/image_transforms/equalize_histogram_abstract.h This global function computes an image's histogram and returns it in the form of a column or row matrix object. edge_orientation dlib/image_transforms.h dlib/image_transforms/edge_detector_abstract.h This global function takes horizontal and vertical gradient magnitude values and returns the orientation of the gradient. equalize_histogram dlib/image_transforms.h dlib/image_transforms/equalize_histogram_abstract.h This global function performs histogram equalization on an image. hysteresis_threshold dlib/image_transforms.h dlib/image_transforms/thresholding_abstract.h This global function performs hysteresis thresholding on an image. sobel_edge_detector dlib/image_transforms.h dlib/image_transforms/edge_detector_abstract.h This global function performs spatial filtering on an image using the sobel edge detection filters. image_ex.cpp.html find_bright_lines dlib/image_transforms.h dlib/image_transforms/edge_detector_abstract.h This routine is similar to sobel_edge_detector(), except instead of finding an edge it finds a bright/white line. For example, the border between a black piece of paper and a white table is an edge, but a curve drawn with a pencil on a piece of paper makes a line. find_dark_lines dlib/image_transforms.h dlib/image_transforms/edge_detector_abstract.h This routine is similar to sobel_edge_detector(), except instead of finding an edge it finds a dark line. For example, the border between a black piece of paper and a white table is an edge, but a curve drawn with a pencil on a piece of paper makes a line. image_gradients dlib/image_transforms.h dlib/image_transforms/edge_detector_abstract.h This class is a tool for computing first and second derivatives of an image. It does this by fitting a quadratic surface around each pixel and then computing the gradients of that quadratic surface. For the details see the paper:
Quadratic models for curved line detection in SAR CCD by Davis E. King and Rhonda D. Phillips
This technique gives very accurate gradient estimates and is also very fast since the entire gradient estimation procedure, for each type of gradient, is accomplished by cross-correlating the image with a single separable filter. This means you can compute gradients at very large scales (e.g. by fitting the quadratic to a large window, like a 99x99 window) and it still runs very quickly.

For example, the filters used to compute the X, Y, XX, XY, and YY gradients at a scale of 130 are shown below:
X: Y:
XX: XY: YY:

suppress_non_maximum_edges dlib/image_transforms.h dlib/image_transforms/edge_detector_abstract.h This global function performs non-maximum suppression on a gradient image. image_ex.cpp.html normalize_image_gradients dlib/image_transforms.h dlib/image_transforms/edge_detector_abstract.h This function takes two gradient images produced by a function like sobel_edge_detector and then unit normalizes the gradient vectors. remove_incoherent_edge_pixels dlib/image_transforms.h dlib/image_transforms/edge_detector_abstract.h This routine takes a list of edge pixels and discards the ones that have outlying gradient directions. In particular, it finds the largest group of pixels in an edge that are all within a user specified angle of each other. Therefore, if you are looking for straight lines then this routine is a useful tool since it allows you to discard noisy edge pixels with inconsistent angles. zero_border_pixels dlib/image_transforms.h dlib/image_transforms/assign_image_abstract.h This global function zeros the pixels on the border of an image. spatially_filter_image dlib/image_transforms.h dlib/image_transforms/spatial_filtering_abstract.h This global function performs spatial filtering on an image with a user supplied filter. gaussian_blur dlib/image_transforms.h dlib/image_transforms/spatial_filtering_abstract.h This global function blurs an image by convolving it with a Gaussian filter. spatially_filter_image_separable dlib/image_transforms.h dlib/image_transforms/spatial_filtering_abstract.h This global function performs spatial filtering on an image with a user supplied separable filter. float_spatially_filter_image_separable dlib/image_transforms.h dlib/image_transforms/spatial_filtering_abstract.h This global function performs spatial filtering on an image with a user supplied separable filter. It is optimized to work only on float valued images with float valued filters. spatially_filter_image_separable_down dlib/image_transforms.h dlib/image_transforms/spatial_filtering_abstract.h This global function performs spatial filtering on an image with a user supplied separable filter. Additionally, it produces a downsampled output. max_filter dlib/image_transforms.h dlib/image_transforms/spatial_filtering_abstract.h This function slides a rectangle over an input image and outputs a new image which contains the maximum valued pixel found inside the rectangle at each position in the input image. sum_filter dlib/image_transforms.h dlib/image_transforms/spatial_filtering_abstract.h This function slides a rectangle over an input image and adds the sum of pixel values in each rectangle location to another image. sum_filter_assign dlib/image_transforms.h dlib/image_transforms/spatial_filtering_abstract.h This function slides a rectangle over an input image and outputs a new image which contains the sum of pixels inside the rectangle at each position in the input image. separable_3x3_filter_block_grayscale dlib/image_transforms.h dlib/image_transforms/spatial_filtering_abstract.h This routine filters part of an image with a user supplied 3x3 separable filter. The output is a grayscale sub-image. separable_3x3_filter_block_rgb dlib/image_transforms.h dlib/image_transforms/spatial_filtering_abstract.h This routine filters part of an image with a user supplied 3x3 separable filter. The output is a RGB sub-image. create_tiled_pyramid dlib/image_transforms.h dlib/image_transforms/image_pyramid_abstract.h This function creates an image pyramid and packs the entire pyramid into one big image. It does this by tiling the different pyramid layers together and outputting the result. Here is an example:

Also, you can use the image_to_tiled_pyramid() and tiled_pyramid_to_image() routines to convert between the input image coordinate space and the tiled pyramid coordinate space. pyramid_down dlib/image_transforms.h dlib/image_transforms/image_pyramid_abstract.h This is a simple function object to help create image pyramids. It downsamples an image by a ratio of N to N-1 where N is supplied by the user as a template argument. pyramid_disable dlib/image_transforms.h dlib/image_transforms/image_pyramid_abstract.h This object downsamples an image at a ratio of infinity to 1. That means it always outputs an image of size zero. This is useful because it can be supplied to routines which take a pyramid_down function object and it will essentially disable pyramid processing. This way, a pyramid oriented function can be turned into a regular routine which processes just the original undownsampled image. interpolate_nearest_neighbor dlib/image_transforms.h dlib/image_transforms/interpolation_abstract.h This object is a tool for performing nearest neighbor interpolation on an image. interpolate_bilinear dlib/image_transforms.h dlib/image_transforms/interpolation_abstract.h This object is a tool for performing bilinear interpolation on an image. interpolate_quadratic dlib/image_transforms.h dlib/image_transforms/interpolation_abstract.h This object is a tool for performing quadratic interpolation on an image. transform_image dlib/image_transforms.h dlib/image_transforms/interpolation_abstract.h This routine is a tool for transforming images using some kind of point mapping function (e.g. point_transform_affine) and pixel interpolation tool (e.g. interpolate_quadratic). An example application of this routine is for image rotation. Indeed, it is how rotate_image is implemented. rotate_image dlib/image_transforms.h dlib/image_transforms/interpolation_abstract.h This is a routine for rotating an image. pyramid_up dlib/image_transforms.h dlib/image_transforms/interpolation_abstract.h This routine upsamples an image. In particular, it takes a pyramid_down object (or an object with a compatible interface) as an argument and performs an upsampling which is the inverse of the supplied pyramid_down object. resize_image dlib/image_transforms.h dlib/image_transforms/interpolation_abstract.h This is a routine capable of resizing or stretching an image. letterbox_image dlib/image_transforms.h dlib/image_transforms/interpolation_abstract.h Scales an image so that it fits into a size * size square, while preserving the aspect ratio of the actual contents by appropriate 0 padding. dnn_yolo_train_ex.cpp.html extract_image_chips dlib/image_transforms.h dlib/image_transforms/interpolation_abstract.h This function extracts "chips" from an image. That is, it takes a list of rectangular sub-windows (i.e. chips) within an image and extracts those sub-windows, storing each into its own image. It also allows the user to specify the scale and rotation for the chip. face_landmark_detection_ex.cpp.html extract_image_4points dlib/image_transforms.h dlib/image_transforms/interpolation_abstract.h This function allows you to extract an arbitrary quadrilateral patch from an image. For example, given the image on the left and the 4 corners of the red box you can extract the image on the right:

random_cropper dlib/image_transforms.h dlib/image_transforms/random_cropper_abstract.h This object is a tool for extracting random crops of objects from a set of images. The crops are randomly jittered in scale, translation, and rotation but more or less centered on objects specified by mmod_rect objects. random_cropper_ex.cpp.html dnn_mmod_ex.cpp.html dnn_mmod_find_cars_ex.cpp.html dnn_mmod_train_find_cars_ex.cpp.html jitter_image dlib/image_transforms.h dlib/image_transforms/interpolation_abstract.h Randomly jitters an image by slightly rotating, scaling, and translating it. There is also a 50% chance it will be mirrored left to right. dnn_metric_learning_on_images_ex.cpp.html dnn_face_recognition_ex.cpp.html sub_image dlib/image_transforms.h dlib/image_transforms/interpolation_abstract.h This function returns a lightweight sub-image of another image. In particular, the returned sub-image simply holds a pointer to the original image, meaning there is no overhead for using or creating the sub-image. add_image_left_right_flips dlib/image_transforms.h dlib/image_transforms/interpolation_abstract.h This routine takes a set of images and bounding boxes within those images and doubles the size of the dataset by adding left/right flipped copies of each image as well as the corresponding bounding boxes. Therefore, this function is useful if you are training and object detector and your objects have a left/right symmetry. fhog_object_detector_ex.cpp.html add_image_rotations dlib/image_transforms.h dlib/image_transforms/interpolation_abstract.h This routine takes a set of images and bounding boxes within those images and grows the dataset by computing many different rotations of each image. It will also adjust the positions of the bounding boxes so that they still fall on the same objects in each rotated image. rotate_image_dataset dlib/image_transforms.h dlib/image_transforms/interpolation_abstract.h This routine takes a set of images and bounding boxes within those images and rotates the entire dataset by a user specified angle. This means that all images are rotated and the bounding boxes are adjusted so that they still sit on top of the same visual objects in the new rotated images. flip_image_dataset_left_right dlib/image_transforms.h dlib/image_transforms/interpolation_abstract.h This routine takes a set of images and bounding boxes within those images and mirrors the entire dataset left to right. This means that all images are flipped left to right and the bounding boxes are adjusted so that they still sit on top of the same visual objects in the new flipped images. upsample_image_dataset dlib/image_transforms.h dlib/image_transforms/interpolation_abstract.h This routine takes a set of images and bounding boxes within those images and upsamples the entire dataset. This means that all images are upsampled and the bounding boxes are adjusted so that they still sit on top of the same visual objects in the new images. flip_image_left_right dlib/image_transforms.h dlib/image_transforms/interpolation_abstract.h This is a routine which can flip an image from left to right. (e.g. as if viewed through a mirror). flip_image_up_down dlib/image_transforms.h dlib/image_transforms/interpolation_abstract.h This routine flips an image upside down. partition_pixels dlib/image_transforms.h dlib/image_transforms/thresholding_abstract.h This function finds a threshold value that would be reasonable to use with threshold_image. It does this by finding the threshold that partitions the pixels in an image into two groups such that the sum of absolute deviations between each pixel and the mean of its group is minimized. threshold_image dlib/image_transforms.h dlib/image_transforms/thresholding_abstract.h This global function performs a simple binary thresholding on an image with a user supplied threshold. partition_pixels dlib/image_transforms.h dlib/image_transforms/thresholding_abstract.h Finds a threshold value that would be reasonable to use with threshold_image. It does this by finding the threshold that partitions the pixels in an image into two groups such that the sum of absolute deviations between each pixel and the mean of its group is minimized. label_connected_blobs dlib/image_transforms.h dlib/image_transforms/label_connected_blobs_abstract.h This function labels each of the connected blobs in an image with a unique integer label. label_connected_blobs_watershed dlib/image_transforms.h dlib/image_transforms/label_connected_blobs_abstract.h This function performs a watershed segmentation of an image and labels each resulting flooding region with a unique integer label.

For example, given an image like this, which shows part of the Hubble ultra deep field:


The algorithm would flood fill it and produce the following segmentation. If you play the video you can see the actual execution of the algorithm where it starts by flooding from the brightest areas and progressively fills each region.

segment_image dlib/image_transforms.h dlib/image_transforms/segment_image_abstract.h Attempts to segment an image into regions which have some visual consistency to them. In particular, this function implements the algorithm described in the paper:
Efficient Graph-Based Image Segmentation by Felzenszwalb and Huttenlocher.
find_candidate_object_locations dlib/image_transforms.h dlib/image_transforms/segment_image_abstract.h This function takes an input image and generates a set of candidate rectangles which are expected to bound any objects in the image. It does this by running a version of the segment_image routine on the image and then reports rectangles containing each of the segments as well as rectangles containing unions of adjacent segments. The basic idea is described in the paper:
Segmentation as Selective Search for Object Recognition by Koen E. A. van de Sande, et al.
Note that this function deviates from what is described in the paper slightly. See the code for details. find_candidate_object_locations.py.html min_barrier_distance dlib/image_transforms.h dlib/image_transforms/segment_image_abstract.h This function implements the salient object detection method described in the paper:
"Minimum barrier salient object detection at 80 fps" by Zhang, Jianming, et al.
This routine takes an image as input and creates a new image where objects that are visually distinct from the borders of the image have bright pixels while everything else is dark. This is useful because you can then threshold the resulting image to detect objects. For example, given the left image as input you get the right image as output:

binary_dilation dlib/image_transforms.h dlib/image_transforms/morphological_operations_abstract.h This global function performs the morphological operation of dilation on an image. binary_erosion dlib/image_transforms.h dlib/image_transforms/morphological_operations_abstract.h This global function performs the morphological operation of erosion on an image. binary_open dlib/image_transforms.h dlib/image_transforms/morphological_operations_abstract.h This global function performs a morphological opening on an image. binary_close dlib/image_transforms.h dlib/image_transforms/morphological_operations_abstract.h This global function performs a morphological closing on an image. binary_intersection dlib/image_transforms.h dlib/image_transforms/morphological_operations_abstract.h This global function computes the intersection of two binary images. binary_union dlib/image_transforms.h dlib/image_transforms/morphological_operations_abstract.h This global function computes the union of two binary images. binary_difference dlib/image_transforms.h dlib/image_transforms/morphological_operations_abstract.h This global function computes the difference of two binary images. binary_complement dlib/image_transforms.h dlib/image_transforms/morphological_operations_abstract.h This global function computes the complement of a binary image. scan_image dlib/image_processing.h dlib/image_processing/scan_image_abstract.h This global function is a tool for sliding a set of rectangles over an image space and finding the locations where the sum of pixels in the rectangles exceeds a threshold. It is useful for implementing certain kinds of sliding window classifiers. scan_image_movable_parts dlib/image_processing.h dlib/image_processing/scan_image_abstract.h This global function is a tool for sliding a set of rectangles over an image space and finding the locations where the sum of pixels in the rectangles exceeds a threshold. It is useful for implementing certain kinds of sliding window classifiers. The behavior of this routine is similar to scan_image except that it can also handle movable parts in addition to rigidly placed parts within the sliding window. find_points_above_thresh dlib/image_processing.h dlib/image_processing/scan_image_abstract.h This routine finds all points in an image with a pixel value above a threshold. It also has the ability to produce an efficient random subsample of such points if the number of them is very large. find_peaks dlib/image_processing.h dlib/image_processing/scan_image_abstract.h This routine finds all the points in an image with a pixel value above a user supplied threshold that are also local maximizers. scan_image_pyramid dlib/image_processing.h dlib/image_processing/scan_image_pyramid_abstract.h This object is a tool for running a sliding window classifier over an image pyramid. This object can also be understood as a general tool for implementing the spatial pyramid models described in the paper:
Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories by Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce
It also includes the ability to represent movable part models.

The following feature extractors can be used with the scan_image_pyramid object: object_detector_ex.cpp.html object_detector_advanced_ex.cpp.html scan_fhog_pyramid dlib/image_processing.h dlib/image_processing/scan_fhog_pyramid_abstract.h This object is a tool for running a fixed sized sliding window classifier over an image pyramid. In particular, it slides a linear classifier over a HOG pyramid as discussed in the paper:
Histograms of Oriented Gradients for Human Detection by Navneet Dalal and Bill Triggs, CVPR 2005
However, we augment the method slightly to use the version of HOG features from:
Object Detection with Discriminatively Trained Part Based Models by P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010
Since these HOG features have been shown to give superior performance. fhog_object_detector_ex.cpp.html train_object_detector.py.html scan_image_boxes dlib/image_processing.h dlib/image_processing/scan_image_boxes_abstract.h This object is a tool for running a classifier over an image with the goal of localizing each object present. The localization is in the form of the bounding box around each object of interest.

Unlike the scan_image_pyramid object which scans a fixed sized window over an image pyramid, the scan_image_boxes tool allows you to define your own list of "candidate object locations" which should be evaluated. This is simply a list of rectangle objects which might contain objects of interest. The scan_image_boxes object will then evaluate the classifier at each of these locations and return the subset of rectangles which appear to have objects in them.

This object can also be understood as a general tool for implementing the spatial pyramid models described in the paper:
Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories by Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce


The following feature extractors can be used with the scan_image_boxes object: scan_image_custom dlib/image_processing.h dlib/image_processing/scan_image_custom_abstract.h This object is a tool for running a classifier over an image with the goal of localizing each object present. The localization is in the form of the bounding box around each object of interest.

Unlike the scan_image_pyramid and scan_image_boxes objects, this image scanner delegates all the work of constructing the object feature vector to a user supplied feature extraction object. That is, scan_image_custom simply asks the supplied feature extractor what boxes in the image we should investigate and then asks the feature extractor for the complete feature vector for each box. That is, scan_image_custom does not apply any kind of pyramiding or other higher level processing to the features coming out of the feature extractor. That means that when you use scan_image_custom it is completely up to you to define the feature vector used with each image box.

full_object_detection dlib/image_processing.h dlib/image_processing/full_object_detection_abstract.h This object represents the location of an object in an image along with the positions of each of its constituent parts. mmod_rect dlib/image_processing.h dlib/image_processing/full_object_detection_abstract.h This is a simple struct that is used to give training data and receive detections from the Max-Margin Object Detection loss layer. yolo_rect dlib/image_processing.h dlib/image_processing/full_object_detection_abstract.h This is a simple struct that is used to give training data and receive detections from the YOLO loss layer for object detection. get_frontal_face_detector dlib/image_processing/frontal_face_detector.h dlib/image_processing/frontal_face_detector_abstract.h This function returns an object_detector that is configured to find human faces that are looking more or less towards the camera. It is created using the scan_fhog_pyramid object. face_detection_ex.cpp.html face_detector.py.html webcam_face_pose_ex.cpp.html object_detector dlib/image_processing.h dlib/image_processing/object_detector_abstract.h This object is a tool for detecting the positions of objects in an image. In particular, it is a simple container to aggregate an instance of an image scanner object (either scan_fhog_pyramid, scan_image_pyramid, scan_image_boxes, or scan_image_custom), the weight vector needed by one of these image scanners, and finally an instance of test_box_overlap. The test_box_overlap object is used to perform non-max suppression on the output of the image scanner object.

Note that you can use the structural_object_detection_trainer to learn the parameters of an object_detector. See the example programs for an introduction.

Also note that dlib contains more powerful CNN based object detection tooling, which will usually run slower but produce much more general and accurate detectors.

fhog_object_detector_ex.cpp.html face_detection_ex.cpp.html object_detector_ex.cpp.html object_detector_advanced_ex.cpp.html train_object_detector.cpp.html face_detector.py.html train_object_detector.py.html correlation_tracker dlib/image_processing.h dlib/image_processing/correlation_tracker_abstract.h This is a tool for tracking moving objects in a video stream. You give it the bounding box of an object in the first frame and it attempts to track the object in the box from frame to frame.

This tool is an implementation of the method described in the following paper:

Danelljan, Martin, et al. "Accurate scale estimation for robust visual tracking." Proceedings of the British Machine Vision Conference BMVC. 2014.

video_tracking_ex.cpp.html correlation_tracker.py.html evaluate_detectors dlib/image_processing.h dlib/image_processing/scan_fhog_pyramid_abstract.h This function allows you to efficiently run a bunch of scan_fhog_pyramid based object_detectors over an image. Importantly, this function is faster than running each detector individually because it computes the HOG features only once and then reuses them for each detector. fhog_object_detector_ex.cpp.html test_box_overlap dlib/image_processing.h dlib/image_processing/box_overlap_testing_abstract.h This object is a simple function object for determining if two rectangles overlap. remove_unobtainable_rectangles dlib/image_processing.h dlib/image_processing/remove_unobtainable_rectangles_abstract.h Recall that the scan_image_pyramid and scan_image_boxes objects can't produce all possible rectangles as object detections since they only consider a limited subset of all possible object positions. Therefore, when training an object detector that uses these tools you must make sure the training data does not contain any object locations that are unobtainable by the image scanning model. The remove_unobtainable_rectangles() routine is a tool to filter out these unobtainable rectangles from the training. compute_box_dimensions dlib/image_processing.h dlib/image_processing/detection_template_tools_abstract.h This function is a tool for computing a rectangle with a particular width/height ratio and area. create_single_box_detection_template dlib/image_processing.h dlib/image_processing/detection_template_tools_abstract.h This function is a tool for creating a detection template usable by the scan_image_pyramid object. This particular function creates a detection template with exactly one feature extraction region. create_overlapped_2x2_detection_template dlib/image_processing.h dlib/image_processing/detection_template_tools_abstract.h This function is a tool for creating a detection template usable by the scan_image_pyramid object. This particular function creates a detection template with four overlapping feature extraction regions. create_grid_detection_template dlib/image_processing.h dlib/image_processing/detection_template_tools_abstract.h This function is a tool for creating a detection template usable by the scan_image_pyramid object. This particular function creates a detection template with a grid of feature extraction regions. determine_object_boxes dlib/image_processing.h dlib/image_processing/scan_image_pyramid_tools_abstract.h The scan_image_pyramid object represents a sliding window classifier system. For it to work correctly it needs to be given a set of object boxes which define the size and shape of each sliding window and these windows need to be able to match the sizes and shapes of targets the user wishes to detect. Therefore, the determine_object_boxes() routine is a tool for computing a set of object boxes which can meet this requirement. setup_grid_detection_templates dlib/image_processing.h dlib/image_processing/scan_image_pyramid_tools_abstract.h This routine uses determine_object_boxes to obtain a set of object boxes and then adds them to a scan_image_pyramid object as detection templates. It also uses create_grid_detection_template to create each feature extraction region. Therefore, the detection templates will extract features from a regular grid inside each object box. setup_grid_detection_templates_verbose dlib/image_processing.h dlib/image_processing/scan_image_pyramid_tools_abstract.h This function is identical to setup_grid_detection_templates except that it also outputs information regarding the selected detection templates to standard out. object_detector_ex.cpp.html train_object_detector.cpp.html setup_hashed_features dlib/image_processing.h dlib/image_processing/setup_hashed_features_abstract.h This is a tool for configuring the hashed_feature_image or binned_vector_feature_image object with a random projection hash. object_detector_ex.cpp.html train_object_detector.cpp.html randomly_sample_image_features dlib/statistics.h dlib/statistics/image_feature_sampling_abstract.h Given a feature extractor such as the hog_image, this routine selects a random subsample of local image feature vectors from a set of images. make_uniform_lbp_image dlib/image_transforms.h dlib/image_transforms/lbp_abstract.h This function extracts the uniform local-binary-pattern feature at every pixel of an image and stores the output in a new image object. We use the idea of uniform LBPs from the paper:
Face Description with Local Binary Patterns: Application to Face Recognition by Ahonen, Hadid, and Pietikainen.
extract_histogram_descriptors dlib/image_transforms.h dlib/image_transforms/lbp_abstract.h This function extracts histograms of pixel values from a set of windows in an image and returns the histograms. extract_uniform_lbp_descriptors dlib/image_transforms.h dlib/image_transforms/lbp_abstract.h Extracts histograms of uniform local-binary-patterns from an image. The histograms are from densely tiled windows that do not overlap and cover all of the image. We use the idea of uniform LBPs from the paper:
Face Description with Local Binary Patterns: Application to Face Recognition by Ahonen, Hadid, and Pietikainen.
extract_highdim_face_lbp_descriptors dlib/image_transforms.h dlib/image_transforms/lbp_abstract.h This function extracts the high-dimensional LBP feature described in the paper:
Blessing of Dimensionality: High-dimensional Feature and Its Efficient Compression for Face Verification by Dong Chen, Xudong Cao, Fang Wen, and Jian Sun