Localizing Parts of Faces Using a Consensus of Exemplars

Keywords: face parts, localization, fiducial points, exemplars, non-parametric, real-world
Fall 2009 - Summer 2011
teaser image


Over the last decade, new applications in computer vision and computational photography have arisen due to earlier advances in methods for detecting human faces in images, including face detection-based autofocus and white balancing in cameras, new methods for sorting and retrieving images in digital photo management software, systems for automatic face recognition, etc. Face detectors usually return the image location of a rectangular bounding box containing a face. Yet, all of the above mentioned applications, as well as numerous ones yet to be conceived, would benefit from the accurate detection and localization of face parts -- e.g., eyebrow corners, eye corners, tip of the nose, mouth corners, chin -- within the specified bounding box.

In this work, we present a novel approach to localizing parts in images of human faces. The approach combines the output of local detectors with a non-parametric set of global models for the part locations based on over one thousand hand-labeled exemplar images. By assuming that the global models generate the part locations as hidden variables, we derive a Bayesian objective function. This function is optimized using a consensus of models for these hidden variables. The resulting localizer handles a much wider range of expression, pose, lighting and occlusion than prior ones. We show excellent performance on a new dataset titled "Labeled Face Parts in the Wild (LFPW)" gathered from the internet and show that our localizer achieves state-of-the-art performance on the less challenging BioID dataset.

This research was performed at Kriegman-Belhumeur Vision Technologies and was funded by the CIA through the Office of the Chief Scientist.


  • "Localizing Parts of Faces Using a Consensus of Exemplars,"
    IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI),
    vol. 35, no. 12, pp. 2930--2940, December 2013.
  • "Localizing Parts of Faces Using a Consensus of Exemplars,"
    Proceedings of the 24th IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
    June 2011.


Labeled Face Parts in the Wild (LFPW)

Labeled Face Parts in the Wild (LFPW):

1,432 Images with 29 points labeled on each image


Labeled face parts

Labeled face parts:

One of the images in LFPW. Overlaid, we show hand-labeled points obtained using MTurk. The points are numbered to match graphs (see below).
Asymmetry comparison of LFPW and BioID datasets

Asymmetry comparison of LFPW and BioID datasets:

The distribution of the asymmetry measure over images in the LFPW (blue) and BioID (red) datasets. Note that BioID has mostly symmetric faces, as seen by the large peak near the y-axis. In contrast, our new LFPW dataset has a much broader distribution of symmetries, highlighting its difficulty.
Comparison of our localizer accuracy vs human labelers

Comparison of our localizer accuracy vs human labelers:

This graph shows the mean error of our face part localizer on the LFPW dataset compared to the mean variation in human labeling. The points are numbered according to the earlier figure (see above). The error is the fraction of inter-ocular distance. Our detector is almost always more accurate.
Comparison of our localizer accuracy vs other systems

Comparison of our localizer accuracy vs other systems:

This graph shows a comparison of our localizer's accuracy compared against a commercial off-the-shelf detector and the detector of [Everingham et al. 2006]. Our detector is roughly twice as accurate as both.
Performance on LFPW

Performance on LFPW:

Cumulative error distribution of our system on the LFPW dataset (blue line) compared to locations predicted using the face detector box (green dashed line) or found with just our local detectors (red dashed line). Note that x-axis has a different scale from the previous graph.
Comparison of our accuracy on BioID vs other published results

Comparison of our accuracy on BioID vs other published results:

Cumulative error distribution curves comparing our system to several others on the BioID dataset. All comparative results are from [Valstar et al. 2010]. We outperform all previously published results.
Results on LFPW

Results on LFPW:

Images from the new Labeled Face Parts in the Wild (LFPW) dataset, along with parts located by our detector. We perform very well on this extremely challenging, real-world dataset, which contains large variation in pose, illumination, expression, image quality, severe occlusions, and all the other confounding factors that makes this a difficult problem.
Results on BioID

Results on BioID:

Images from BioID, along with parts localized by our detector. Note the much more controlled images present in this dataset, as compared to the more realistic LFPW dataset.