Over the last decade, new applications in computer vision and computational photography have arisen due to earlier advances in methods for detecting human faces in images, including face detection-based autofocus and white balancing in cameras, new methods for sorting and retrieving images in digital photo management software, systems for automatic face recognition, etc. Face detectors usually return the image location of a rectangular bounding box containing a face. Yet, all of the above mentioned applications, as well as numerous ones yet to be conceived, would benefit from the accurate detection and localization of face parts -- e.g., eyebrow corners, eye corners, tip of the nose, mouth corners, chin -- within the specified bounding box.

In this work, we present a novel approach to localizing parts in images of human faces. The approach combines the output of local detectors with a non-parametric set of global models for the part locations based on over one thousand hand-labeled exemplar images. By assuming that the global models generate the part locations as hidden variables, we derive a Bayesian objective function. This function is optimized using a consensus of models for these hidden variables. The resulting localizer handles a much wider range of expression, pose, lighting and occlusion than prior ones. We show excellent performance on a new dataset titled "Labeled Face Parts in the Wild (LFPW)" gathered from the internet and show that our localizer achieves state-of-the-art performance on the less challenging BioID dataset.

This research was performed at Kriegman-Belhumeur Vision Technologies and was funded by the CIA through the Office of the Chief Scientist.