In this work, we advance the state-of-the-art for face verification ("are these two images of the same person?") in uncontrolled settings with non-cooperative subjects. To this end, we present two novel and complementary methods for face verification. Common to both methods is the idea of extracting and comparing "high-level" visual features, or traits, of a face image that are insensitive to pose, illumination, expression, and other imaging conditions. Our first method -- based on attribute classifiers -- uses binary classifiers trained to recognize the presence, absence, or degree of describable visual attributes (gender, race, age, hair color, etc.). Our second method -- based on simile classifiers -- removes the manual labeling required to train attribute classifiers. The simile classifiers are binary classifiers trained to recognize the similarity of faces, or regions of faces, to specific reference people. The idea is to automatically learn similes that distinguish a person from the general population. An unseen face might be described as having a mouth that looks like Barack Obama's and a nose that looks like Owen Wilson's.

Comparing two faces is simply a matter of comparing trait vectors (i.e., from the attribute and/or simile classifiers). We present experimental evaluation results on the challenging Labeled Faces in the Wild (LFW) data set. This data set is remarkable in its variability, exhibiting all of the differences mentioned above. Remarkably, both the attribute and simile classifiers achieve state-of-the-art results on the LFW "restricted images" benchmark, and a hybrid of the two results in a 31.68% drop in error rates compared to the previous best. To our knowledge, this is the first time that a list of such visual traits have been used for face verification. For testing beyond the LFW data set, we introduce PubFig -- a new data set of real-world images of public figures (celebrities and politicians) acquired from the internet. The PubFig data set is both larger (60,000 images) and deeper (on average 300 images per individual) than existing data sets, and allows us to present verification results broken out by pose, illumination, and expression. Finally, we measure human performance on LFW, showing that humans do very well on it -- given image pairs, verification of identity can be performed almost without error.

This research was funded in part by NSF award IIS-03-25867 and ONR award N00014-08-1-0638. We are grateful to Omron Technologies for providing us the OKAO face detection system.