Studying the world's rapidly-dwindling biodiversity and educating the public about these issues is an important task for botanists. One of the primary ways to help with this task is to automate the tedious and error-prone process of identifying existing plant species. Computer vision techniques are now sufficiently sophisticated that we can do this reliably using an electronic field guide. The guide runs as an application on a mobile device such as the iPhone, allowing users to take photos which are sent to a server where they are automatically recognized. Including high-quality photos taken by the non-profit Finding Species and expert curation by botanists at the Smithsonian Institution, this field guide becomes a state-of-the-art and authoritative resource for both botanists and the general public.

The benefits to the end-user are obvious and significant: they now have available at their fingertips access to a field guide with high-quality, attractive photos, advanced search capabilities, and automatic recognition of all plant species in their area. The app also includes compelling games that help train users in learning to recognize and distinguish different plant species on the basis of their leaves, fruits, or flowers. The whole package also makes a much better introduction to botany and taxonomy for children. Existing, printed field guides, while adequate for recognition by adults and those with experience in classification, are often too difficult for children to use. Perhaps even more importantly, they are not very inviting to kids, and the diagramatic images of plants they contain are neither compelling nor life-like.

Leafsnap turns users into citizen scientists, automatically sharing images, species identifications, and geo-coded stamps of species locations with a community of scientists who will use the stream of data to map and monitor the ebb and flow of flora nationwide.

The recognition process, developed by the research groups of Peter Belhumeur at Columbia University and David Jacobs at the University of Maryland, consists of:

1. Segmenting the image to obtain a binary image separating the leaf from the background. This is currently implemented using an Expectation-Maximization framework, estimating foreground and background color distributions in the HSV colorspace.
2. Extracting features from the binarized image for compactly and discriminatively representing the shape of the leaf. We use histograms of curvature over scale as the feature representation, robustly and efficiently implemented using integral measures of curvature.
3. Comparing the features to those from a labeled database of leaf images and returning the species with the closest matches. Due to the discriminative power of the features and the size of our labeled dataset, we use a simple nearest neighbor (NN) approach with the L₁-norm.