Using SIFT or SURF for comparing images of human poses

Problem description

Given a query image Q of a human person / pose find

(i) similar images I1,…,Im in a database of images of human poses

or

(ii) similar patches (image parts) within such a database

Method

We try out SIFT/SURF detectors/descriptors + straightforward NN matching techniques (e.g. FLANN) for solving the above problem. For our experiments we use a reference implementaton for the

  1. interest point detection
  2. interest point description
  3. descriptor matching

Namely: OpenCV's detector/descriptor/matcher framework which provides all you need already ready-to-use.

OpenCV now comes with a easy to use sample program which you find at

<OpenCVInstallationDir>\samples\cpp\matching_to_many_images.cpp

Call it with

matching_to_many_images.exe [detectorType] [descriptorType] [matcherType] [queryImage] [fileWithTrainImages] [dirToSaveResImages]

e.g.

matching_to_many_images.exe SURF SURF FlannBased myquery.png txtfile_with_list_of_images_to_compare_with.txt C:\resultdir

Here is the description what this sample program does:

This is a sample on matching descriptors detected on one image to
descriptors detected in image set. So we have one query image and
several train images. For each keypoint descriptor of query image the
one nearest train descriptor is found the entire collection of train
images. To visualize the result of matching we save images, each of
which combines query and train image with matches between them (if
they exist). Match is drawn as line between corresponding points.
Count of all matches is equal to count of query keypoints, so we have
the same count of lines in all set of result images (but not for each
result (train) image).

Example query image:

Database image #1,#2,#3:

Resulting (SURF) descriptor matches found by the matching_to_many_images.exe tool:

You can see, we have a good change to find the query image (CD box with cover) in the 3 scenes again - despite of translation, some rotation and scaling differences.

But the central question here is: is the SIFT/SURF descriptor also the right descriptor type when trying to compare images of human poses?

The background of this question is that the gradients on which the SIFT/SURF descriptor are based on can relatively fast change on images of humans (due to articulation - non-rigid objects! and clothings) compared to the quite stable gradients on rigid objects!

The SIFT/SURF descriptors were developed for rigid objects and tests like the above one were done for rigid objects and not for non-rigid objects (humans, animals,…)

Experiment #1: find most similar descriptors in database for each query image descriptor

Query image:

Database image #1 (same person!),#2,#3,#4:

Found (SURF) descriptor matches:

Experiment #2: show only really similar descriptors

While OpenCV's sample tool matching_to_many_images.exe is forced to show for each descriptor of the query image the most similar descriptor in the set of database image descriptors, there is also another nice OpenCV sample program, which is called matcher_simple.exe which only compares two images and only shows really similar descriptors.

For each descriptor computed in the first image, matcher_simple.exe finds the most similar descriptor in second image (for this, there always starts a line at all descriptors in image 1, but not necessarily in image2). Actually, all the lines should go - more or less - in the same direction, because it is the same person and nearly the same pose (!). But this is not the case.

Conclusion

The descriptor matches between the query image of the human person and the database images are not very helpful. Mostly there are wrong! This is probably due to the fact, that there is much variance due to e.g.

  1. clothing differences
  2. small articulation differences
  3. lightening differences

between the query image and the database images which leads to different gradients and therefore to different SIFT/SURF descriptors.

The SIFT/SURF descriptors cannot help us here to find the corresponding body parts.

Alternative descriptors?

 
public/using_sift_or_surf_for_comparing_images_of_human_poses.txt · Last modified: 2011/11/03 12:12 (external edit) · []
Recent changes RSS feed Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki