tombone's blog

drawing sexy graphs in matlab

2012-01-27T16:06:00.002-05:00

Everybody who loves computer sciences loves graphs. But the fat 'n juicy graphs, the ones with complex structure you just gotta visualize. To enjoy these beautiful data structures, the hackers at AT&T gave us, the world, Graphviz as a powerful tool for visualizing complex graphs in two dimensions. I do a lot of stuff in Matlab, so I've put my simple graphviz matlab wrappers up on Github so everybody can enjoy them. I do a lot of stuff with graphs...

My repository, which I'm already using as a submodule in many of my projects, can be found here:
https://github.com/quantombone/graphviz_matlab_magic

Here is a matlab script (included as a Github gist), which should be ran in an empty directory, and it will download a nice mat file plus clone my repo and show the following nice graph. I perform two graphviz passes where the first one is used to read the graphviz coordinates (from the sfdp embedding) and use Matlab's jet colormap to color the edges based on distances in this space. In other words, nearby nodes which are connected will be connected by red (hot) edges and faraway nodes will be connected by blue (cold) edges.

The matrix visualized comes from an electromagnetic model, the details can be found here: http://www.cise.ufl.edu/research/sparse/matrices/Bai/qc324.html

The original picture generated by Yifan Hu is here for comparison:

Enjoy
--Tomasz

100,000+ page views on my computer vision blog

2012-01-10T23:46:00.000-05:00

I like high-risk / high-reward activity. While some say that this is my temperament (perhaps a vestige of youth?) I simply say: "that's how I roll." Maybe I was too young when I read Kuhn's Structure of Scientific Revolutions, or maybe I was born with iconoclastic ideals, but I earnestly believe that life is too short to always do what you've been told. One of my favorite maxims is the following: "The only limits we have are the ones we impose upon ourselves."

I took a gamble when I started this blog, blurring the line between all things related to computer vision, philosophy, artificial intelligence, machine learning, and other fun things which constitute my intellectual life. During my PhD I was even discouraged from blogging, because "my superiors" incessantly reminded me that "you get famous by writing CVPR papers" and not by wasting time maintaining a "cute" blog. Today I'd like to argue that my adventure in blogging has not been a failure at all!

I had multiple reasons for wanting to blog, several of which I list below:

I wanted to practice my writing, and what better way to practice writing than by writing!
I wanted an outlet to discuss certain ideas which I find invaluable in my pursuit of building intelligence, but which aren't necessarily publishable. On my blog I am the sole contributor, the sole editor. If you don't like what I have to say, start your own blog. I don't need anonymous reviews, the CVPR submission process stresses me out enough for one lifetime.
I wanted a medium to advertise my own work as well other works which I find important for graduate students in Computer Vision to know about.
I wanted to expose the field of Computer Vision to a broader audience and hopefully get others excited about this amazing research field.

Today I'm glad to announce that according to statcounter, my computer vision blog has reached over 100,000 views. In an absolute sense, this really is nothing to be excited about. By since my CMU homepage has approximately 30,000 views, this means that my blog is 3x as popular as my academic homepage! Next goal: 1,000,000 page views!

I actually meet more people that know me through my blog than through my research papers, even though I put in 100x the effort in doing the research behind those papers. I don't plan on taking up blogging full time anytime soon, but it feels good to know that my blogging adventure has paid off.

Here are some of the top keywords which have been used to find my blog:

computer vision blog

tombone

cmu computer vision blog

newton fractal matlab

tombone's blog

Here are some of my most popular blog posts of all time:

Computer Vision is Artificial Intelligence

The vision hacker culture at Google

graph visualizations as sexy as fractals

Simple Newton's Method Fractal code in MATLAB

Kinect Object Datasets: Berkeley's B3DO, UW's RGB-D, and NYU's Depth Dataset

I encourage anybody who reads my blog to shoot me a quick "yo what's up!" at a local conference or where ever else our paths might cross. I also encourage everybody to suggest the types of things they would like to read about on my blog.

learning to "borrow" examples for object detection. Lim et al, NIPS 2011

2011-12-13T21:07:00.000-05:00

Let's say you want to train a cat detector... If you're anything like me, then you probably have a few labeled cats (~100), as well as a source of non-cat images (~1000). So what do you do when you can't get any more labeled cats? (Maybe Amazon's Mechanical Turk service was shut down by the feds, you've got a paper deadline in 48 hours, and money can't get you out of this dilemma.)

Answer:
1) Realize that there are some labeled dogs/cows/sheep in your dataset!
2) Transform some of the dogs/cows/sheep in your dataset to make them look more like cats. Maybe some dogs are already sufficiently similar to cats! (see cheezburger.com image below)
3) Use a subset of those transformed dogs/cows/sheep examples as additional positives in your cat detector!

Some dogs just look like cats! (and vice-versa)

Image courtesy of cheezburger.com

Using my own internal language, I view this phenomenon as "exemplar theft." But not the kind of theft which sends you to prison, 'tis the kind of theft which gives you best-paper prizes at your local conference.

Note that this was the answer provided by the vision hackers at MIT in their most recent paper, "Transfer Learning by Borrowing Examples for Multiclass Object Detection," which was just presented at this year's big machine learning-oriented NIPS conference, NIPS 2011. See the illustration from the paper below, which depicts this type of "example borrowing"-sharing for some objects in the SUN09 dataset.

The paper empirically demonstrates that instead of doing transfer learning (also known as multi-task learning) the typical way (regularizing weight vectors towards each other), it is beneficial to simply borrow a subset of (transformed) examples from a related class. Of course the problem is that we do not know apriori which categories to borrow from, nor which instances from those categories will give us a gain in object detection performance. The goal of the algorithm is to learn which categories to borrow from, and which examples to borrow. Not all dogs will help the cat detector.

Here are some examples of popular object categories, the categories from which examples are borrowed, and the categories from which examples are shared once we allow transformations to happen. Notice the improvement in AP (the higher the average precision the better) when you allow sharing.

They also looked at what happens if you want to improve a single category badass detector on one particular dataset, such as the PASCAL VOC. Note that these days just about everybody is using the one-and-only "badass detector" and trying to beat it in its own game. These are the different ways you'll hear people talk about the Latent-SVM-based Deformable Part Model baseline. "badass detector"="state-of-the-art detector"="Felzenszwalb et al. detector"="Pedro's detector"="Deva's detector","Pedro/Deva detector","LDPM detector","DPM detector"

Even if you only care about your favourite dataset, such as PASCAL VOC, you're probably willing to use additional positive data points from another dataset. In their NIPS paper, the MIT hackers show that simply concatenating datasets is inferior to their clever example borrowing algorithm (mathematical details are found in the paper, but feel free to ask me detailed questions in the comments). In the figure below, the top row shows cars from one dataset (SUN09), the middle row shows PASCAL VOC 2007 cars, and the bottom row shows which example the SUN09-car detector wants to borrow from PASCAL VOC.

Here the the cross-dataset generalization performance on the SUN09/PASCAL duo. These results were inspired by the dataset bias work of Torralba and Efros.

In case you're interested, here is the full citation for this excellent NIPS2011 paper:

Joseph J. Lim, Ruslan Salakhutdinov, and Antonio Torralba. "Transfer Learning by Borrowing Examples for Multiclass Object Detection," in NIPS 2011. [pdf]

To get a better understanding of Lim et al's paper, it is worthwhile going back in time to CVPR2011 and taking a quick look the following paper, also from MIT:

Ruslan Salakhutdinov, Antonio Torralba, Josh Tenenbaum. "Learning to Share Visual Appearance for Multiclass Object Detection," in CVPR 2011. [pdf]

Of course, these authors need no introduction (they are all professors at big-time institutions). Ruslan just recently became a Professor and is now back on home turf (where he got his PhD) in Toronto, where he is likely to become the next Hinton. In my opinion, this "Learning to share" paper was one of the best papers of CVPR 2011. In this paper they introduced the idea of sharing across rigid classifier templates, and more importantly learning a tree to organize hundreds of object categories. The tree defines how the sharing is supposed to happen. The root note is global and shared across all categories, the mid-level nodes can be interpreted as super-categories (i.e., animal, vehicle), and the leaves are the actual object categories (e.g., dog, chair, person, truck).

The coolest thing about the paper is that they use a CRP (chinese restaurant process) to learn a tree without having to specify the number of super-categories!

Finally, we can see some learned weights for three distinct object categories: truck, van, and bucket. Please see the paper if you want to learn more about sharing -- the clarity of Ruslan's paper is exceptional.

In conclusion, it is pretty clear everybody wants some sort of visual memex. (It is easy to think of the visual memex as a graph where the nodes are individual instances and the edges are relationships between these entities) Sharing, borrowing, multi-task regularization, exemplar-svms, and a host of other approaches are hinting at the breakdown of the traditional category-based way of approaching the problem of object recognition. However, our machine learning tools were designed for supervised machine learning with explicit class information. So what we, the researchers do, is try to break down those classical tools so that we can more effectively exploit the blurry line between not-so-different object categories. At the end of the day, rigid categories can only get us so far. Intelligence requires interpretation at multiple and potentially disparate levels. When it comes to intelligence, the world is not black and white, there are many flavours of meaningful image interpretation.

Graphics meets Big Data meets Machine Learning

2011-12-06T17:20:00.001-05:00

We've all played Where's Waldo as children, and at least for me it was quite a fun game. So today let's play an image-based Big Data version of Where's Waldo. I will give you a picture, and you have to find it in a large collection of images! This is a form of image retrieval, and this particular formulation is also commonly called "image matching."

The only catch is that you are only given one picture, and I am free to replace the picture with a painting or a sketch. Any two-dimensional pattern is a valid query image, but the key thing to note is that there is only a single input image. Life would be awesome if Google's Picasa had this feature built in!

The classical way of solving this problem is via a brute-force nearest neighbor algorithm, an algorithm which won't match pixel pattern directly, but an algorithm which will also use a state-of-the-art image descriptor such as GIST for comparison. Back in 2007, at SIGGRAPH, James Hays and Alexei Efros have shown this to work quite well once you have a very large database of images! But the reason why the database had to be so large is because a naive Nearest Neighbor algorithm is actually quite dumb. The descriptor might be cleverer than matching raw pixel intensities, but for a machine, an image is nothing but a matrix of numbers, and nobody told the machine which patterns in the matrix are meaningful and which ones aren't. In short, the brute-force algorithm works if there are similar enough images such that all parts of the input image will match a retrieved image. But ideally we would like the algorithm to get better matches by automatically figuring out which parts of the query image are meaningful (e.g., the fountain in the painting) and which parts aren't (e.g., the reflections in the water).

A modern approach to solve this issue is to collect a large set of related "positive images" and a large set of un-related "negative images" and then train a powerful classifier which can hopefully figure out the meaningful bits of the image. But in this approach the problem is twofold. First, working with a single input image it is not clear whether standard machine learning tools will have a chance of learning anything meaningful. The second issue, a significantly worse problem, is that without a category label or tag, how are we supposed to create a negative set?!? Exemplar-SVMs to the rescue! We can use a large collection of images from the target domain (the domain we want to find matches from) as the negative set -- as long as the "negative set" contains only a small fraction of potentially related images, learning a linear SVM with a single positive still works.

Here is an excerpt from a Techcrunch article which summarizes the project concisely:

"Instead of comparing a given image head to head with other images and trying to determine a degree of similarity, they turned the problem around. They compared the target image with a great number of random images and recorded the ways in which it differed the most from them. If another image differs in similar ways, chances are it’s similar to the first image. " -- Techcrunch

Abhinav Shrivastava, Tomasz Malisiewicz, Abhinav Gupta, Alexei A. Efros. Data-driven Visual Similarity for Cross-domain Image Matching. In SIGGRAPH ASIA, December 2011. Project Page

Here is a short listing of some articles which mention our research (thank Abhinav!).

http://techcrunch.com/2011/12/06/cmu-researchers-one-up-google-image-search-and-photosynth-with-visual-similarity-engine/

http://news.cs.cmu.edu/article.php?a=2858

http://futureoftech.msnbc.msn.com/_news/2011/12/06/9252228-computer-mimics-human-ability-to-match-images

http://www.physorg.com/news/2011-12-team-computerized-method-images-photos.html

http://nanopatentsandinnovations.blogspot.com/2011/12/carnegie-mellon-creates-computerized.html

http://www.sciencecodex.com/read/carnegie_mellon_creates_computerized_method_for_matching_images_in_photos_paintings_sketches-82595

http://www.cra.org/ccc/rh-imatch.php

An accidental face detector

2011-12-05T21:14:00.001-05:00

Disclaimer #1: I don't specialize in faces. When it comes to learning, I like my objectives to be convex. When it comes to hacking on vision systems, I like to tackle entry-level object categories.

Fun fact #1: Faces are probably the easiest objects in the world for a machine to localize/detect/recognize.

Note #1: I supplied the images, my algorithm supplied the red boxes.

Note #2: Sorry to all my friends who failed to get detected by my accidental face detector! (see below)

So I was hackplaying with some of my PhD thesis code over Thanksgiving, and I accidentally made a face detector. oops! I immediately ran to my screenshot capture tool and ran my code on my Mac desktop while browsing Google Images and Facebook. It seems to work pretty well on real faces as well as sketches/paintings of faces (see below)! I even caught two Berkeleyites (an Alyosha and a Jianbo), but you gotta find them for yourself. The detector is definitely tuned to frontal faces, but runs pretty fast and produces few false positives. Not too shabby for some midnight hackerdom.

Yes, I'm doing dense multiscale sliding windows here. Yes, I'm HoGGing the hell outta these images. Yes, I'm using a single frontal-face tuned template. And yes, I only used faces of myself to train this accidental face detector.

Note: If I've used one of your pictures without permission, and you would like a link back to your home on the interwebs, please leave a comment indicating the image and link to original.

Google Scholar, My Citations, a new paradigm for finding great Computer Vision research papers

2011-12-02T11:15:00.001-05:00

I have been finding great computer vision research papers by using Google Scholar for the past 2+ years. My recipe is straightforward and has two key ingredients. First, by finding new papers that cite one of my published papers, I automatically get to read papers which will be relevant to my own research interests. The best bit is that by using Google Scholar, I'm not limiting my search to a single conference -- Google finds papers from the raw web.

Second, I have a short list of superstar vision researchers (Jitendra Malik, among others) and I basically read anything and everything these gurus publish. Regularly visiting academic homepages is the best way to do this, but Google Scholar also lets me search by name. In addition, nobody lists on their homepage their papers' citation counts. This means if I visit a researcher's personal website, I have to make a decision as to what paper to read based on (title, co-authors, publication venue). But highly-cited papers are likely to be more important to read first. I believe that this is a good rule of thumb, and very important if you are new to the field.

I am really glad that Google finally let researchers make public profiles to view their papers and see their citations, etc. See Google Scholar blog for more information. I've been using statcounter to monitor my blog's visitors, and now I can use Google Scholar to monitor who is citing my research papers! I'm not claiming that the only way for me to read one of your papers is to cite one of my papers, but believe me, even if we never met at a vision conference, if you cited one one my papers there's a good chance I already know about your research :-) I would love to see Google Scholar Citations pages one day replace "my publications" sections on academic homepages...

My Citations screenshot

My only complaint with Google Scholar is that I can't seem to get it to recognize my two most recent papers. I have these papers listed on my homepage, so do my co-authors, but Google isn't picking them up!!! I manually added them to my Google My Citations page, and using Google Scholar I was able to find at least one other paper which cites on of these two papers.

I read the inclusion guidelines, and I'm still baffled. The PDFs are definitely over 5MB, but my older papers which were indexed by Google were also over 5MB. Dear Google, are you seriously not indexing my recent papers because they are over 5MB? It takes us, researchers, months of hard work to get our work out the door. We see the sun rise for weeks straight when we are in deadline-mode, and the conferences/journals give us size limitations -- we work hard to make our stuff fit within these limits (something like 20MB per PDF). And we, researchers, are crazy about Google and what it means for organizing the world's information -- naturally we are jumping on the Google Scholar bandwagon. I really hope there's some silly reason why I can't find my own papers using Google Scholar, but if I can't find my own work, that means others can't find my own work, and until I can be confident that Google Scholar is bug-free, I cannot give it my full recommendation.

Problematic papers for Google Scholar:

Abhinav Shrivastava, Tomasz Malisiewicz, Abhinav Gupta, Alexei A. Efros. Data-driven Visual Similarity for Cross-domain Image Matching. In SIGGRAPH ASIA, December 2011.

Tomasz Malisiewicz, Abhinav Gupta, Alexei A. Efros. Ensemble of Exemplar-SVMs for Object Detection and Beyond. In ICCV, November 2011.

If anybody has any suggestions (there's a chance I'm doing something wrong), or an explanation as to why my papers haven't been indexed, I would love to hear from you.

don't throw away old code: github-it!

2011-11-16T18:29:00.001-05:00

My thesis experiments on Exemplar-SVMs (my PhD thesis link: Note, 33MB) would have taken approximately 20 CPU years to finish. But not on a fat CMU cluster! Here is some simple code which helped make things possible in ~1month of 200+ cores of crunching. That scale of computation is not quite Google-scale computing, but it was a unforgettable experience as a CMU PhD student. I've recently had to go back to the SSH / GNU Screen method of starting scripts at MIT, since we do not have torque/pbs there, but I definitely use these scripts. Fork it, use it, change it, hack it, improve it, break it, learn from it, etc.

https://github.com/quantombone/warp_scripts

I used these scripts to drive the experiments in my Exemplar-SVM framework (also on Github).

The basic take home message is "do not throw away old code" which you found useful at some time. C'mon ex-phd students, I know you wrote a lot of code, you graduated and now you feel embarrassed to share your code. Who cares if you never had a chance to clean it up, if the world never gets to see it then it will die a silent death from lack of use. Just put it on Github, and let others take a look. Git is the world's best source control/versioning system. Its distributed nature makes it perfect for large-scale collaboration. Now with github sharing is super easy! Sharing is caring. Let's make the world a better place for hackerdom, one repository at a time. I've met some great hackers at MIT, such as the great cvondrick, who is still teaching me how to branch like a champ.

Mathematicians share proofs. Hackers share code. Embrace technology, embrace Github. If you ever want to hack with me, it is probably as important for you to know the basics of git as it is for you to be a master of linear algebra.

Additional Reading:

Distributed Version Control: The Future of History, an article about Git by some Kitware software engineers

scene recognition with part-based models at ICCV 2011

2011-11-08T21:05:00.001-05:00

Today, I wanted to point everyone's attention to a super-cool paper from day 1 of this year's ICCV 2011 Conference. Megha Pandey is the lead on this, and Lana Lazebnik (of spatial pyramid fame) is the seasoned vision community member supervising this research. The idea is really simple (and simplicity is a plus!): train a latent deformable part-based model for scenes. Some of the scene models look really cool, and I encourage everybody interested in scene recognition to take a look.

A Part-based Scene Model

One of the reasons why I like this paper is because just like our SIGGRAPH ASIA 2011 paper on cross-domain image matching, they are using HOG features to represent scenes and applying these models in a sliding-window fashion. This is much different than the traditional image-to-feature-vector mapping used in systems based on the GIST descriptor. These types of approaches allow the detection of a scene inside another image! Framing issues are elegantly handled by allowing the model to slide.

Scene Recognition and Weakly Supervised Object Localization with Deformable Part-Based Models. Megha Pandey and Svetlana Lazebnik. Proceedings of the IEEE International Conference on Computer Vision, 2011. Project Page [pdf]

Abstract: Weakly supervised discovery of common visual structure in highly variable, cluttered images is a key problem in recognition. We address this problem using deformable part-based models (DPM’s) with latent SVM training. These models have been introduced for fully supervised training of object detectors, but we demonstrate that they are also capable of more open-ended learning of latent structure for such tasks as scene recognition and weakly supervised object localization. For scene recognition, DPM’s can capture recurring visual elements and salient objects; in combination with standard global image features, they obtain state-of-the-art results on the MIT 67-category indoor scene dataset. For weakly supervised object localization, optimization over latent DPM parameters can discover the spatial extent of objects in cluttered training images without ground-truth bounding boxes. The resulting method outperforms a recent state-of-the-art weakly supervised object localization approach on the PASCAL-07 dataset.

Weakly Supervised Object Localization (see paper for details)

the fun begins at ICCV 2011

2011-11-05T03:27:00.001-05:00

All the cool vision kids are going, so why aren't you?
http://www.iccv2011.org/

This will be my first ICCV ever! and my first trip to Spain! Seriously though, if you need to find me over the next week, come to Barcelona. There are lots of great papers out this year and I'll be sure the write about the few which I find interesting (and haven't already blogged about). If you want to learn more about the craziness behind ExemplarSVMs, or just to say 'Hi' don't hesitate to find me walking around the conference. I'll be there during all the workshop days too.

If anybody has a favourite ICCV2001 paper they want me to look and perhaps write something about (hardcore object recognition please -- I don't care about illumination models), please send me your requests (in the comments below).

Google Internship in Vision/ML

2011-10-26T12:05:00.000-05:00

Disclaimer: the following post is cross-posted from Yaroslav's "Machine Learning, etc" blog. Since I always rave about my experiences at Google as an intern (did it twice!), I thought some of my fellow readers would find this information useful. If you are a vision PhD student at CMU or MIT, feel free to ask me more about life at Google. If you have questions regarding the following internship offer, you'll have to ask Yaroslav.

Original post at: http://yaroslavvb.blogspot.com/2011/10/google-internship-in-visionml.html

My group has intern openings for winter and summer. Winter may be too late (but if you really want winter, ping me and I'll find out feasibility). We use OCR for Google Books, frames from YouTube videos, spam images, unreadable PDFs encountered by the crawler, images from Google's StreetView cameras, Android and few other areas. Recognizing individual character candidates is a key step in OCR system. One that machines are not very good at. Even with 0 context, humans are better. This shall not stand!

For example, when I showed the picture below to my Taiwanese coworker he immediately said that these were multiple instance of Chinese "one".

Here are 4 of those images close-up. Classical OCR approaches, have trouble with these characters.

This is a common problem for high-noise domain like camera pictures and digital text rasterized at low resolution. Some results suggest that techniques from Machine Vision can help.

For low-noise domains like Google Books and broken PDF indexing, shortcomings of traditional OCR systems are due to
1) Large number of classes (100k letters in Unicode 6.0)
2) Non-trivial variation within classes
Example of "non-trivial variation"

I found over 100k distinct instances of digital letter 'A' from just one day's crawl worth of documents from the web. Some more examples are here

Chances are that the ideas for human-level classifier are out there. They just haven't been implemented and tested in realistic conditions. We need someone with ML/Vision background to come to Google and implement a great character classifier.

You'd have a large impact if your ideas become part of Tesseract. Through books alone, your code will be run on books from 42 libraries. And since Tesseract is open-source, you'd be contributing to the main OCR effort in the open-source community.

You will get a ton of data, resources and smart people around you. It's a very low bureocracy place. You could run Matlab code on 10k cores if you really wanted, and I know someone who has launched 200k core jobs for a personal project. The infrastructure also makes things easier. Google's MapReduce can sort a petabyte of data (10 trillion strings) with 8000 machines in just 30 mins. Some of the work in our team used features coming from distributed deep belief infrastructure.

In order to get an internship position, you must pass general technical screen that I have no control of. If you are interested in more details, you could contact me directly. -- Yaroslav

(the link to apply is usually here, but now it's down, will update when it's fixed)

NIPS 2011 preview: person grammars and machines-in-the-loop for video annotation

2011-10-25T16:20:00.001-05:00

Ross Girshick, Pedro Felzenszwalb, David McAllester

Object Detection with Grammar Models

To appear in NIPS 2011 pdf

Today, I want to point out two upcoming NIPS papers which might be of interest to the Computer Vision community. First, we have a person detection paper from the hackers who brought you Latent Discriminatively Trained Part-based Models (aka voc-release-3.1 and voc-release-4.0). I personally don't care for grammars (I think exemplars are a much more data-driven and computation-friendly way of modeling visual concepts), but I think any paper with Pedro on the author list is really worth checking out. Maybe after I digest all the details, I'll jump on the grammar bandwagon (but I doubt it). Also of note, is the fact that Pedro Felzenszwalb has relocated to Brown University.

The second paper, is by Carl Vondrick and Deva Ramanan (also of latent-svm fame). Carl is the author of vatic and a fellow vision@github hacker. Carl, like myself, has joined Antonio Torralba's group at MIT this fall. He just started his PhD, so you can only expect the quality of his work to increase without bound over the next ~5 years. vatic is an online, interactive video annotation tool for computer vision research that crowdsources work to Amazon's Mechanical Turk. Vatic makes it easy to build massive, affordable video data sets and can be deployed on a cloud. Written in Python + C + Javascript, vatic is free and open-source software. The video below showcases the power of vatic.

In this paper, Vondrick et al. use active learning to select the frames which require human annotation. Rather than simply doing linear interpolation between frames, they are truly putting the "machine-in-the-loop." When doing large-scale video annotation, this approach can supposedly save you tens of thousands of dollars.

Carl Vondrick and Deva Ramanan. "Video Annotation and Tracking with Active Learning" Neural Information Processing Systems (NIPS) Granada, Spain, December 2011. [paper] [slides]

Kinect Object Datasets: Berkeley's B3DO, UW's RGB-D, and NYU's Depth Dataset

2011-10-06T13:26:00.000-05:00

Why Kinect?

www.pirobot.org

The Kinect, made by Microsoft, is starting to become quite a common item in Robotics and Computer Vision research. While the Robotics community has been using the Kinect as a cheap laser sensor which can be used for obstacle avoidance, the vision community has been excited about using the 2.5D data associated with the Kinect for object detection and recognition. The possibility of building object recognition systems which have access to pixel features as well as 2.5D features is truly exciting for the vision hacker community!

Berkeley's B3DO

First of all, I would like to mention that it looks like the Berkeley Vision Group jumped on the Kinect bandwagon. But the data collection effort will be crowdsourced -- they need your help! They need you to use your Kinect to capture your own home/office environments and upload it to their servers This way, a very large dataset will be collected, and we, the vision hackers, can use machine learning techniques to learn what sofas, desks, chairs, monitors, and paintings look like. They Berkeley hackers have a paper on this at one of the ICCV 2011 workshops in Barcelona, here is the paper information:

kinectdata.com

A Category-Level 3-D Object Dataset: Putting the Kinect to Work
Allison Janoch, Sergey Karayev, Yangqing Jia, Jonathan T. Barron, Mario Fritz, Kate Saenko, Trevor Darrell
ICCV-W 2011
[pdf] [bibtex]

UW's RGB-D Object Dataset

On another note, if you want to use 3D for your own object recognition experiments then you might want to check out the following dataset: University of Washington's RGB-D Object Dataset. With this dataset you'll be able to compare against UW's current state-of-the-art.

In this dataset you will find RGB+Kinect3D data for many household items taken from different views. Here is the really cool paper which got me excited about the RGB-D Dataset:
A Scalable Tree-based Approach for Joint Object and Pose Recognition
Kevin Lai, Liefeng Bo, Xiaofeng Ren, and Dieter Fox
In the Twenty-Fifth Conference on Artificial Intelligence (AAAI), August 2011.

NYU's Depth Dataset

I have to admit that I did not know about this dataset (created by by Nathan Silberman of NYU), until after I blogged about the other two datasets. Check out the NYU Depth Dataset homepage. However the internet is great, and only a few hours after posted this short blog post, somebody let me know that I left out this really cool NYU dataset. In fact, it looks like this particular dataset might be at the LabelMe-level regarding dense object annotations, but with accompanying Kinect data. Rob Fergus & Co strike again!

Nathan Silberman, Rob Fergus. Indoor Scene Segmentation using a Structured Light Sensor. To Appear: ICCV 2011 Workshop on 3D Representation and Recognition

plenoptica theoretica: fields vs. particles

2011-09-29T13:12:00.003-05:00

"All bodies together, and each by itself, give off to the surrounding air an infinite number of images which are all-pervading and each complete, each conveying the nature, colour and form of the body which produces it." --Leonardo da Vinci

Yesterday, Edward H. Adelson ("Ted Adelson") gave a lecture at MIT on the plenoptic function and its role in understanding (and unifying) early vision. Ted has been at MIT for quite some time. He is sometimes described as being (1/3 human vision, 1/3 computer vision, and 1/3 computer graphics) and was Bill Freeman's advisor.

What is the plenoptic function?

Etymology: Plenoptic comes from plenus+optic.
plenus: full, filled
optic: relating to eye or vision

Ted Adelson imagined a sort of unified field theory for vision -- instead of proposing a jungle of atoms such as edges, corners, and peaks, the plenoptic function offers a unifying principle under which color, texture, motion, etc. can all be viewed as gradients of the plenoptic function. The plenoptic function is a complete representation which contains, implicitly, a description of every possible photograph that could be taken of a particular space-time chunk of the world. Omniscience is to knowing as the plenoptic function is to seeing.

Ted remarked that if you asked him 20 years ago what he was working on in vision, you might have gotten a confusing answer. "Do you work on texture, motion, stereo, or illumination?" you might ask. "All of them. Aren't they the all the same thing?" he might reply. Ted argues that vision scientists in the 80s and early 90s tried to cut up the world of vision into neat little "particles" and would develop theories with their favorite particle -- here the particles are early vision concepts such as color, texture, and motion.

In their seminal paper on the plenoptic function, The Plenoptic Function and the Elements of Early Vision, Adelson and Bergen state that "the elemental operations of early vision involve the measurement of local change long various directions within the plenoptic function." As a theoretical device, the plenoptic function has left a long-standing impression on me. I first came across Ted's ideas back in 2006 -- thanks to Alyosha Efros' course on vision. Having just completed a BS in Physics, I was well aware of unified field theories in physics, and the plenoptic function seemed too cool to forget.

What the plenoptic function means to me
However, if the plenoptic function is the Maxwell's equations equivalent for early (low-level) vision, then what I'm ultimately after is the Schrodinger's equation of late (high-level) vision. In his lecture, Ted Adelson acknowledged that vision scientists have a sort of Atom Envy -- they envy the physicists who are able to understand the world in terms of a few fundamental ontologically meaningful entities. First of all, I like particles, but I have no apriori reason to be in the particle camp all of my life. Secondly, the plenoptic function was all about early vision, but my research in vision is all about high-level vision such as object recognition. I might be young and foolish, but the search for a "mind mechanics" has been a part of my research life (at least partially) since ~2003. Right now, my best shot at an answer is that exemplars and associations are the basic building blocks of high-level vision -- but unlike the British Empricisits (the champions of associationism), I would argue that the atomic building blocks of associations are object instances, and not ideas such as "roundness" and "blueness". Complex ideas are then the object categories which arise out of the interactions between these concrete elements of experience.

Conclusion
The Adelson and Bergen paper is a must read for anybody serious about vision. While it might not offer much in terms of "what next" in vision research, it is nevertheless a useful construct in thinking about vision. I get excited when it comes down to unifying principles and I wish there were more papers like this in vision, especially for high-level vision.

Kant's Intuitions, the intentional stance, and reverse-engineering the mind

2011-09-28T13:59:00.002-05:00

Immanuel Kant (1724-1804), Daniel Dennett (1942-), Josh Tenenbaum (~1971-)

“Thoughts without content are empty, intuitions without concepts are blind. The understanding can intuit nothing, the senses can think nothing. Only through their unison can knowledge arise.” -- Immanuel Kant

“We live in a world that is subjectively open. And we are designed by evolution to be "informavores", epistemically hungry seekers of information, in an endless quest to improve our purchase on the world, the better to make decisions about our subjectively open future.” -- Daniel Dennett

"For scientists studying how humans cometo understand their world, the central challenge is this: How do our minds get so much from so little? We build rich causal models,make strong generalizations, and construct powerful abstractions, whereas the input data are sparse, noisy, and ambiguous—in every way far too limited. A massive mismatch looms between the information coming in through our senses and the outputs of cognition." -- Josh Tenenbaum

Organizing by space(space, time, and physics)

There are two faculties of understanding which it isunlikely we have acquired from experience. The first is that of understanding objects as extended bodies in a 3D spaceand thus occupying some volume. I believeit is Kant argued best against the hardcore British Empiricists, whoproclaimed that experience is the sole originator of knowledge. Experiences are the pen strokes, which fill theEmpricisit’s tabula rasa. Kant argued (against Hume) that the concept of a spatially extended object is not acquired from experience– the very notion of experience requires that we already possess the notion ofan object in order to have a meaningful percept. It is as if the Empiricists failed toacknowledge that to make strokes on a sheet of paper, we need to already have apen. Kant’sintuitions are the pens of experience. The requirement of having suitable intuitions for grouping percepts into experiences is what Kant described as a form of transcendental idealism. “Objectness” is a faculty of humanunderstanding, not something acquired from experience. If you are a vision researcher, being aware of this can have drastic implications on your research programme.

It has also been argued that there are some primitivenotions of object dynamics, aka folk-physics, which can are possessed by very youngchildren. Given the uniformity of humanexperience (at least I have no ostensible reason to double that my colleague’sexperiences significantly differ from my own), and the diversity in our individual upbringing, it is also unlikely that folk-physics is learned from experience. However, I don't want to make any strong claims regarding folk-physics. I feel safe to say that Quantum Mechanics is another story -- it requires years of mathematics and thousands hours of deliberate problem solving to grasp.

Organizing by mind(psychology, mind, and intent)

The second faculty of understanding, which can be found inmany aspects of human intelligence, is that of understanding the world in termsof cognitive agents. Humans have anamazing capability when it comes to attributing stuff with having a mind. This way of thinking about the world is socommon and uniform among children all over the world, that the differences intheir upbringing cannot be reconciled with the uniformity of their capabilityto project humanness onto objects. Consider the following video (thanks J. Tenenbaum's videos/lectures for pointing this out).

We cannot just view this video os triangles, dots, and lines. Each one of understands the story in terms of a narrative based on agents and their intent. We are stimulated by the external world, we take asinput sense-data, and the brain helps us make sense of it -- it turns the hodgepodge of data into experience. But the brain is a mold, it conforms perceptsto some shape defined by the mold. Thesemolds are the faculties of understanding which let us understand things, it islike the faculties of understanding arebasis vectors onto which we project all input sense data. The data is weak and noisy, the priors are strong, andunderstanding is the result of their union. An experience without a proper basis is blind, it is just a ball ofpercepts. These faculties allow us tohave experience. The experiences,coupled with memory, allow us to obtain understanding – where understanding isthe relationship between a given experience and past experiences, either in theform of direct associations between currently-experienced-objects andpreviously-experienced-objects, or rules abstracted away frompreviously-experienced-objects being directly applied to the current sensedata.

What I am talking about is what philosopher Daniel Dennett refers to by the “intentional stance.” Given my background in AI and philosophy of mind, it is very likely thatDennett and I have had the same influences. I liketo juxtapose my ideas with those of the classical philosophers such asDescartes, Locke, Kant, Wittgenstein and Pinker -- I’m not sure how Dennett motivates his philosophy nor do I know against whose ideas he juxtaposes his own stance.

At MIT, J. Tenenbaum is pushing these ideas to the next level. I only wish there was more perception in his work -- toy worlds just don't do it for me. I want to build intelligent machines, and really cannot afford to sidestep the issue of perception. Here is a great talk by Josh Tenenbaum on reverse engineering the mind from NIPS 2010. Video is on videolectures.net, just click the link.

Implications for Artificial Intelligence and Machine Vision

Following Josh Tenenbaum, I think that a criticism of classical machine learning is long overdue. Machine Learning, as a field, has been spewing out hardcore empiricists. “Let me download your features, my machine learning algorithm will take care of the rest,” they say. It is like the glory is in the mathematics, which manipulates N-D vectors. But I argue that intelligence isn’t “in the calculus,” it is what the primitives in the calculus actually represent. As an undergraduate I proclaimed, “I am not a mathematician, I am a physicists. I care about the structure of the world, not the structure of proofs. “ As a graduate student I proclaimed, “The glory isn’t in the manipulation of vectors, the glory is understanding the what/why of encoding information about the world into vectors. I am a computer vision researcher, not a machine learning researcher.” That is why the view of the world as coming from K different classes is wrong – this is merely a convenient view if the statistician’s toolbox is at your disposal. It is all about structuring the input to match a researcher’s high-level intuitions about the world.

My first week at MIT: What is intelligence?

2011-09-09T13:50:00.000-05:00

In case anybody hasn't heard the news, I am no longer a PhD student at CMU. After I handed in my camera-ready dissertation, it didn't take long for my CMU advisor to promote me from his 'current students' to 'former students' list on his webpage. Even though I doubt there is anyplace in the world which can rival CMU when it comes to computer vision, I've decided to give MIT a shot. I had wanted to come to MIT for a long time, but 6 years ago I decided to choose CMU's RI over MIT's CSAIL for my computer vision PhD. Life is funny because the paths we take in life aren't dead-ends -- I'm glad I had a second chance to come to MIT.

In case you haven't heard, MIT is a little tech school somewhere in Boston. Lots of undergrads can be caught wearing math Tshirts and posters like the following can be found on the walls of MIT:

A cool (undergrad targeted) poster I saw at MIT

As of last week I'm officially a postdoc in CSAIL and I'll be working with Antonio Torralba and Aude Oliva. I've been closely following both Antonio's and Aude's work over the last several years and getting to work with these giants of vision will surely be a treat. In case you don't know what a postdoc is, it is a generic term used to describe post-PhD researchers with generally short term (1-3 year) appointments. People generally use the term Postdocotral Fellow or Postdoctoral Associate to describe their position in a university. I guess 3 years working on vision as an undergrad and 6 years of working on vision as a grad student just wasn't enough for me...

I've been getting adjusted to my daily commute through scenic Boston, learning about all the cool vision projects in the lab, as well as meeting all the PhD students working with Antonio. Today was the first day of a course which I'm sitting-in on, titled "What is intelligence?". When I saw a course offered by two computer vision titans (Shimon Ullman and Tomaso Poggio), I couldn't resist. Here is the information below:

What is intelligence?

"What is intelligence?" course homepage: http://web.mit.edu/9.s915/www/

Class Times:	Friday 11:00-2:00 pm
Units:	3-0-9
Location:	46-5193 (NOTE: we had to choose a bigger room)
Instructors:	Shimon Ullman and Tomaso Poggio

The class was packed -- we had to relocate to a bigger room. Much of today's lecture was given by Lorenzo Rosasco. Lorenzo is the Team Leader of IIT@MIT. Here is a blurb from IIT@MIT's website describe what this 'center' is all about:

The IIT@MIT lab was founded from an agreement between the Massachusetts Institute of Technology(MIT) and the Istituto Italiano di Tecnologia (IIT). The scientific objective is to develop novel learning and perception technologies – algorithms for learning, especially in the visual perception domain, that are inspired by the neuroscience of sensory systems and are developed within the rapidly growing theory of computational learning. The ultimate goal of this research is to design artificial systems that mimic the remarkable ability of the primate brain to learn from experience and to interpret visual scenes.

Another cool class offered this semester at MIT is Antonio Torralba's Grounding Object Recognition and Scene Understanding.

The vision hacker culture at Google ...

2011-08-24T08:39:00.000-05:00

I sometimes get frustrated when developing machine learning algorithms in C++. And since working in object recognition basically means you have to be a machine learning expert, trying something new and exciting in C++ can be extremely painful. I don't miss the C++ heavy workflow for vision projects at Google. C++ is great for building large-scale systems, but not for pioneering object recognition representations. I like to play with pixels and I like to think of everything as matrices. But programming languages, software engineering philosophies, and other coding issues aren't going to be today's topic. Today I want to talk about the one thing that is more valuable that is computers, and that is people. Not just people, but a community of people, and in particular the culture at Google -- in particular, vision@Google.

I miss being around the hacker culture at Google.

The people at Google aren't just hackers, they are Jedis when it comes to building great stuff -- and that is why I recommend a Google internship to many of my fellow CMU vision Robograds (fyi, Robograds are CMU Robotics Graduate Students). CMU-ers, like Googlers, like to build stuff. However, CMU-ers are typically younger.

Image from http://www.rabittooth.com/

What is a software engineering Jedi, you might ask? Tis' one who is not afraid of million cores, one who is not afraid of building something great. While little boys get hurt by the guns 'n knives of C++, Jedi use their tools like ninjas use their swords. You go into Google as a boy, you come out a man. NOTE: I do not recommend going to Google and just toying around in Matlab for 3 months. Build something great, find a Yoda-esque mentor, or at least strive to be a Jedi. There's plenty of time in grad school for Matlab and writing papers. If you get a chance to go to Google, take the opportunity to go large-scale and learn to MapReduce like the pros.

Every day I learn about more and more people I respect in vision and learning going to Google, or at least interning there (e.g., Andrej Karpathy who is starting his PhD@Stanford and Santosh Divvala who is a well-known CMU PhD student and vision hacker). And I really can't blame them for choosing Google over places like Microsoft for the summer. I can't think of many better places to be -- the culture is inimitable. I spent two summers at Jay Yagnik's group some of the great people I interned with are already full-time Googlers (e.g. Luca Bertelli and Mehmet Emre Sargin). And what is really great about vision@google is that these guys get to publish surprisingly often! Not just throw-away-code kind of publish, but stuff that fits inside large-scale systems -- stuff which is already inside Google products. The technology is often inside the Google product before the paper goes public! Of course it's not easy to publish at a place like Google because there is just way too much exciting large-scale stuff going on. Here is a short list of some cool 2010/2011 vision papers (from vision conferences) with significant Googler contributions.

Kernelized Structural SVM Learning

“Kernelized Structural SVM Learning for Supervised Object Segmentation”, Luca Bertelli, Tianli Yu, Diem Vu, Burak Gokturk, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition 2011.
[abstract] [pdf]

Finding Meaning on YouTube

“Finding Meaning on YouTube: Tag Recommendation and Category Discovery”, George Toderici,Hrishikesh Aradhye, Marius Pasca, Luciano Sbaiz, Jay Yagnik, Computer Vision and Pattern Recognition, 2010.
[abstract] [pdf]

Here is a very exciting and new paper from SIGGRAPH 2011. It is a sort of Visual Memex for faces -- congratulations on this paper, guys! Check out the video below.

Exploring Photobios Movie

Exploring Photobios from Ira Kemelmacher on Vimeo

Ira Kemelmacher-Shlizerman, Eli Shechtman, Rahul Garg, Steven M. Seitz. "Exploring Photobios." ACM Transactions on Graphics (SIGGRAPH), Aug 2011. [pdf]

Finally, here is a very mathematical paper with a sexy title from the vision@google team. It will be presented at the upcoming ICCV 2011 Conference in Barcelona -- the same conference where I'll be presenting my Exemplar-SVM paper.

The Power of Comparative Reasoning
Jay Yagnik, Dennis Strelow, David Ross, Ruei-Sung Lin. ICCV 2011. [PDF]

P.S. If you're a fellow vision blogger, then come find me in Barcelona@iccv2011 -- we'll go brag a beer.

Question: What makes an object recognition system great?

2011-08-16T16:16:00.000-05:00

Today, instead of discussing my own perspectives on object recognition or sharing some useful links, I would like to ask a general question geared towards anybody working in the field of computer vision:

What makes an object recognition system great?

In particular, I would like to hear a broad range of perspectives regarding what is necessary to provide an impact-creating open-source object recognition system for the research community to use. As a graduate student you might be interested in building your own recognition system, as a researcher you might be interested in extending or comparing against a current system, and as an educator you might want to to direct your students to a fully-functional object recognition system which could be used to bootstrap their research.

To start the discussion I would like to first enumerate a few elements which I find important in making an object recognition system great.

Open Source

In order for object recognition to progress, I think releasing binary executables is simply not enough. Allowing others to see your source code means that you gain more scientific credibility and you let others extend your system -- this means letting others both train and test variants of your system. More people using an object recognition system also translates to a high citation count, which is favorable for researchers seeking career advancement. Felzenszwalb et al. have released multiple open-source version of their Discriminatively Trained Deformable Part Model -- each time we see a new release it gets better! Such continual development means that we know the authors really care about this problem. I feel Github, with its distributed version control and social-coding features, is a powerful took the community should adopt, something which I believe is very much needed to take the community's ideas to the next level. In my own research (e.g., the Ensemble of Exemplar-SVMs approach), I have started using Github (for both private and public development) and I love it. Linux might have been started by a single individual, but it took a community to make it great. Just look at where Linux is now.

Ease of use

For ease of use, it is important that the system is implemented in a popular language which is known by a large fraction of the vision community. Matlab, Python, C++, and Java are such popular language and many good implementations are a combination of Matlab with some highly-optimized routines in C++. Good documentation is also important since one cannot expect only experts to be using such a system.

Strong research results

The YaRS approach, which is the "yet-another-recognition-system" approach, doesn't translate to high usage unless the system actually performs well on a well-accepted object recognition task. Every year at vision conferences, many new recognition frameworks are introduced, but really only a few of them ever pass the test of time. Usually an ideas withstands time because it is a conceptual contribution to science, but systems such as the HOG-based pedestrian detector of Dalal-Triggs and the Latent Deformable Part Model of Felzenszwalb et al. are actually being used by many other researchers. The ideas in these works are not only good, but the recognition systems are great.

Question:

So what would you like to see in the next generation of object recognition systems? I will try my best to reply to any comments posted below. Any really great comment might even trigger a significant discussion; enough to warrant its own blog post. Anybody is welcome to comment/argue/speculate below, either using their real name or anonymously.

CMU's Black Fridays: a graduating PhD student's perspective

2011-08-15T22:38:00.001-05:00

I've always been amazed that the CMU department really knows what its PhD students are up to -- the big stuff as well as the little stuff.

Allow me to elaborate. As a PhD student at CMU, you receive a sort of "report card" at the end of every semester during on what is known as "Black Friday." You first submit a short summary of your accomplishments and goals for next semester. Then, on this special day, the professors talk to each other about their students (probably in some secret sound-proof discussion room). While faculty are discussing our fates, we, the students do the opposite. We relax, watch movies, play games, and *imagine* what our superiors are discussing. Black Friday letters serve two roles, a role for the faculty, and a role for the students.

Image courtesy of diypapers

For the faculty, Black Friday is a way for the department to evaluate and monitor the progress of their PhD students. Faculty members get a chance to discuss their students' hardships as well as their successes. For the students, these letters are way to keep us in *check*.

"You better check yo self before you wreck yo self" - Ice Cube

The Black Friday letter lets us know explicitly what research qualifiers, writing qualifiers, etc they expect us to complete next semester. They let us know if they are happy with our progress or unhappy.

Photo by Mattox

In practice, I found the letters to be a combination of "the good" and "the bad." The good (a statement such as "we are happy that you got your paper accepted to ___") is like getting A's in grade school -- a yipee! moment. The bad (a statement such as "we have noticed that you are struggling taking your experimental research to the next level, and feel that you are spending too much time on ___"), is what pushes us to experience those yipee! moments the following semester. In the past, my Black Friday letters have included details regarding elements of my research and coursework that I didn't know any faculty member even knew about. But the faculty care! The students are the future, and there is nothing like critical feedback to help us achieve our goals. There have been several times during my 6 years at CMU's Robotics Institute when my letter helped keep me in check.

I only wish there was a way for students to provide Black Friday letters to their faculty mentors...

Further reading:
Black Friday from Peter Lee
How do you evaluate your grad students? by Matt Welsh

blazing fast nms.m (from exemplar-svm library)

2011-08-13T19:46:00.000-05:00

If you care about building large-scale object recognition systems, you have to care about speed. And every little bit of performance counts -- so why not first optimize the stuff which is slowing you down?

NMS (non-maximum suppression) is a very popular post-processing method for eliminating redundant object detection windows. I have take Felzenszwalb et al.'s nms.m and made it significantly faster by eliminating an inner loop. 6 years of grad school, 6 years of building large-scale vision systems in Matlab, and you really learn how to vectorize code. The code I call millions of times needs to be fast, and nms is one of those routines I call all the time.

The code is found below as a Github gist -- which was taken from my Exemplar-SVM object recognition library (from my ICCV2011 paper: Ensemble of Exemplar-SVMs for Object Detection and Beyond). The same file, nms.m, can also be found a part of the Exemplar-SVM library on Github. In fact, this code produces the same result as Pedro's code, but is much faster. Here is one timing experiment I performed when performing nms on ~300K windows, where my version is roughly 100 times faster. When you deal with exemplar-SVMs you have to deal with lots of detectors (i.e., lots of detection windows), so fast NMS is money.

>> tic;top = nms_original(bbs,.5);toc
Elapsed time is 58.172313 seconds.

>> tic;top = nms_fast(bbs,.5);toc
Elapsed time is 0.532638 seconds.

Ensemble of Exemplar-SVMs for Object Detection and Beyond

2011-08-12T23:51:00.000-05:00

Over the next couple of days I will be announcing some very exciting news. As many of you know, I defended my PhD this past Monday at CMU. My family and friends came for the presentation as I defended 6 years of my life in front of Alyosha Efros, Martial Hebert, Takeo Kanade, and Pietro Perona. You might be wondering what I've been up this past year -- what sort of new vision research have I produced since the Visual Memex paper.

Throughout the last year or so I have slowly abandoned the segment-then-recognize approach and fully embraced the exemplar-based component of my research. Because once you go exemplar, you don't go back! If only Nosofsky was here, he would be proud. Once you have established a good exemplar-detection alignment, problems such as segmentation become trivial. In fact, exemplar association enables a host of meta-data transfer applications. Here is a quick overview of my recent ICCV 2011 paper with Alexei Efros and Abhinav Gupta (the super new and exciting professor at CMU who will likely revolutionize they way we, vision researchers, think about the interplay of geometric reasoning and object recognition).

I will be defending my work to the ICCV crowd this fall in Barcelona. Here is the paper.

Paper:

Tomasz Malisiewicz, Abhinav Gupta, Alexei A. Efros. Ensemble of Exemplar-SVMs for Object Detection and Beyond . In ICCV, 2011. [PDF] [Project Page]

Abstract:

This paper proposes a conceptually simple but surprisingly powerful method which combines the effectiveness of a discriminative object detector with the explicit correspondence offered by a nearest-neighbor approach.

Exemplar Associations go Beyond Bounding Boxes

The method is based on training a separate linear SVM classifier for every exemplar in the training set. Each of these Exemplar-SVMs is thus defined by a single positive instance and millions of negatives.

An ensemble of exemplars

While each detector is quite specific to its exemplar, we empirically observe that an ensemble of such Exemplar-SVMs offers surprisingly good generalization. Our performance on the PASCAL VOC detection task is on par with the much more complex latent part-based model of Felzenszwalb et al., at only a modest computational cost increase.

Generalization from a single positive instance

But the central benefit of our approach is that it creates an explicit association between each detection and a single training exemplar. Because most detections show good alignment to their associated exemplar, it is possible to transfer any available exemplar meta-data (segmentation, geometric structure, 3D model, etc.) directly onto the detections, which can then be used as part of overall scene understanding.

This paper can be rightfully seen as a marriage of my older work on learning per-exemplar distances with the discriminative training method of Felzenszwalb et al.

Here are some summary pictures from my paper and a short description of each one:

1. Going beyond object detection (i.e., produce a category-labeled bounding box), we look at several meta-data transfer applications. Meta-data transfer is a way interpreting an object detection in a way which transcends category membership. The first task is that of geometry transfer.

Geometry Transfer

2. Segmentation is a well-known problem in computer vision -- generally tackled with bottom-up approaches which strive to produce coherent regions based on pixel-pixel appearance similarity. We show that a recognize-then-segment is possible, and in particular an associate-then-segment approach based on transferring segmentations from exemplars onto detection windows.

Segmentation Transfer

3. Object exemplar often show an interplay of objects, suggesting that it is possible to use the recognition of one object to prime the presence of another.

Related Object Priming

P.S. Dr. Abhinav Gupta is looking for students, so if you are a 1st year CMU visionary (CMU visionary = robotics vision student@CMU), check out his presentation during the RI Immigration Course.

P.S.S. Anonymous Reviewer#3: Not only have you single-handedly saved my paper from the clutches of ICCV death, but you have resurrected a graduate student's faith in the justice of the vision peer review process.

CMU Robotics Instititue's vision finds a home at Google

2011-07-24T14:54:00.000-05:00

Congratulations to PittPatt for their recent acquisition by Google. PittPatt, a Pittsburgh-based startup, has its roots in CMU's Robotics Institute (where I'm currently a PhD student). Henry Schneiderman, the CEO of PittPatt, did some truly hardcore computer vision work while doing his PhD under Takeo Kanade.

Two famous papers by Hendry Schneiderman and Takeo Kanade are the following:
H. Schneiderman, T. Kanade. "A Statistical Method for 3D Object Detection Applied to Faces and Cars". IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2000) pdf format

H. Schneiderman, T. Kanade. "Probabilistic Modeling of Local Appearance and Spatial Relationships for Object Recognition." IEEE Conference on Computer Vision and Pattern Recognition (CVPR 1998). pdf format

Here is what the front page of PittPatt states:

Joining Google is the next thrilling step in a journey that began with research at Carnegie Mellon University's Robotics Institute in the 1990s and continued with the launching of Pittsburgh Pattern Recognition (PittPatt) in 2004. We've worked hard to advance the research and technology in many important ways and have seen our technology come to life in some very interesting products. At Google, computer vision technology is already at the core of many existing products (such as Image Search, YouTube, Picasa, and Goggles), so it's a natural fit to join Google and bring the benefits of our research and technology to a wider audience. We will continue to tap the potential of computer vision in applications that range from simple photo organization to complex video and mobile applications.

We look forward to joining the team at Google!

The team at Pittsburgh Pattern Recognition

Perhaps Henry's success is yet another reason to come to CMU to get a vision PhD...

vampiric professors vs. industry-leading werewolves

2011-07-13T22:54:00.000-05:00

"Vampires need blood from human donors. Professors need publications, which they extract from their grad students." - Peter Tu

Photo courtesy of Werewolves.com

I wanted to share with everybody a short article by Peter Tu of GE Research (and a Computer Vision hacker) in which he compares Professors to Vampires and Industry Researchers to Werewolves. If you are in the mood for a witty and insightful read, check this out:

Computer Vision and the Supernatural by Peter Tu

David Weinberger: Reading Aristotle

2011-07-07T15:56:00.002-05:00

Here are some quotes from David Weinberger's Reading Aristotle -- a short article on KMWorld from 2004.

"After all, the music of Aristotle's thought comes from his assumption that the principles of knowledge are the same as the principles of the universe. The categories are not "mere" categories of thought for him. They are also the way the cosmos is arranged. If the order of knowledge and the order of the world are not the same, reasoned Aristotle, then knowledge isn't possible. Knowledge is only possible if the universe is ordered in knowable ways."

"a world apart from the categories of understanding would be by definition unknowable"

Weinberger's ideas are very relevant to my thesis on the Visual Memex and his writings have been a great influence on me. I've recently been on a bit of a Weinberger-reading/listening-binge after I found some of his lectures as Podcasts!

---

David Weinberger is the author of Everything Is Miscellaneous: The Power of the New Digital Disorder. The book's accompanying blog can be found here: http://www.everythingismiscellaneous.com/

cvpr 2011: highlights from day 1

2011-06-22T02:15:00.003-05:00

Today was the first main day of the CVPR 2011 conference. Here are some papers which I found particularly exciting and a brief reason why they are super cool:

Becuase you can’t afford to miss this one:

Abhinav Gupta, Scott Satkin, Alexei A. Efros and Martial Hebert. From Scene Geometry to Human Workspace. In CVPR 2011.

Because large scale structure from motion meets graphical models:

David Crandall, Andrew Owens, Noah Snavely, and Dan Huttenlocher. Discrete-Continuous Optimization for Large-Scale Structure from Motion. In CVPR 2011.

Because segmentation provides a good basis for object discovery:

Galleguillos C., McFee B., Belongie S., Lanckriet G. From Region Similarity to Category Discovery. In CVPR 2011.

Because imitation provides a category-free way of understanding pose:

Graham Taylor, Ian Spiro, Christoph Bregler, and Rob Fergus. Learning Invarance through Imitation. In CVPR 2011. Project page (with links to supplementary material)

Because you don’t need to solve intractable graphical model inferences:

S. Ross, D. Munoz, M. Hebert, J. A. Bagnell. Learning Message-Passing Inference Machines for Structured Prediction. In CVPR 2011.

Because a little bit of bottom-up never hurt a lot of top-down:

Thomas Brox, Lubomir Bourdev, Subhransu Maji, Jitendra Malik. Object Segmentation by Alignment of Poselet Activations to Image Contours. In CVPR 2011

Because you might want to perform segmentation simultaneously on several related images:

Sara Vicente, Carsten Rother and Vladimir Kolmogorov. Object Cosegmentation. In CVPR 2011.

I missed some of the posters later in the evening because I went for a short hike organized by Jianbo Shi (to Seven Falls). After all, CVPR is in Colorado Springs, and it is silly to stay inside the conference the entire time. Below is a pic of the falls (only a short drive from the CVPR11 conference hotel).

After several hours of talks it was good to get some fresh air and see some waterfalls in Colorado Springs! On Friday, I'm going on the Pikes Beak downhill bike ride with some friends and it should be super-fun.

cvpr 2011 in colorado springs

2011-06-14T00:39:00.036-05:00

CVPR (CVPR 2011) will be held this year in Colorado Springs, and I'll hopefully get a chance to blog about the newest and exciting ideas coming out of the Computer Vision research community while I'm there. CVPR is a conference which I try to attend every year because 1.) the quality of the work there is superb and 2.) it is a perfect opportunity to network with all the amazing vision researchers I've met over the years.

I will also be at the Fine Grained Visual Categorization workshop on June 25th, which has an exciting schedule of talks and posters. This is the first ever FGVC workshop, and it is being organized by Ryan Farrell (UMD), Steve Branson (UCSD), and Peter Welinder (Caltech).

vision talks at CMU are the best

2011-04-28T03:54:00.000-05:00

At CMU, we get some of the best people in object recognition to visit and give talks. The CMU VASC seminar is a place where all the cool vision researchers come and advertise their own research. Just last week Deva Ramanan gave a spectacular talk about his most recent part-based detector. Deva made an everlasting impression on me as a first year PhD student -- 5 years ago he showed the world that visualizations of person's parts as marginals are much sexier than just boxes.

Deva's Sexy Part Marginals

Today, Rob Fergus gave a talk on his most recent deep learning research. This research topic has been promulgated by hardcore machine learning titans such as Andrew Ng, Yann LeCun, and Geoff Hinton, so it will be exciting to see how Rob applies these ideas to object recognition. Unsupervised Feature Learning seems pretty exciting; unfortunately, I just cannot take results on Caltech101/Caltech256 very seriously :-(

Deep Learning Learning Architectures

Matlab Trick: Counting via Sparse

2011-03-25T13:13:00.000-05:00

Here is a simple little matlab demo script of how to count items using the command sparse. Counting the occurrences of items is a frequently performed task in vision and not many people know that sparse can do this. Remember, if you're going to be a matlab jedi do not write for loops:

I am using github's gists to embed this code in style. Github lets me choose the language, highlights the syntax accordingly, and the snippet is its own repository!

Computer Vision is Artificial Intelligence

2011-03-24T21:04:00.001-05:00

Computer vision is a diverse field and its researchers have multifaceted interests and aspirations. It should not be surprising that no two vision researchers think about the field in the same way. Different academic backgrounds foster alternative and potentially incommensurable interpretations. It is as if W.V.O Quine's thesis that no observation can be "theory-independent" directly applies to vision: a researcher in computer vision cannot uphold a view on his own field that is objective and independent of their own predispositions, upbringing, and educational program. While I cannot speak clearly about the long-term goals of the entire body researchers in vision, today I would like discuss my own take on computer vision. I do not offer the world an objective account of why computer vision intrigues me, but by sharing with the world the reasons why I find vision exciting, perhaps together we can break the boundaries of machine intelligence.

Cognitive Science is a computational study of the mind: McGill Cognitive Science

One of the biggest accomplishments in the field of Artificial Intelligence was when Deep Blue, a chess playing program developed at IBM, beat the world chess champion, Garry Kasparov. But this was in the early days of artificial intelligence -- when computer scientists still weren't sure on what it means for a machine to be intelligent. Chess is a well-known thinking-man's game, and at first glance it seems that a machine can only be worthy of being dubbed intelligent if it performs competitively on intelligent-people activities such as chess.

Chess: Human vs. Machine: Slate article about Deep Blue

Given the plethora of tasks that humans can effortlessly perform in daily life, is engineering a machine to rival humans on just one such task bringing researchers any closer to building truly intelligent machines?

The problem with chess is that it has a "finite universe problem" -- there is a finite number of primitives (the chess pieces) which can be manipulated by choosing a move from a finite set of allowable actions. But if we think of normal life (going to work, eating dinner, talking to a friend) as a game, then it is not hard to see that most everyday situations involving humans involve a sea of infinite objects (just look around and name all the different objects you can see around you!) and an equally capacious space of allowable actions (consider all the things you could with all those objects around you!). Intelligence is what allows us to cope with the complexities of the universe by focusing our attention on a limited set of relevant variables -- but the working set of objects/concepts we must consider at any single instant is chosen from a seemingly infinite set of alternatives.

I believe that everyday human-level visual intelligence is greatly undervalued by people -- and there is a very good reason for this! The ability to make sense of what is going on in a single picture is such a trivial and autonomous task for humans, that we don't even bother quantifying just how good we are at it. But let me reassure you that automated image understanding is no trivial feat. The world is not composed of 20 visual object categories and the space of allowable and interpretable utterances we could associate with a static picture is seemingly infinite. While the 20 category object detection task (as popularized by the PASCAL VOC) does have a finite universe problem, the grander version of the vision master problem (a combination of detection/recognition/categorization where you can interpret an input any way you like) is much more complex and mirrors the structure of the external world well.

Robotics Challenge: Build a Robot like Bender

Any application which calls for automated analysis of images requires vision. A robot, if it is to be successful interacting with the world and performing useful tasks, needs to perceive the external world and organize it. While some see vision as just one small piece of the "Robotics Challenge" (build a robot and make it do cool stuff), it totally unclear to me where to draw the boundary between low-level pixel analysis and high-level cognitive scene understanding. Over the years, I have been thinking more and more about this problem, and I've convinced myself that the interesting part of vision is precisely at the boundary between what is commonly thought of as low-level representation of signal and what is considered high-level representation of visual concepts. While some view computer vision as "applied mathematics" or "applied machine learning" or "image processing in disguise", I passionately believe the following:

Computer Vision is Artificial Intelligence

I am not promulgating the thesis that all aspects of machine intelligence are visual, but I want to assure you that there are enough high-level semantic capabilities which must be set in place for vision to work, that it is not worthwhile to think of vision as smaller problem than general purpose intelligence. I believe that once we have made progress on vision (not in the narrow-universe setting) to the point where generic visual scene understanding is effectively solved, there won't be much left that needs to go into the "ethereal" mind which cognitive scientists want to empower machines with! The only way to make machines truly understand scenes, objects, and their interactions is to make machines know something about the fabric of human life, and it is important for machines to learn this for themselves from real-world experience. This goes beyond representing object appearance because folk physics, folk psychology, causality, spatio-temporal continuity, etc are all faculties which vision systems will need (at least the vision systems I want to ultimately build) for general purpose scene understanding. I don't want to undermine the efforts of cognitive scientists (which work on many of the theories/ideas I've delineated before), but perhaps only to convince them that I have been a cognitive scientist all along. I don't think placing a label on myself, by calling myself as either a cognitive scientist, a computer vision researcher, or AI researcher is very conducive to good research.

Kristen Grauman's CVPR 2011 papers

2011-03-23T02:49:00.001-05:00

This year there are lots of new papers from Kristen Grauman's Computer Vision Group at UT-Austin, to be presented at CVPR 2011. There are lots of them, but not abstracts/PDFS yet. Here is a melange of pictures to entice us..

Learning the Easy Things First: Self-Paced Visual Category Discovery. Y. J. Lee and K. Grauman. To appear,Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, June 2011. [project page]

Boundary-Preserving Dense Local Regions. J. Kim and K. Grauman. To appear, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, June 2011. (Oral)

Large-Scale Live Active Learning: Training Object Detectors with Crawled Data and Crowds. S. Vijayanarasimhan and K. Grauman. To appear, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, June 2011. (Oral)

haring Features Between Objects and Their Attributes. S. J. Hwang, F. Sha, and K. Grauman. To appear, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, June 2011.

Clues from the Beaten Path: Location Estimation with Bursty Sequences of Tourist Photos. C.-Y. Chen and K. Grauman. To appear, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, June 2011. [project page]

Efficient Region Search for Object Detection. S. Vijayanarasimhan and K. Grauman. To appear, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, June 2011.

Interactively Building a Discriminative Vocabulary of Nameable Attributes. D. Parikh and K. Grauman. To appear, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, June 2011.

On another note, Antonio Torralba has a CVPR 2011 paper with Ruslan Salakhutdinov (Hinton's ex-PhD student) and Josh Tenenbaum, but I'll post info and pics after I get to read the paper.

Learning to Share Visual Appearance for Multiclass Object Detection
Ruslan Salakhutdinov, Antonio Torralba , and Josh Tenenbaum.

vision@github

2011-03-23T02:29:00.000-05:00

Version control is a must if you're serious about software development. Like it or not, you have to be serious about software development if you want to build a large computer vision system (and perhaps get a PhD while doing it). Over the years, I've moved from CVS to SVN to git. I refuse to code without version control.

While git lets me easily share my private research code with my colleagues at CMU (much easier than svn), github lets me 'publish' it online in style. This is not the kind of sharing that most researchers do -- most researchers I know just put a tarball online. I've started playing around with github, and I'm really excited about using it for my own Computer Vision project.

I think sharing research code in a high-quality environment is necessary if we want to push the boundaries of machine intelligence in a team-like fashion. I think good papers and good code make good research, and while lots of care goes into high quality publications in the field, very few high-quality object recognition programs are published online "well". I think having forums, version control, wikis, etc is important if one wants a project to stand the test-of-time. I think github is the way to go if you're going to share your code online.

kinect fun

2011-03-15T21:16:00.000-05:00

Ah, the guitar. Ah, a kinect.

Kinect plugged into my Macbook Air.

snowy in Boston, SUn-ny at MIT

2011-01-28T00:55:00.000-05:00

Today is SUnS 2011 @ MIT.

Artwork by Antonio Torralba

SUnS 2011 is a multi-disciplinary symposium with speakers and poster presenters from a variety of disciplines (neurophysiology, cognitive neuroscience, visual cognition and computer vision) who will address a range of topics related to scene understanding and spatial cognition, object recognition, attention, visual search, etc.

ORGANIZERS: Aude Oliva, Thomas Serre, Antonio Torralba

I will be there and will try to update the world about the most exciting stuff I learn about. I'm really excited about Moshe Bar's talk, his research has been a great influence of mine.

if you are starting your research in the field of object recognition / object detection...

2011-01-26T18:33:00.000-05:00

If you are an aspiring computer vision graduate student and hope to one day shatter the boundaries of machine perception, a good place to start is on the shoulders of giants. A key ingredient to successful object recognition research is a powerful codebase, which you will hopefully one day outgrow and/or extend. The single best place to get starter-code is at the following work, titled:

Discriminatively Trained Deformable Part Models

Why not start with some easy-to-understand MATLAB code so you can starting advancing your research this year, not this decade!?! Also, if you are able to build on this work, you will have an easy time publishing object detection papers that will actually be treated seriously by contemporary vision researchers. So my advice is to get voc-release-3.1, and read the following PAMI paper.

P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan
Object Detection with Discriminatively Trained Part Based Models
IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, September 2010
pdf Source code

Be warned that pff is probably smarter than you so you will not be able to understand 100% of everything he says, but because it is well-written code you will not have to understand all of it. If you want to be a Vision Jedi, look at the code, read the paper, discard the downloaded code, and write it yourself.

why I should be hacking with a kinect

2010-12-31T02:12:00.000-05:00

It was recently brought to my attention that Alex Berg a.k.a. Alexander Berg is hacking with a Kinect.
In case you didn't know, Alex Berg is an assistant professor at Stony Brook University as of Sept 2010. He came out of Jitendra Malik's group, and can be thought of as my academic uncle (because he got his PhD with Jitendra at basically the same time as my advisor, Alyosha Efros). I am a big fan of Alex Berg's work. (See the paper at ECCV 2010: What does classifying more than 10,000 image categories tell us? and note his upcoming workshop "Large Scale Learning for Vision" at CVPR 2011).

I had already known that Xiaofeng Ren has been hacking with RGB-D cameras such as the Kinect for some time now. Xiaofeng (pronunciation of first name) Ren is a research scientist at Intel Labs Seattle since 2008 and on the affiliate faculty at the CSE department at UW since 2010. He is another one my many academic uncles and has contributed greatly to the field of Computer Vision. For some of his recent work with Kinects, see his RGB-D project page. Xiaofeng Ren's work has also been very influential during my own research -- it is worthwhile to recall that he coined the term "superpixels", which is prevalent in contemporary Computer Vision literature.

So when I learned that these bad-ass ex-Berkeley hackers are hacking with Kinects, I figured it was the time to acquire one of my own. I bought a Kinect today and plan on playing with Alex Berg's kinect2matlab interface for Mac OS X soon!

So, why aren't you hacking with a kinect?

I, for one, welcome our new Visual Memex-based overlords

2010-11-22T18:53:00.000-05:00

Welcome to the era of visual intelligence -- the era of Visual Memex-based overlords (now in 3D!)

Image from Context-Based Search for 3D Models, by Matthew Fisher and Pat Hanrahan

The goal of today's post is simple: to empower you, the reader, with an exciting and fresh perspective on the problem of visual reasoning. This simple idea is one of the central tenets promulgated in my upcoming doctoral dissertation -- and but I'd like to give this potent meme a head start. Visual Memex-style reasoning is not the kind of reasoning that is described in classic graduate level textbooks on AI (e.g. first-order logic). In the case that you've mentally over-fit to a graduate-level CS curriculum, you might even portray my iconoclastic views as ramblings of a lunatic -- this is okay, I know at least Ludwig would be proud.

The Visual Memex is a mentality/perspective which, I believe, can overcome many limitations faced by modern computer vision systems. What the Visual Memex can do for visual intelligence is akin to what the World Wide Web has done for knowledge (see Weinberger's excellent book "Everything is Miscellaneous" for the full argument). It's akin to using Google for acquiring knowledge instead of going to the library -- maybe knowledge was never meant to be embedded in bookshelves. The idea is embarrassingly simple: replace visual object categories with object exemplars and relationships between those exemplars. Maybe the linguistic categories that we (as humans) cannot seem to live without are mere shadows cast on the wall of a dark cave. Psychologists have long abandoned rigid categories in their models of how humans think about concepts, but the notion of a class is so fundamental to contemporary Machine Learning that many haven't even bothered to question its tenuous foundations. While categories (also referred to as classes) definitely make learning algorithms easier to formalize, maybe its better to let the data speak for itself. Free the data!

One upcoming research paper inspired by this category-free mentality is: Context-Based Search for 3D Models, by Matthew Fisher and Pat Hanrahan, of Stanford University. This paper will be presented at SIGGRAPH Asia 2010. Maybe it is time to abandon those rigid categories and memexify your own research problem?

Further reading:

The Classic: Vannevar Bush's Memex
Plato's Allegory of the Cave
2D object recognition: Beyond Categories: The Visual Memex Model for Reasoning About Object Relationships
3D object modeling: Context-Based Search for 3D Models

CVPR, the A+'s of yesteryear, and robots need us

2010-11-13T13:27:00.000-05:00

It is November yet again, and I'm proud to announce my last CVPR submission as a graduate student! It is that time of the year again -- the post-CVPR downtime. It is time to mentally tuck away the fruits of our labor (NOTE: you might want to create a readme.txt which explains how to use the 20,000 lines of code you wrote in the 7 days preceding the deadline), consider the long-term impact of our work, and perhaps even reconsider our position in life.

I want to build intelligent machines, and I feel vision is the right place to start -- even roboticists such as Rodney Brooks started out in vision. However, I don't feel churning out 'cute' CVPR papers is going to do much. Perhaps if all one cares about in life is getting tenure at a top ranked university, then proof-of-concept papers might be the path of least resistance. But remember when you were a teen, and you wanted to build a rocket which lets you travel at relativistic speeds -- allowing you to go back in time? Or remember when you wanted to build those humanoid robots that would both entertain your kid sister and help out your mother with house chores?

So why did so many intelligent people I know abandon those grandeur dreams and settle for bread crumbs? Getting your paper submitted to a peer-reviewed conference, so that you can pad your CV with another publication, is incommensurable with the dreams you once had. The publication of today is the A+ of yesteryear, and it is just way too easy for us, intellectuals, to stay comfortable with those A's, without asking for more. But robots need us, CVPR papers won't assemble themselves into intelligent machines.

But the deadline is over, and now its time to relax. If my rant did not make sense to you, then I envy you. I have to move on to more positive things -- I need to finish reading Pinker's Blank Slate, read some more Wittgenstein (and fully assimilate his criticism of Augustine's theory of language-acquisition), waste two days playing with the Riemann Zeta function (because the Basel problem was only the beginning), play some guitar, etc.

Multifaceted Knowledge Representation: Ideas from Marvin Minsky

2010-08-25T14:21:00.000-05:00

"I think a key to AI is the need for several representations of the knowledge, such that when the system is stuck (using one representation) it can jump to use another. When David Marr at MIT moved into computer vision, he generated a lot of excitement, but he hit up against the problem of knowledge representation; he had no good representations for knowledge in his vision systems." -- Marvin Minsky

Check out the full interview with Marvin Minsky here -- a must read for anybody serious about building intelligent machines! This interview appears to be a part of a larger volume: Hal's Legacy.

I believe that in order to make the enterprise of computer vision of success, we must seriously broaden our outlook on the problem. Are we seriously expecting algorithms to delineate object boundaries from real images based on statistics of patch descriptors without any sort of model of the world?

I don't know about you, but I seriously want to build intelligent machines. I don't think there will ever be any sort of low-level SIFT-esque algorithm that "solves vision." It is a much grander picture of intelligence that I'm really after -- and successful computer vision will be a result(component?) of a higher-level intelligent machine. Machines need to know about a whole lot more than is found in a single image -- and the necessary conceptual tools might not be present in the computer vision community.

A recurring theme in my blog is my belief that we must become renaissance men -- a unison of *nix hackers, vision scientists, cognitive scientists, philosophers, athletes, machine learning scientists, skilled orators, and much more -- if we are to have any hope of chiseling away at the problem of computational intelligence. Minsky was a pioneer of computational intelligence, and his words revitalize my own research efforts in this direction.

Beyond pixel-wise labeling: Blocks World Revisited

2010-08-23T12:24:00.001-05:00

"Thoughts without content are empty, intuitions without concepts are blind." -- Immanuel Kant

The Holy Grail problem of computer vision research is general-purpose image understanding. Given as input a digital image (perhaps from Flickr or from Google Image search), we want to recognize the depicted objects (cars, dogs, sheep, Macbook Pros), their functional properties (which of the depicted objects are suitable for sitting), and recover the underlying geometry and spatial relations (which objects are lying on the desk).

The early days of vision were dominated via the "Image Understanding as Inverse Optics" mentality. In order to make the problem easier, as well as to cope with the meager computational resources of the 60s, early computer vision researchers tried to recover the 3D geometry of simple scenes consisting of arrangements of blocks. One of the earlier efforts in this direction, is the PhD thesis Machine Perception of Three-Dimensional Solids by Larry Roberts from MIT back in 1963.

But wait -- these block-worlds are unlike anything found in the real world! The drastic divide between the imagery that vision researchers were studying in the 60s and what humans observe during their daily experiences ultimately led to the disappearance of block-worlds in computer vision research.

Image Parsing Concept Image from Computer Blindness Blog

Over the past couple of decades, we have seen the success of Machine Learning, and it is of no surprise that we are currently living in the "Image Understanding as statistical inference" era. While a single 256x256 grayscale image might have been okay to use in the 1960s, today's computer vision researchers use powerful computer clusters and do serious heavy-lifting on millions of real-world megapixel images. The man-made blocks-world of the 1960s is a thing of the past, and the variety found on random images downloaded from Flickr is the complexity we must now cope with.

While the style of computer vision research has shifted since its early days in the 1960s/1970s, many old ideas (and perhaps prematurely considered outdated) are making a comeback!

Assigning basic-level object category labels to pixels is a very popular theme in vision. Unfortunately, to gain a deeper understanding of an image, robots will inevitably have to go beyond pixel-level class labels. (This is one of the central themes in my thesis -- coming out soon!) Given human-level understanding of a scene, it is trivial to represent it as a pixel-wise labeling map, but given a pixel-wise labeling map it is not trivial to convert it to human-level understanding.

What sort of questions can be answered about a scene when the output of an "image understanding" system is represented as a pixel-wise label map?

1. Is there a car in the image?
2. Is there a person at this location in the image?

What questions cannot be answered given a pixel-wise label map?

1. How many cars are in this image? (While there are some approaches that strive to deal with delineating object instance boundaries, most image parsing approaches fail to recognize boundaries between two instances of the same category)
2. Which surfaces can I sit on?
3. Where can I park my car?
4. How geometrically stable are the objects in the scene?

While I have more criticisms than tentative solutions, I believe that vision students shouldn't be parochially preoccupied with solely the most recent approach to image understanding. It is valuable to go back several decades in the literature and gain a broader perspective on image understanding. However, some progress is being made! A deeply insightful upcoming paper from ECCV 2010, is the following:

Abhinav Gupta, Alexei A. Efros and Martial Hebert, Blocks World Revisited: Image Understanding Using Qualitative Geometry and Mechanics, European Conference on Computer Vision, 2010. (PDF)

What Abhinav Gupta does very elegantly in this paper is connect the blocks-world research of the 1960s with the geometric-class estimation problem, as introduced by Derek Hoiem. While the final system is evaluation in a Hoiem-like pixel-wise labeling task, the actual scene representation is 3D. The blocks in this approach are more abstract than the Lego-like volumes in the 1960s -- Abhinav's blocks are actually cars, buildings, and trees. I included the infamous Immanuel Kant quote, because I feel it describes Abhinav's work very well. Abhinav introduces the block as a theoretical construct which glues together a scene's elements and provides a much more solid interpretation -- Abhinav's blocks add the content to geometric image understanding which is lacking in the purely pixe-wise approaches.

While integrating large-scale categorization into this type of geometric reasoning is still an open problem, Abhinav provides us visionaries with a glimpse of what image understanding should be. The integration of robotics with image understanding technology will surely drive pixel-based "dumb" image understanding approaches to extinction.

more papers to check out from cvpr

2010-06-17T17:30:00.000-05:00

Here are more CVPR 2010 papers which I either found interesting or plan on reading when I get back to PIT. Enjoy!

Connecting Modalities: Semi-supervised Segmentation and Annotation of Images Using Unaligned Text Corpora
Authors: Richard Socher (Stanford University) , Li Fei-Fei (Stanford University)

Cascade Object Detection with Deformable Part Models
Authors: Pedro Felzenszwalb (University of Chicago) , Ross Girshick (University ) , David McAllester (Toyota Technological Institute, Chicago)

Beyond Active Noun Tagging: Modeling Contextual Interactions for Multi-Class Active Learning
Authors: Behjat Siddiquie (UMIACS) , Abhinav Gupta (Carnegie Mellon University)

Tiered Scene Labeling with Dynamic Programming
Authors: Pedro Felzenszwalb (University of Chicago) , Olga Veksler (University of Western Ontario)

Layered Object Detection for Multi-Class Segmentation
Authors: Yi Yang (UCI) , Sam Hallman () , Deva Ramanan () , Charless Fowlkes (UC Irvine)

Efficiently Selecting Regions for Scene Understanding
Authors: M. Pawan Kumar (Stanford University) , Daphne Koller (Stanford)

Image Webs: Computing and Exploiting Connectivity in Image Collections
Authors: Kyle Heath (Stanford) , Natasha Gelfand (Nokia Research - Palo Alto, CA) , Maks Ovsjanikov (Stanford University) , Mridul Aanjaneya (Stanford University) , Leonidas Guibas (Stanford University)

constrained parametric min-cuts: exciting segmentation for the sake of recognition

2010-06-13T20:54:00.000-05:00

I would like to introduce two papers about Constrained Parametric Min-Cuts from C. Sminchisescu's group. These papers are very relevant to my research direction (which lies at the intersection of segmentation and recognition). Like my own work, these papers are about segmentation for recognition's sake. The segmentation algorithm proposed in the paper is a sort of "segment sliding approach", where many binary graph-cuts optimization problems are solved for different Grab-Cut style initializations. These segments are then scored using a learned scoring function -- think regression versus classification. They show that these top segments are actually quite meaningful and correspond to object boundaries really well. Finally a tractable number of top hypothesis (still overlapping at this stage), are piped into a recognition engine.

The idea that features derived from segments are better for recognition than features from the spatial support of a sliding rectangle resonates in all of my papers. Regarding these CVPR2010 papers, I like their ideas of learning a category-free "segmentation-function" and the sort of multiple-segmentation version of this algorithm is very appealing. If I remember correctly, the idea of learning a segmentation function comes to us from X. Ren, and the idea of using multiple segmentation comes from D. Hoiem. These papers are a cool new idea utilizing both insights.

J. Carreira and C. Sminchisescu. Constrained Parametric Min-Cuts for Automatic Object Segmentation. In CVPR 2010.

F. Li, J. Carreira, and C. Sminchisescu. Object Recognition as Ranking Holistic Figure-Ground Hypotheses. In CVPR 2010.

-------

Spotlights for these papers are during these tracks at CVPR2010:
Object Recognition III: Similar Shapes
Segmentation and Grouping II: Semantic Segmentation tracks

Sven Dickinson at POCV 2010, ACVHL Tomorrow

2010-06-13T15:25:00.000-05:00

This morning, Sven Dickinson gave a talk to start the POCV 2010 Workshop at CVPR2010. For those of you who might not know, POCV stands for Perceptual Organization in Computer Vision. While segmentation can be thought of as a perceptual grouping process, contiguous regions don't have to be the end product of a meaningful perceptual grouping process. There are many popular and useful algorithms which group non-accidental contours yet come short of a full-blown image segmentation.

The title of Dickinson's talk was "The Role of Intermediate Shape Priors in Perceptual Grouping and Image Abstraction." In the beginning of his talk, Sven pointed out how perceptual organization was at its prime in the mid 90s and declined in the 2000s due to the popularity of machine learning and the "detection" task. He believes that good perceptual grouping is what is going to make vision scale -- that is, without first squeezing out all that we can out of the bottom level we are doomed to fail.

Dickinsons showed some nice results from his most recent research efforts where objects are broken down into generic "parts" -- this reminded me of Biederman's geons, although Sven's fitting is done in the 2D image plane. Sven emphasized that successful shape primitives must be category-independent if we are to have scalable recognition of thousands of visual concepts in images. This is much different than the mainstream per-category object detection task which has been popularized by contests such as the PASCAL VOC.

While I personally believe that there is a good place for perceptual organization in vision, I wouldn't view it as the Holy Grail. It is perhaps the Holy Bridge we must inevitably cross on the way to finding the Holy Grail. I believe that for full-grown fully-functional members of society, our ability to effortlessly cope with the world is chiefly due to its simplicity and repeatability, and not due to some amazing internal perceptual organization algorithm. Perhaps it is when we were children -- viewing the world through a psychedelic fog of innocence -- that perceptual grouping helped us cut up the world into meaningful entities.

A common theme in Sven's talk was the idea of Learning to Group in a category-independent way. This means that all of the successes of Machine Learning aren't thrown out the door, and this appears to a quite different way of grouping than what has been done in the 1970s.

Tomorrow I will be at ACVHL Workshop "Advancing Computer Vision with Humans in the Loop". I haven't personally "turked" yet, but I feel I will be jumping on the bandwagon soon. Anyways, the keynote speakers should make for an awesome workshop. They do not need introductions: David Forsyth, Aude Oliva, Fei-Fei Li, Antonio Torralba, and Serge Belongie -- all influential visionaries.

everything is misc -- torralba cvpr paper to check out

2010-06-13T01:23:00.001-05:00

Weinberger's Everything is Miscellaneous is a delightful read -- I just finished it today while flying from PIT to SFO. It was recommended to me by my PhD advisor, Alyosha, and now I can see why! Many of the key motivations behind my current research on object representation deeply resonate in Weinberger's book.

Weinberger motivates Rosch's theory of categorization (the Prototype Model), and explains how it is a significant break from the thousand years of Aristotelian thought. Aristotle gave us the notion of a category -- centered around the notion of a definition. For Aristotle, every object can be stripped to its essential core, and place in its proper place in a God's-eye objective organization of the world. It was Rosch who showed us that categories are much fuzzier and more hectic than suggested by the rigid Aristotelian system. Just like Copernicus single-handedly stopped the Sun and set the Earth in motion, Rosch disintegrated our neatly organized world-view and demonstrated how an individual's path through life shapes h/er concepts.

I think it is fair to say that my own ideas as well as Weinberger's aren't so much an extension of the Roschian mode of thought, but also a significant break from the entire category-based way of thinking. Given that Rosch studied Wittgenstein as a student, I'm surprised her stance wasn't more extreme, more along the anti-category line of thought. I don't want to undermine her contribution to psychology and computer science in any way, and I want to be clear that she should only be lauded for her remarkable research. Perhaps Wittgenstein was as extreme and iconoclastic as I like my philosophers to be, but Rosch provided us with a computational theory and not just a philosophical lecture.

From my limited expertise in theories of categorization in the field of Psychology, whether it is Prototype Models or the more recent data-driven Exemplar Models, these theories are still theories of categories. Whether the similarity computations are between prototypes and stimuli, or between exemplars and stimuli, the output of a categorization model is still a category. Weinberger is all about modern data-driven notions of knowledge organization, in a way that breaks free from the imprisoning notion of a category. Knowledge is power, so why imprison it in rigid modules called categories? Below is a toy visualization of a web of concepts, as imagined by me. This is very much the web-based view of the world. Wikipedia is a bunch of pages and links.

Artistic rendition of a "web of concepts"

I found it valuable to think of the Visual Memex, the model I'm developing in my thesis research, as an anti-categorization model of knowledge -- a vast network of object-object relationships. The idea of using little concrete bits of information to create a rich non-parametric web is the recurring theme in Weinberger's book. In my case, the problem of extracting primitives from images, and all of the problem in dealing with real-world images are around to plague me, and the Visual Memex must rely on many Computer Vision techniques -- such things are not discussed in Weinberger's book. The "perception" or "segmentation" component of the Visual Memex is not trivial -- where linking words on the web is much easier.

CVPR paper to look out for

However, the category-based view is all around us. I expect most of this year's CVPR papers to fit in this category-based view of the world. One paper, co-authored by the great Torralba, looks relevant to my interests. It is yet another triumph for the category-based mentality in computer vision. In fact, one of the figures in the paper demonstrates the category-based view of the world very well. Unlike the memex, the organization is explicit in the following figure:

Exploiting Hierarchical Context on a Large Database of Object Categories

Exploiting Hierarchical Context on a Large Database of Object Categories

Myung Jin Choi, Joseph Lim, Antonio Torralba, and Alan S. Willsky. CVPR 2010.

blogging from CVPR2010

2010-06-11T23:17:00.003-05:00

It might not be one of those glamorous Apple events during which Steve Jobs introduces a new shiny gadget for the masses to desire, but plenty of exciting stuff happens at CVPR which instills desire into our souls, that is, the souls of computer vision scientists. Wouldn't you rather see the Great Torralba give a talk over some big company's chief executive officer? For those of you who do not know what CVPR is -- it is one of the big Computer Vision conferences during which we (the geeks, scientists, engineers, developers, hackers, and mathematicians) exchange ideas regarding our most recent research in the world of computer vision.

I am flying to SF tomorrow morning, and will be blogging about some of the cool papers I encounter at this year's CVPR. I do not have a paper at this year's conference so I'm in full assimilate-knowledge mode where I hope to absorb thousands of ideas related to my field. I already mentioned some of Kristen Grauman's cool segmentation papers, but expect to see in the next several blog posts many additional discussions for what I think are "exciting" papers. I am already getting excited and have plenty of papers to read during my flight, in addition to finishing Everything is Miscellaneous. I will be blogging from CVPR, like an Apple fanboy would at one of those Apple WWDC events -- but I will share math, theory, algorithms, and the like.

As always, the list of CVPR 2010 papers on the web can be found here.

my average face && second half marathon

2010-05-25T15:49:00.000-05:00

In case you didn't know, Picasa now performs face recognition in your photos. I found it amusing to see the progression of my own face over the past several years. Picasa lets you extract these face tiles into an 'export' directory, and it is trivial to load them up in Matlab for additional fun. I produced some TorralbaArt by averaging over 400 faces of myself (with no alignment whatsoever) collected over the past several years. These photos come from my personal photo collection, so I'm not making them publicly available. But here's the average face! I resized all images to 500x500 before averaging and resized the average to the mean aspect ratio of all images. The "black-eyes" come from the fact that I was wearing black sunglasses in about 10% of the photos.

On another note, I ran the Pittsburgh Half Marathon this year. This was my second half marathon ever -- my first one was last summer in San Francisco. This time, my finishing time was 1hour 40 minutes, which happens to be the goal I set for myself (10 minutes faster than my SF time). The first 20 minutes I was passing everybody in front of me since it was quite crowded. I could probably shave another 2-3 minutes off if I start towards the front of the herd, but I'll need some serious training if I'm going to reach 1:30 in a future race. I might even run a full marathon one of these days...

graph visualizations as sexy as fractals

2010-05-09T18:45:00.001-05:00

I love to display mathematical phenomena -- often for me the proof is in the visualization. If you ever steal one of my personal research notebooks you'll see that the number of graphs I've been drawing over the years has been increasing at a steady rate. This is a habit I acquired from studying Probabilistic Graphical Models and the machine learning-heavy curriculum at CMU.

Back in high school I was amazed by the beauty of fractals based on Newton's method for finding roots, but as I've slowly been shifting my mode of thought from continuous optimization problems to discrete ones, automated graph visualization is as close as I've ever gotten to being an artist. Here is one such sexy graph visualization from Yifan Hu's gallery.

Andrianov/lpl1 via sfdp by Yifan Hu

I have been using Graphviz for about 8 years now, and I just can't get enough. I never thought it would produce anything as beautiful as this! I generally used graphviz to produce graphs like this:

Inspired by Yifan Hu and his amazing multilevel force directed algorithm for visualizing graphs I've started using sfdp for some of my own visualizations. sfdp is now inside graphviz, and can be used with the -K switch as follows (also with overlap=scale):

$ dot -Ksfdp -Tpdf memex.gv > memex.pdf

Inspired by Yifan Hu's coloring scheme based on edge length, I color the edges using a standard matlab jet colormap with shorter edges being red and longer ones being blue. To get the resulting lengths of edges, I actually run sfdp twice -- once to read off the vertex positions (this is what the graph drawing optimization produces), and once again to assign the edge colors based on those lengths. I could process the resulting postscript with one run like Yifan, but I don't want to figure out how to parse postscript files today. Here is an example using some of my own data.

Car Concept Visual Memex via sfdp by Tomasz Malisiewicz

This is a visualization of the car subset of the Visual Memex I use as an internal organization of visual concepts to be used for image understanding. If you click on this image, it will show you a significantly larger png.

As a sanity check, I also created a visualization of a standard UF Sparse Matrix (here is both mine and Yifan's result)

UTM1700b via sfdp by Yifan Hu

UTM1700b via sfdp by Tomasz Malisiewicz

As you can see, the graphs are pretty similar, modulo some coloring strategy differences -- but since the colors are somewhat arbitrary this is not an issue. If you click on these pictures you can see the PDFs which were generated via graphviz. Now only if my real-world computer vision graph were as structured as these toy problems then others could view me as both an artist and a scientist (like a true Renaissance man).

Internet-scale vision at CVPR 2010

2010-04-14T23:57:00.001-05:00

I've been reading some of the recent CVPR 2010 papers (check out the CVPR papers on the web page to see the full list), and I came a cool video produced by Yasutaka Furukawa. I met Yasutaka when I was a visitor at Jean Ponce's WILLOW group in Paris during Spring 2008, and I was truly amazed by some of the cool geometry-based work he has done. Being a recognition/machine-learning guy myself, I can only appreciate and wonder at the amazing work produced by in-depth knowledge of geometry. In this particular case, the images aren't ones that Yasutaka collected himself. The idea behind internet-scale vision is that you can use the millions of photos on sites such as Flickr.

Here is a cool video below, very much in the spirit of Photosynth.

It is also not a surprise to find that Yasutaka is now working at Google. One can only imagine where Google is going to apply the "Street-View" mentality next. Cities like NYC already have nice high-resolution building facades, see picture below from Google Earth Blog.

I want to one day run all of my object recognition experiments on Google Street view, and there is probably only a handful of places in the world that have the computational infrastructure to play with such experiments. I drool at the idea of one day building a Visual Memex from billions of online images (and this can only happen at at place like Google).

Ontology is Overrated: Categories, Links, and Tags

2010-04-05T14:28:00.001-05:00

Ontology is Overrated: Categories, Links, and Tags

This is the title of a powerful treatise written by Clay Shirky, in which he strives to "convince you that a lot of what we think we know about categorization is wrong." Much thanks to David Weinberger's blog www.everythingismiscellaneous.com for pointing out this article. The take home message is quite similar to some of the "Beyond Categories" ideas I've tried to promulgate in my meager attempt to understand why progress in computer vision has reached a standstill. For anybody interested in understanding the limitations of classical systems of categorization, this article is a worth a read.

Exciting Computer Vision papers from Kristen Grauman's UT-Austin Group

2010-04-05T00:10:00.004-05:00

Back in 2005, I remember meeting Kristen Grauman at MIT's accepted PhD student open house. Back then she was a PhD student under Trevor Darrell (and is known for her work on the Pyramid Match Kernel), but now she has her own vision group at UT-Austin. She is the the advisor behind many cool vision projects there, and here are a few segmenatation/categorization related papers from the upcoming CVPR2010 conference. I look forward to checking out these papers because they are relevant to my own research interests. NOTE: some of the papers links are still not up -- I just used the links from Kristen's webpage.

Object-Graphs for Context-Aware Category Discovery. Y. J. Lee and K. Grauman

Collect-Cut: Segmentation with Top-Down Cues Discovered in Multi-Object Images. Y. J. Lee and K. Grauman

Asymmetric Region-to-Image Matching for Comparing Images with Generic Object Categories. J. Kim and K. Grauman

PhDs make many smart programmers become software engineering n00bs

2010-03-22T01:50:00.007-05:00

This is true. A couple of years in a PhD program -- reading papers and writing throw-away code in Matlab, and it easy to become a throw-away programmer, a sort of liability in the real world. It is no surprise many companies look down on hiring PhDs. I've seen kids enter the PhD program with real programming talent and exit real software engineering n00bs. In graduate school, you might code for 6 years without anybody grading your code. If you get sloppy, you will be worse off than when you started.

The problem is that many advisors don't care about their students writing good code. Writing good papers and giving good presentations -- you will be told that this is what makes good PhD students. Who cares about writing good code? -- we'll just have some 'engineering' people re-write it once you become famous. This is what students across the globe are being fed. This is no surprise, because your advisor won't get tenure by turning you into a mean mathematically-inclined super hacker. Then again, your advisor won't care if you go bald, are malnutritioned, and have no life outside research. There are many things that one has to take care of themselves, and software development skills aren't any different.

Note to the real world looking to hire talent: You should grill, I mean really grill fresh PhDs regarding the software development skills. Don't become mesmerized by their 4.0s, their long publication lists, and all their 'achievements.' If you want to hire a fresh PhD to write code, whether in a research or an engineering setting, then give them one hell-of-an-interview. I agree with Google's interview process. I studied for it, I am proud of my own software engineering skills, and I was proud to have been an intern at Google (twice). But I know of companies who were sorry they hired PhDs only to learn these recent graduates could only dabble on the board and would utterly fail at the terminal.

Note to PhDs looking to one day take our skill-set and impact the real world: Never stop learning and never stop writing good code. Never stop taking care of yourself. You were the brightest of the brightest before you started your PhD, and now you have 5-6 years to exit as a real superman. With all the mathematics and presentations skills you will acquire during a PhD ,on top of good software engineering skills, you will become invaluable to the real world. Its a real shame to become less valuable to the outside world after 6 years of a strenuous PhD program. But nobody will give you the recipe for success. Nobody will tell you to exercise, but if you want to pound your brain with mental challenges for decades to come, you will need physical exercise in your daily regiment. Your advisors won't tell you that keeping up to date on the tools of the trade, and being a real hacker, is very valuable in the real world. You will be told that fast results = many papers and its not worth writing good code.

After obtaining a PhD we should be role-models for the entire world. Seriously, why not? If a PhD is the highest degree that an institution can grant, then we should feel proud about getting one. But we are human, and one is only as strong as their weakest link. We should become super hackers, fear no quantum mechanics, fear no presentation in front of a crowd, and be all that one can be.

This is a part of a serious of posts aimed at finding flaws in the academic/PhD process and how it pertains to building strong/intelligent/confident individuals.

Back to basics: Vision Science Transcends Mathematics

2010-03-18T18:46:00.010-05:00

Vision (a.k.a. image understanding, image interpretation, perception, object recognition) is quite unlike some of the mathematical problems we were introduced to in our youth. In fact, thinking of vision as a "mathematical problem" in the traditional sense is questionable. An important characteristic of such "problems" is that by pointing them out we already have a notion of what it would be like to solve them. Does a child think of gravity as such a problem? Well, probably not, because without the necessary mathematical backbone there is no problem with gravity! It's just the way the world works! But once a child has been perverted by mathematics and introduced into the intellectual world of science, the world ceases to just be. The world become a massive equation.

Consider the seemingly elementary problem of finding the roots of a cubic polynomial. Many of us can recite the quadratic equation by heart, but not the one for cubics (try deriving the simpler quadratic formula by hand). If we were given one evening and a whole lot of blank paper, we could try tackling this problem (no Google allowed!). While the probability of failure is quite high (and arguably most of us would fail), it would still make sense of "coming closer to the solution". Maybe we could even solve the problem when some terms are missing, etc. The important thing here is that the notion of having reached a solution is well-defined. Also, once we've found the solution it would probably be easier to convince ourselves that it is correct (verification would be easier than actually coming up with the solution).

Vision is more like theoretical physics, psychology, and philosophy and less like the well-defined math problem I described above. When dealing with the math problem described above, we know what the symbols mean, we know valid operations -- the game is already set in place. In vision, just like physics, psychology and philosophy, the notion of a fundamental operational unit (which happens to be an object for vision) isn't rigidly defined as the Platonic Ideals used throughout mathematics. We know what a circle is, we know what a real-valued variable is, but what is a "car"? Consider your mental image of a car. Now remove a wheel and ask yourself, is this still a car? Surely! But what happens as we start removing more and more elements. At what point does this object cease to be a car and become a motor, a single tire, or a piece of metal? The circle, a Platonic Ideal, ceases to become a circle once it has suffered the most trivial of all perturbations -- any deviation from perfection, and boom! the circle ceases to be a circle.

Much of Computer Vision does not ask such metaphysical questions, as objects of the real world are seamlessly mapped to abstract symbols that our mathematically-inclined PhD students love to play with. I am sad to report that this naive mapping between objects of the real world and mathematical symbols isn't so much a questions of style, it is basically the foundation of modern computer vision research. So what must be done to expand this parochial field of Vision into a mature field? Wake up and stop coding! I think Vision needs a sort of a mental coup d'état, a fresh outlook on old problem. Sometimes to make progress we have start with a clean slate -- current visionaries do not possess the right tools for this challenging enterprise. Instead of throwing higher-level mathematics at the problem, maybe we are barking up the wrong tree? However, if mathematics is the only thing we are good at, then how are we to have a mature discussion which transcends mathematics? The window through which we peer circumscribes the world we see.

I believe if we are to make progress in this challenging endeavor, we must first become Renaissance men, a sort of Neitzschean Übermensch. We must understand what has been said about perception, space, time, and the structure of the universe. We must become better historians. We must study not only more mathematics, but more physics, more psychology, read more Aristotle and Kant, build better robots, engineer stabler software, become better sculptors and painters, become more articulate orators, establish better personal relationships, etc. Once we've mastered more domains of reality, and only then, will we have a better set of tools for coping with paradoxes inherent in artificial intelligence. Because a better grasp on reality -- inching closer to enlightenment -- will result in asking more meaningful questions.

I am optimistic. But the enterprise which I've outlined will require a new type of individual, one worthy of the name Renaissance Man. We aren't interested in toy problems here, nor cute solutions. If we want to make progress, we must shape our lives and outlooks around this very fact. Two steps backwards and three steps forward. Rinse, lather, repeat.

Representation and Use of Knowledge in Vision: Barrow and Tenenbaum's Conclusion

2010-03-05T10:48:00.006-05:00

To gain a better perspective on my research regarding the Visual Memex, I spent some time reading Object Categorization: Computer and Human Vision Perspectives which contains many lovely essays on Computer Vision. This book contains recently written essays by titans of Computer Vision and contains a great deal lessons learned from history. While such a 'looking back' on vision makes for a good read, it is also worthwhile to find old works 'looking forward' and anticipating the successes and failures of the upcoming generations.

In this 'looking forward' fashion, I want to share a passage regarding image understanding systems, from "Representation and Use of Knowledge in Vision," by H. G. Barrow and J. M. Tenenbaum, July 1975. This is a short paper worth reading for both graduate students and professors interested in pushing Computer Vision research to its limits. I enjoyed the succinct and motivational ending so much, it is worth repeating it verbatim:

--------

III Conclusion

We conclude by reiterating some of the major premises underlying this paper:

The more knowledge the better.
The more data, the better.
Vision is a gigantic optimization problem.
Segmentation is low-level interpretation using general knowledge.
Knowledge is incrementally acquired.
Research should pursue Truth, not Efficiency.

A further decade will determine our skill as visionaries.

-------------

Data-Driven Image Parsing With the Visual Memex: Thesis Proposal Complete!

2010-02-19T11:46:00.005-05:00

Yesterday, I successfully gave my thesis proposal talk at CMU and it was a great experience. The feedback I obtained from my committee members was invaluable, especially the comments from Takeo Kanade. It was a great honor for me to have Takeo Kanade, one of the titans of vision, on my committee. My external member, Pietro Perona, is also a key player object recognition, and provided some perceptive comments.

I gave my talk on my Macbook Pro using Keynote. I use the dvi output to connect to the projector and on my screen I was able to see the current slide as well as the upcoming slide. Using Skype I was able to connect to Pietro in California and share the presentation screen (not my two-slide screen!) with him. This was, he was able to follow along and see the same slides as everybody in the room. Skype was a great success!

I would like to thank everybody who came to my talk!

Beyond Categories =? Doing without Concepts

2010-01-26T02:28:00.004-05:00

The term "beyond category," from my limited knowledge, was originally coined to describe the music of Duke Ellington. It is a term of praise that acknowledges that one's style is inimitable and transcends barriers.

"Beyond Categories" was the first part of my NIPS 2009 paper's title. To "go beyond" means to transcend, to abandon or do without some limitation and strive higher -- there is nothing magical about my use of the term. I used the term category to refer to object categories, as are commonly used in computer vision, artificial intelligence, machine learning, as well as psychology, philosophy, and other branches of cognitive science. One of my research goals is to go beyond the use of categories as the basis for machine perception and visual reasoning. It has been argued by Machery that the term category is roughly equivalent to the term concept as used in psychology literature. In some sense the title of Machery's recent book, "Doing without concepts," is analogous to the phrase "Beyond categories" but to reassure myself I'll have to finish reading Machery's book.

So far the first chapter has been a delightful exposition into the world of concepts, a term dear to researchers in machine perception (AI) as well as human categorization (psychology). I look forward to reading the rest of the book, which I accidentally found while looking for Estes' book on categorization. I had already digested/assimilated some of Machery's work, in particular his paper titled Concepts are not a natural kind, so seeing his name on a book at the CMU library piqued my interest. In this 2005 paper, Machery argues that the debate between prototypes vs. exemplars vs. theories in the literature on concepts is not well-founded and there is no reason to believe a single theory should prevail. I'll attempt to summarize some of his take-home messages and their relevance to computer vision once I finish this book.

Heterarchies and Control Structure in Image Interpretation

2010-01-20T14:23:00.014-05:00

Several days ago I was reading one of Takeo Kanade's classic computer vision papers from 1977 titled "Model Representation and Control Structure in Image Understanding" and I came across a new term, heterarchy. I think motivating this concept is as important as its definition. At the representational level, Kanade does a good job at advocating the use of multiple levels of representation -- from pixels to patches to regions to subimages to objects. In addition to discussing the representational aspects of image understanding systems, Kanade analyzes different strategies for using knowledge in such systems (he uses the term control structure to signify the overall flow of information between subroutines). On one extreme is pass-oriented processing (this is Kanade's term -- I prefer to use the terms feed-forward or bottom-up) which relies on iteratively building higher levels of interpretation from lower ones. Marr's vision pipeline is mostly bottom-up, but that discussion will be left for another post. Another extreme is top-down processing, where the image is analyzed in a global-to-local fashion. Of course, as of 2010 these ideas are being used on a regular basis in vision. One example is the paper Learning to Combine Bottom-Up and Top-Down Segmentation by Levin and Weiss.

Kanade acknowledges that the flow of a vision algorithm is very much dependent on the representation used. For image understanding, bottom-up as well as top-down processing will both be critical components of the entire system. However the exact strategy for combining these processes, in addition to countless other mid-level stages, is not very clear. Directly quoting Kanade, "The ultimate style would be a heterarchy, in which a number of modules work together like a community of experts with no strict central executive control." According to this line of thought, processing would occur in a loopy and cooperative style. Kanade attributes the concept of a heterarchy to Patrick Winston who worked with robots in the golden days of AI at MIT. Like Kanade, Winston criticizes a linear flow of information in scene interpretation (this criticism dates back to 1971). The basic problem outlined by both Kanade and Winston is that modules such as line-finders and region-finders (think segmentation) are simply not good enough to be used in subsequent stages of understanding. In my own research I have used the concept of multiple image segmentations to bypass some of the issued with relying on the output of low/mid -level processing for high-level processing. In 1971 Winston envisioned an algorithmic framework that is a melange of subroutines -- a web of algorithms created by different research groups -- that would interact and cooperate to understand an image. This is analogous to the development of an operating system like Linux. There is no overall theory developed by a single research group that made Linux a success -- it is the body of hackers and engineers that produced a wide range of software products that make using Linux a success.

Unfortunately given the tradition of computer vision research, I believe that an open-source-style group effort in this direction will not come out of university-style research (which is overly coupled with the publishing cycle). It would be a noble effort, but would more of a feat of engineering and not science. Imagine a group of 2-3 people creating an operating system from scratch -- it seems like a crazy idea in 2010. However, computer vision research is often done in such small teams (actually there is often a single hacker behind a vision project). But maybe going open-source and allowing several decades of interaction will actually produce usable image understanding systems. I would like to one day lead such an effort -- being both the theoretical mastermind as well as the hacker behind this vision. I am an INTJ, hear me roar.

Understanding versus Interpretation -- a philosophical distinction

2010-01-18T14:28:00.007-05:00

Today I want to bring up an interesting discussion regarding the connotation of the word "understanding" versus "interpretation," particularly in the context of "scene understanding" versus "scene interpretation." While many vision researchers use these terms interchangeably, I think it is worthwhile to make the distinction, albeit a philosophical one.

On Understanding
While everybody knows that the goal of computer vision is to recognize all of the objects in an image, there is plenty of disagreement about how to represent objects and recognize them in the image. There is a physicalist account (from Wikipedia: Physicalism is a philosophical position holding that everything which exists is no more extensive than its physical properties), where the goal of vision is to reconstruct veridical properties of the world. This view is consistent with the realist stance in philosophy (think back to Philosophy 101) -- there exists a single observer-independent 'ground-truth' regarding the identities of all of the objects contained in the world. The notion of vision as measurement is very strong under this physicalist account. The stuff of the world is out there just waiting to be grasped! I think the term "understanding" fits very well into this truth-driven account of computer vision.

On interpretation
The second view, a postmodern and anti-realist one, is of vision as a way of interpreting scenes. The shift is from veridical recovery of the properties of the world from an image (measurement) to the observer-dependent interpretation of the input stimulus. Under this account, there is no need to believe in a god's eye 'objective' view of the world. Image interpretation is the registration of an input image with a vast network of past experience, both visual and abstract. The same person can vary their own interpretation of an input as time passes and the internal knowledge based has evolved. Under this view, two distinct robots could provide very useful yet distinct 'image interpretations' of the same input image. The main idea is that different robots could have different interpretation-spaces, that is they could obtain incommensurable (yet very useful!) interpretations of the same image.

It has been argued by Donald Hoffman (Interface Theory of Perception) that there is no reason why we should expect evolution to have driven humans towards veridical perception. In fact, Hoffman argues that natures drives veridical perception towards extinction and it only makes sense to speak of perception as guiding agents towards pragmatic interpretations of their environment.

In philosophy of science, there is the debate of whether the field of physics is unraveling some ultimate truth about the world versus physics painting a coherent and pragmatic picture of the world. I've always viewed science as an art and I embrace my anti-realist stance -- which has been shaped by Thomas Kuhn, William James, and many others. While my scientific interests have currently congealed in computer vision, it is no surprise that I'm finding conceptual agreement between my philosophy of science and my concrete research efforts in object recognition.

wordle word summary image

2010-01-14T00:53:00.004-05:00

This is the wordle summary of the keywords from this blog. Its a cute visualization of topics.

Image Interpretation Objectives

2010-01-12T14:27:00.004-05:00

An example of a typical complex outdoor natural scene that a general knowledge-based image interpretation system might be expected to understand is shown in Figure 1. An objective of such systems is to identify semantically meaningful visual entities in a digitized and segmented image of some scene. That is, to correctly assign semantically meaningful labels (e.g., house, tree, grass, and so on) to regions in an image -- see [29,30]. A computer-based image interpretation system can be viewed as having two major components, a "low-level" component and a "high-level" component [19],[31]. In many respects, the low-level portion of the system is designed to mimic the early stages of visual image processing in human-like systems. In these early stages, it is believed that scenes are partitioned, to some extent, into regions that are homogeneous with respect to some set of perceivable features (i.e., feature vector) in the scene [6],[40],[39]. To this extent, most low-level general purpose computer vision systems are designed to perform the same task. An example of a partitioning (i.e., segmentation) of Figure 1 into homogeneous regions is shown in Figure 2. The knowledge-based computer vision system we shall describe in this paper is not currently concerned with resegmenting portions of an image. Rather, its task is to correctly label as many regions as possible in a given segmentation.

This a direct quote from a 1984 paper on computer vision. A great example of segmentation-driven scene understanding. The content is similar enough to my own line of work that it could have been an excerpt from my own thesis.

It is actually in a section called Image Interpretation Objectives from "Evidential Knowledge-Based Computer Vision" by Leonard P. Wesley, 1984. I found this while reading lots of good tech reports from SRI International's AI Center in Menlo Park. Some good stuff there by Tenenbaum, Barrow, Duda, Hart, Nillson, Fischler, Pereira, Pentland, Fua, Szeliski, to name a few. Lots of stuff there is relevant to scene understanding and grounds the problem in robotics (since there was no "internet" vision back in the 70s and 80s).

On another note, I still haven't been able to find a copy of the classic paper, Experiments in Interpretation-Guided Segmentation by Tenenbaum and Barrow from 1978. If anybody knows where to find a pdf copy send me an email. UPDATE: Thanks to the quick reply! I have the paper now.

Computer Vision Papers at NIPS 2009

2009-12-10T14:22:00.003-05:00

Here I some computer vision papers I found interesting at NIPS 2009 in Vancouver.

[pdf][bib] Unsupervised Detection of Regions of Interest Using Iterative Link Analysis (NIPS 2009)
Gunhee Kim, Antonio Torralba

[pdf][bib] Region-based Segmentation and Object Detection (NIPS 2009)
Stephen Gould, Tianshi Gao, Daphne Koller

[pdf][bib] Segmenting Scenes by Matching Image Composites (NIPS 2009)
Bryan Russell, Alyosha Efros, Josef Sivic, Bill Freeman, Andrew Zisserman

[pdf][bib] Beyond Categories: The Visual Memex Model for Reasoning About Object Relationships (NIPS 2009)
Tomasz Malisiewicz, Alyosha Efros

On Science: passion towards solving a problem must come from within

2009-12-06T15:45:00.004-05:00

I'm currently in Chicago while traveling to Vancouver (NIPS 2009 conference) where I'll be defending my research during Tuesday's poster session. Instead of delving into the computational challenges that motivate my research, I want to take a step back and criticize what (sometimes? often?) happens during the publishing cycle.

According to me, good research starts with the passion to solve a particular problem or address a specific concern. Quite often, good research will raise more questions than it successfully solves. Unfortunately, when we submit papers to conferences we are judged on the clarity of presentation, level of experimental validation, as well as overall completeness. This means that the publishing cycle quite often promotes writing "cute" papers that have little long term impact in the field and can only be viewed as thorough and complete due to their narrow scope. This is why we should not solely rely on peer review nor cater our scientific lives towards pleasing others. Sometimes being a good scientist means breaking free from the norms that the world around us rigidly follows, sometimes publishing too often skews our research focus, and sometimes falling off the face of the earth for a period of time is necessary to push science in a new direction.

I want to challenge every scientist to follow their dreams and attempt to solve the problems they truly care about and not just attempt to please peer review. Maybe some think that the perversion of science (that is evaluating scientists by the number of publications they have) is okay, but in my book a scientific career which produces a single grand idea is superior to a career saturated with myriad "cute and thorough" papers. I'm not particularly upset with the progress of Computer Vision, but I think more people should ponder the negative consequences of pulling the publish-trigger too often.

Understanding the role of categories in object recognition

2009-11-24T11:17:00.007-05:00

If we set aside our academic endeavors of publishing computer vision and machine learning papers and sincerely ask ourselves, "What is the purpose of recognition?" a very different story emerges.

Let me first outline the contemporary stance on recognition (that is object recognition as is embraced by the computer vision community), which is actually a bit of a "non-stance" because many people working on recognition haven't bothered to understand the motivations, implications, and philosophical foundations of their work. The standard view of recognition is that it is equivalent to categorization -- assigning an object its "correct" category is the goal of recognition. Object recognition, as is found in vision papers, is commonly presented as single image recognition task which is not tied to an active and mobile agent that must understand and act in an environment around them. These contrived tasks are partially to blame for making us think that categories are the ultimate truth. Of course, once we've pinpointed the correct category we can look up information about the object category at hand in some sort of grand encyclopedia. For example, once we've categorized an object as a bird we can simply recall the fact that "it flies" from such a source of knowledge.

Most object recognition research is concerned with object representations (what features to compute from an image) as well as supervised (and semi-supervised) machine learning techniques to learn object models from data in order to discriminate and thus "recognize" object categories. The reason why object recognition has become so popular in the recent decade is that many researchers in AI/Robotics envision a successful vision system as a key component in any real-world robotic platform. If you ask a human to describe their environment, we will probably use a bunch of nouns to enumerate the stuff around them, so surely nouns must be the basic building blocks of reality! In this post I want to question this commonsense assumption that categories are the building blocks of reality and propose a different way of coping with reality, one that doesn't try to directly estimate a category from visual data.

I argue that just because nouns (and the categories they refer to) are the basis of effability for humans, it doesn't mean that nouns and categories are the quarks and gluons of recognition. Language is a relatively recent phenomenon for humans (think evolutionary scale here), and it is absent in many animals inhabiting the earth beside us. It is absurd to think that animals do not possess a faculty for recognition just because they do not have a language. Since animals can quite effectively cope with the world around them, there must be hope for understanding recognition in a way that doesn't invoke linguistic concepts.

Let me make my first disclaimer. I am not against categories altogether -- they have their place. The goal of language is human-human communication and intelligent robotic agents will inevitably have to map their internal modes of representation onto human language if we are to understand and deal with such artificial beings. I just want to criticize the idea that categories are found deep within our (human) neural architecture and serve as the basis for recognition.

Imagine a caveman and his daily life which requires quite a bit of "recognition"-abilities to cope with the world around him. He must differentiate pernicious animals from edible ones, distinguish contentious cavefolk from his peaceful companions, and reason about the plethora of plants around him. For each object that he recognizes, he must be able to determine whether it is edible, dangerous, poisonous, tasty, heavy, warm, etc. In short, recognition amounts to predicting a set of attributes associated with an object. Recognition is the linking of perceptible attributes (it is green and the size of my fist) to our past experiences and predicting attributes that are not conveyed by mere appearance. If we see a tiger, it is solely on our past experiences that we can call it dangerous.

So imagine a vector space, where each dimension encodes an attribute such as edible, throwable, tasty, poisonous, kind, etc. Each object can be represented as a point in this attribute space. It is language that gives us categories as a shorthand to talk about commonly found objects. Different cultures would give rise to different ways of cutting up the world, and this is consistent with what has been observed by psychologists. Viewing categories as a way of compressing attribute vectors not only makes sense but is in agreement with the idea that categories culturally arose much later than the ability for humans to recognize objects. Thus it makes sense to think of category-free recognition. Since a robotic agent who was programmed to think of the world in terms of categories will have to unroll categories to understand objects in terms of tangible properties if they are to make sense of the world around them, why not use the properties/attributes as the primary elements of recognition in the first place!?

from Describing objects by their attributes

These ideas are not entirely new. In Computer Vision, there is a CVPR 2009 paper Describing objects by their attributes by Farhadi, Endres, Hoiem, and Forsyth (from UIUC) which strives to understand objects directly using the ideas discussed above. In the domain of thought recognition, the paper Zero-Shot Learning with Semantic Output Codes by Palatucci, Pomerleau, Hinton, and Mitchell strives to understand concepts in a similar semantic basis.

I believe the field of computer vision has been conceptually stuck and the vehement reliance on rigid object categories is partially to blame. We should read more Wittgenstein and focus more on understanding vision as a mere component of artificial intelligence. If we play the recognize objects in a static image game (as Computer Vision is doing!) then we obtain a fragmented view of reality and cannot fully understand the relationship between recognition and intelligence.

Learning and Inference in Vision: from Features to Scene Understanding

2009-11-12T18:11:00.003-05:00

Tomorrow, Jonathan Huang and I are giving a Computer Vision tutorial at the First MLD (Machine Learning Department) Research Symposium at CMU. The title of our presentation is Learning and Inference in Vision: from Features to Scene Understanding.

The goal of the tutorial is to expose Machine Learning students to state-of-the-art object recognition, scene understanding and the inference problems associated with such high-level recognition problems. Our target audience is graduate students with little or no prior exposure to object recognition who would like to learn more about the use of probabilistic graphical models in Computer Vision. We outline the difficulties present in object recognition/detection and outline several different models for jointly reasoning about multiple object hypotheses.

A model of thought: The Associative Indexing of the Memex

2009-11-07T16:23:00.007-05:00

The Memex "Memory Extender" is an organizational device, a conceptual device, and a framework for dealing with conceptual relationships in an associative way. Abandoning the Aristotelian tradition of rooting concepts in definitions, the Memex suggests an association-based, non-parametric, and data-driven representation of concepts.

Since the mind=software analogy is so deeply engraved in my thoughts, it is hard for me to see intelligent reasoning as anything but a computer program (albeit one which we might never discover/develop). It is worthwhile to see sketches of the memex from an era before computers. (See Figure below). However, with the modern Internet, a magnificent example of a Bush's ideology, with links denoting the associations between pages, we need no better analogy. Bush's critique of the artificiality of traditional schemes of indexing resonates in the world wide web.

A Mechanical Memex Sketch

By extrapolating Bush's anti-indexing argument to visual object recognition, I realize that the blunder is to assign concepts to rigid categories. The desire to break free from categorization was the chief motivation for my Visual Memex paper. If Bush's ideas were so successful in predicting the modern Internet, we should ask ourselves, "Why are categories so prevalent in computational models of perception?" Maybe it is machine learning, with its own tradition of classes in supervised learning approaches, that has scarred the way we computer scientists see reality.

“The human mind does not work that way. It operates by association. With one item in its grasp, it snaps instantly to the next that is suggested by the association of thoughts, in accordance with some intricate web of trails carried by the cells of the brain. It has other characteristics, of course; trails that are not frequently followed are prone to fade, items are not fully permanent, memory is transitory. Yet the speed of action, the intricacy of trails, the detail of mental pictures, is awe-inspiring beyond all else in nature.” -- Vannevar Bush

Is Google's grasp of the world of information anything more than a Memex? I'm not convinced that it is not. While the feat of searching billions of web pages in real time has already been demonstrated by Google (and reinforced every day), the best computer vision approaches as of today resemble nothing like Google's data-driven way of representing concepts. I'm quite interested in pushing this link-based data-driven mentality to the next level in the field of Computer Vision. Breaking free from the categorization assumptions that plague computational perception might the the key ingredient in the recipe for success.

Instead of summarizing, here is another link to a well-written article on the Memex by Finn Brunton. Quoting Brunton, "The deepest implications of the Memex would begin to become apparent here: not the speed of retrieval, or even the association as such, but the fact that the association is arbitrary and can be shared, which begins to suggest that, at some level, the data itself is also arbitrary within the context of the Memex; that it may not be “the shape of thought,” emphasis on the the, but that it is the shape of a new thought, a mediated and mechanized thought, one that is described by queries and above all by links."

The Visual Memex: Visual Object Recognition Without Categories

2009-11-05T16:43:00.009-05:00

Figure 1

I have discussed the limitations of using rigid object categories in computer vision, and my CVPR 2008 work on Recognition as Association was a move towards developing a category-free model of objects. I was primarily concerned with local object recognition where the recognition problem was driven by the appearance/shape/texture features derived from within a segment (a region extraction from an image using an image segmentation algorithm). Recognition of objects was done locally and independently per region, since I did not have good model of category-free context at that time. I've given the problem of contextual object reasoning much thought over the past several years, and equipped with the power of graphical models and learning algorithms I now present a model for category-free object relationship reasoning.

Now its 2009, and its no surprise that I have a paper on context. Context is the new beast and all the cool kids are using it for scene understanding; however, categories are used so often for this problem that their use is rarely questioned. In my NIPS 2009 paper, I present a category-free model of object relationships and address the problem of context-only recognition where the goal is to recognize an object solely based on contextual cues. Figure 1 shows an example of such a prediction task. Given K objects and their spatial configuration, is it possible to predict the appearance of a hidden object at some spatial location?

Figure 2

I present a model called the Visual Memex (visualized in Figure 2), which is a non-parametric graph-based model of visual concepts and their interactions. Unlike traditional approaches to object-object modeling which learn potentials between every pair of categories (the number of such pairs scales quadratically with the number of categories), I make no category assumptions for context.

The official paper is out, and can be found on my project page:

Tomasz Malisiewicz, Alexei A. Efros. Beyond Categories: The Visual Memex Model for Reasoning About Object Relationships. In NIPS, December 2009. PDF

Abstract: The use of context is critical for scene understanding in computer vision, where the recognition of an object is driven by both local appearance and the object's relationship to other elements of the scene (context). Most current approaches rely on modeling the relationships between object categories as a source of context. In this paper we seek to move beyond categories to provide a richer appearance-based model of context. We present an exemplar-based model of objects and their relationships, the Visual Memex, that encodes both local appearance and 2D spatial context between object instances. We evaluate our model on Torralba's proposed Context Challenge against a baseline category-based system. Our experiments suggest that moving beyond categories for context modeling appears to be quite beneficial, and may be the critical missing ingredient in scene understanding systems.

I gave at talk about my work yesterday at CMU's Misc-read and received some good feedback. I'll be at NIPS this December representing this body of research.

Wittgenstein's Critique of Abstract Concepts

2009-10-26T14:59:00.006-05:00

In his Philosophical Investigations, Wittgenstein argues against abstraction -- via several thought experiments he strives to annihilate the view that during their lives humans develop neat and consistent concepts in their minds (akin to building a dictionary). He criticizes the commonplace notions of meaning and concept formation (as were commonly used in philosophical circles at the time) and has contributed greatly to my own ideas regarding categorization in computer vision.

Wittgenstein asks the reader to come up with the definition of the concept "game." While we can look up the definition of "game" in a dictionary, we can't help but feel that any definition will be either too narrow or too broad. The number of exceptions we would need in a single definition scales as the number of unique games we've been exposed to. His point wasn't that game cannot be defined -- it was that the lack of a formal definition does not prevent us from using the word "game" correctly. Think of a child growing up and being exposed to multi-player games, single-player games, fun games, competitive games, games that are primarily characterized by their display of athleticism (aka sports or Olympic Games). Let's not forget activities such as courting and the Stock Market which are also referred to as "games." Wittgenstein criticizes the idea that during our lives we somehow determine what is common between all of those examples of games and form an abstract concept of game which determines how we categorize novel activities. For Wittgenstein, our concept of game is not much more than our exposure to activities labeled as games and our ability to re-apply the word game in future context.

Wittgenstein's ideas are an antithesis to Platonic Realism and Aristotle's Classical notion of Categories, where concepts/categories are pure, well-defined, and possess neatly defined boundaries. For Wittgenstein, experience is the anchor which allows us to measure the similarity between a novel activity and past activities referred to as games. Maybe the ineffability of experience isn't because internal concepts are inaccessible to introspection, maybe there is simply no internal library of concepts in the first place.

An experience-based view of concepts (or as my advisor would say, a data-driven theory of concepts) suggests that there is no surrogate for living a life rich with experience. While this has implications for how one should live their own life, it also has implications in the field of artificial intelligence. The modern enterprise of "internet vision" where images are labeled with categories and fed into a classifier has to be questioned. While I have criticized categories, there are also problems with a purely data-driven large-database-based approach. It seems that a good place to start is by pruning away redundant bits of information; however, judging what is redundant and how is still an open question.

Scene Prototype Models for Indoor Image Recognition

2009-10-19T16:33:00.007-05:00

In today's post I want to briefly discuss a computer vision paper which has caught my attention.

In the paper Recognizing Indoor Scenes, Quattoni and Torralba build a scene recognition system for categorizing indoor images. Instead of performing learning directly in descriptor space (such as the GIST over the entire image), the authors use a "distance-space" representation. An image is described by a vector of distances to a large number of scene prototypes. A scene prototype consists of a root feature (the global GIST) as well as features belonging to a small number of regions associated with the prototype. One example of such a prototype might be an office scene with a monitor region in the center of the image and a keyboard region below it -- however the ROIs (which can be thought of as parts of the scene) are often more abstract and do not neatly correspond to a single object.

The learning problem (which is solved once per category) is then to find the internal parameters of each prototype as well as the per-class prototype distance weights which are used for classification. From a distance function learning point of view, it is rather interesting to see distances to many exemplars being used as opposed to the distance to a single focal exemplar.

Although the authors report results on the image categorization task it is worthwhile to ask if scene prototypes could be used for object localization. While it is easy to be the evil genius and devise an image that is unique enough such that it doesn't conform to any notion of a prototype, I wouldn't be surprised if 80% of the images we encounter on the internet conform to a few hundred scene prototypes. Of course the problem of learning such prototypes from data without prototype-labeling (which requires expert vision knowledge) is still open. Overall, I like the direction and ideas contained in this research paper and I'm looking forward to see how these ideas develop.

What is segmentation-driven object recognition?

2009-10-13T13:46:00.008-05:00

In this post, I want to discuss what the term "segmentation-driven object recognition" means to me. While segmentation-only and object recognition-only research papers are ubiquitous in vision conferences (such as CVPR , ICCV, and ECCV), a new research direction which uses segmentation for recognition has emerged. Many researchers pushing in this direction are direct descendants of the great J. Malik such as Belongie, Efros, Mori, and many others. The best example of segmentation-driven recognition can be found in Rabinovich's Objects in Context paper. The basic idea in this paper is to compute multiple stable segmentations of an input image using Ncuts and use a dense probabilistic graphical model over segments (combining local terms and segment-segment context) to recognize objects inside those regions.

Segmentation-only research focuses on the actual image segmentation algorithms -- where the output of a segmentation algorithm is a partition of a 2D image into contiguous regions. Algorithms such as mean-shift, normalized cuts, as well as 100s of probabilistic graphical models can be used produce such segmentations. The Berkeley group (in an attempt to salvage "mid-level" vision) has been working diligently on boundary detection and image segmentation for over a decade.

Recognition-only research generally focuses on new learning techniques or building systems to perform well on detection/classification benchmarks. The sliding window approach coupled with bag-of-words models has dominated vision and is the unofficial method of choice.

It is easy to relax the bag-of-words model, so let's focus on rectangles for a second. If we do not use segmentation, the world of objects will have to conform to sliding rectangles and image parsing will inevitably look like this:

(Taken from Bryan Russell's Object Recognition by Scene Alignment paper).

It has been argued that segmentation is required to move beyond the world of rectangular windows if we are to successfully break up images into their constituent objects. While some objects can be neatly approximated by a rectangle in the 2D image plane, to explain away an arbitrary image free-form regions must be used. I have argued this point extensively in my BMVC 2007 paper, and the interesting result was that multiple segmentations must by used if we want to produce reasonable segments. Sadly, segmentation is generally not good enough by itself to produce object-corresponding regions.

(Here is an example of the Mean Shift algorithm where to get a single cow segment two adjacent regions had to be merged.)

The question of how to use segmentation algorithms for recognition is still open. If segmentation could tessellate an image into "good" regions in one-shot then the goal of recognition is to simply label these regions and life becomes simple. This is unfortunately far from reality. While blobs of homogeneous appearance often correspond to things like sky, grass, and road, many objects do not pop out as a single segment. I have proposed using a soup of such segments that come from different algorithms being ran with different parameters (and even merging pairs and triplets of such segments!) but this produces a large number of regions and thus making the recognition task harder.

Using a soup of segments, a small fraction of the regions might be of high quality; however, recognition now has to throw away 1000s of misleading segments. Abhinav Gupta, a new addition to CMU vision community, has pointed out that if we want to model context between segments (and for object-object relationships this means a quadratic dependence on the number of segments), using a large soup of segments in simply not tractable. Either the number of segments or the number of context interactions has to be reduced in this case, but non-quadratic object-object context models are an open question.

In conclusion, the representation used by segmentation (that of free-form regions) is superior to sliding window approaches which utilize rectangular windows. However, off-the-shelf segmentation algorithms are still lacking with respect to their ability to generate such regions. Why should an algorithm that doesn't know anything about objects be able to segment out objects? I suspect that in the upcoming years we will see a flurry of learning-based segmenters that provide a blend of recognition and bottom-up grouping, and I envision such algorithms to be used a strictly non-feedforward way.

joint regularization slides

2009-09-19T03:48:00.004-05:00

Trevor Darrell posted his slides from BAVM about joint regularization across classifier learning. I think this is a really cool and promising idea and I plan on applying it to my own research on local distance function learning when I get back to CMU in October.

The idea is there should be significant overlap between what a cat classifier learns and what a dog classifier learns. So why independently learn two classifiers?

My paper on the Visual Memex got accepted to NIPS 2009 so I will be there representing my work in December. Be sure to read future blog posts about this work which strives to break free from using categories in Computer Vision.

On another note, today was my last day interning at Google (a former Robograd was my mentor) and I will be driving back to Pittsburgh from Mountain View this Sunday. Yosemite is the first stop! I plan on doing some light hiking with my new Vibram Five Fingers! I've been using them for deadlifting and they've been great for both working out and just chilling/coding around the Googleplex.

exciting stuff at BAVM2009 #1: joint regularization

2009-08-18T00:22:00.004-05:00

There were a couple of cool computer vision ideas that I was exposed to at BAVM2009. First, Trevor Darrell mentioned some cool work by Ariadna Quattoni on L1/L_inf regularization. The basic idea, which has also recently been used in other ICML 2009 works such as Han Liu and Mark Palatucci's Blockwise Coordinate Descent, is that you want to regularize across a bunch of problems. This is sometimes referred to as multi-task learning. Imagine solving two SVM optimization problems to find linear classifiers for detecting cars and bicycles in images. It is reasonable to expect that in high dimensional spaces these two classifiers will something in common. To provide more intuition, it might be the case that your feature set provides many irrelevant variables and when learning these classifiers independently much work is spent on removing these dumb variables. By doing some sort of joint regularization (or joint feature selection), you can share information across seemingly distinct classification problems.

In fact, when I was talking about my own CVPR08 work Daphne Koller suggested that this sort of regularization might work for my task of learning distance functions. However, I am currently exploiting the independence that I get from not doing any cross-problem regularization by solving the distance function learning problems independently. While regularization might be desirable, it couples problems and it might be difficult to solve hundreds of thousands of such problems jointly.

I will mention some other cool works in future posts.

Bay Area Vision Meeting (BAVM 2009): Image and Video Understanding

2009-08-14T01:43:00.004-05:00

Tomorrow (Friday) afternoon is BAVM 2009, a Bay Area workshop on Image and Video Understanding, which will be held at Stanford this year. It is being organized by Juan Carlos Niebles, one of Fei-Fei Li's students, and I will be there representing CMU. I have a poster about some new research and getting feedback is always good, but I'm really excited about meeting some of the other graduate students who work on image understanding. The Berkeley group has been pushing hard segmentation-driven image understanding so seeing what they're up to should be interesting. There will also be many fellow Googlers and researchers from companies in the Bay Area so it will also be a good place to network.

I look forward to hearing the invited speakers and the seeing the bleeding-edge stuff during the poster sessions. I'll try to blog a little bit about some of the coolest stuff I encounter when I get back.

Graphviz for Object Recognition Research

2009-08-07T23:54:00.007-05:00

Many of the techniques that I employ for object recognition utilize a non-parametric representation of visual concepts. In many such non-parametric models, examples of visual concepts are stored in a database as opposed to "abstracted away" as is commonly done when fitting a parametric appearance model. When designing such non-parametric models, I find it important to visualize the relationships between concepts. The ability to visualize what you're working on creates an intimate link between you and your ideas and can often drive creativity.

One way to visualize a database of exemplar objects, or a "soup of concepts," is as a graph. This generally makes sense when it is meaningful to define an edge between to atoms. While a vector-drawing utility (such as Illustrator) is great for manually putting together graphs for presentations or papers, automated visualization of large graphs is critical for debugging many graph-based algorithms.

A really cool (and secret) figure which I generated using Graphviz somewhat recently can be seen below. I use Matlab to write a simple .dot file and then call something like neato to get the pdf output. Click on the image to see the vectorized pdf automatically produced by Graphviz.

What does this graph show? Its a secret... (details coming soon)

Simple Newton's Method Fractal code in MATLAB

2009-07-31T02:58:00.004-05:00

Due to popular request I've sharing some very simple Newton's Method Fractal code in MATLAB. It produces the following 800x800 image (in about 2.5 seconds on my 2.4Ghz Macbook Pro):

>> [niters,solutions] = matlab_fractal;
>> imagesc(niters)

function [niters,solutions] = matlab_fractal
%Create Newton's Method Fractal Image
%Tomasz Malisiewcz (tomasz@cmu.edu)
%http://quantombone.blogspot.com/
NITER = 40;
threshold = .001;

[xs,ys] = meshgrid(linspace(-1,1,800), linspace(-1,1,800));
solutions = xs(:) + i*ys(:);
select = 1:numel(xs);
niters = NITER*ones(numel(xs), 1);

for iteration = 1:NITER
oldi = solutions(select);

%in newton's method we have z_{i+1} = z_i - f(z_i) / f'(z_i)
solutions(select) = oldi - f(oldi) ./ fprime(oldi);

%check for convergence or NaN (division by zero)
differ = (oldi - solutions(select));
converged = abs(differ) < threshold;
problematic = isnan(differ);

niters(select(converged)) = iteration;
niters(select(problematic)) = NITER+1;
select(converged | problematic) = [];
end

niters = reshape(niters,size(xs));
solutions = reshape(solutions,size(xs));

function res = f(x)
res = (x.^2).*x - 1;

function res = fprime(x)
res = 3*x.^2;

Spin Images for object recognition in 3D Laser Data

2009-07-15T22:58:00.008-05:00

Today's post is about 3D object recognition, that is localization and recognition of objects from 3D laser data (and not the perception/recovery of 3D from 2D images).

My first exposure to object recognition was in the context of specific object recognition inside 3D laser scans. In specific object recognition, you are looking for 'stapler X' or 'computer keyboard Y' and not just any stapler/computer keyboard. If the computer keyboard was black then it will always be black since we assume intrinsic appearance doesn't change in specific object recognition. This is a different (and easier!) problem than category-based recognition where colors and shapes can change due to intra-class variation.

The problem of specific object 3D recognition I'll be discussing is as follows:

Given M detailed 3D object models, localize all (if any) of these objects (in any spatial configuration) in a 3D laser scan of a scene potentially containing much more stuff than just the objects of interest (aka the clutter).

There was actually quite a lot of research in this style of 3D recognition in the 1990's with the belief that 3D recognition would be much simpler than recognition from 2D images. The idea (Marr's idea, actually) was that object recognition in 2D images would by preceded by object-identity independent 3D surface extraction so that 2D recognition would resemble this version of 3D recognition after some initial geometric processing.

However, it ends up that many of the ambiguities present in 2D imagery were also present in 3D laser data -- the problems of bottom-up perceptual grouping were as difficult in 3D as in 2D. Just because you have 3D locations associated with parts of an object does not make it any easier to tell where the object begins and where it ends (namely the problem of segmentation). It is this inability to segment out objects that resulted in the widespread usage of local descriptors such as SIFT.

Many of today's 2D object recognition problems rely on local descriptors which bypass the problem of segmentation, and it isn't surprising that the 3D recognition problem I described above was elegantly approached by A.E. Johnson and M. Hebert as early as 1997 via a local 3D descriptor known as a Spin Image.

The idea behind a Spin Image is actually very similar to that of a SIFT descriptor used in image-based object recognition. A spin image is a regional point descriptor used to characterize the shape properties of a 3D object with respect to a single oriented point. It is called a "spin" image because the process of creating such a descriptor can be envisioned as spinning a sheet around the axis defined by an oriented point and collecting the contributions of nearby points. Since a point's normal can be computed fairly robustly given its neighboring points, the spin image is highly robust to rigid transformations when defined with respect to this canonical frame. Since it is 2D and not 3D it does lose some discriminative power -- two different yet related surfaces chunks can have the same spin image. The idea behind using this descriptor for recognition is that we can compute many of these descriptors all over the surface of our object models as well as the input 3D laser scan. We then have to perform matching over these descriptors to create some sort of correspondences (potentially spatially verified).

(For a fairly recent overview of spin images as well as other similar regional shape descriptors and their applications to 3D object recognition check out Andrea Frome's ECCV 2004 paper, Recognizing Objects in Range Data Using Regional Point Descriptors.)

Spin images aren't a thing of the past, in fact here is a link to a RSS 2009 paper by Kevin Lai and Dieter Fox which uses spin images (and my local distance function learning approach!):
3D Laser Scan Classification Using Web Data and Domain Adaptation

Linguistic Idealism

2009-07-03T13:44:00.003-05:00

I have been an anti-realist since a freshman in college. Due to my lack of philosophical vocabulary I might have even called myself an idealist back then. However, looking back I think it would have been much better to use the word 'anti-realist.' I was mainly opposed to the correspondence theory of truth which presupposes an external, observer independent, reality to which our thoughts and notions are supposed to adhere to. It was in the context of the Philosophy of Science that I acquired my strong anti-realist views, (developing my views while taking Quantum Mechanics, Epistemology, and Artificial Intelligence courses at the same time). Pragmatism -- the offspring of William James -- was the single best view which best summarized my philosophical views. While pragmatism is a rejection of the absolutes, an abandonment of metaphysics, it does not get in the way of making progress in science. It is merely a new a perspective on science, a view that does not undermine the creativity of the creator of scientific theories, a re-rendering of the scientist as more of an artist and less of a machine.

However, pragmatism is not the anything-goes postmodern philosophy that many believe it to be. It is as if there is something about the world which compels scientists to do science in a similar way and for ideas to converge. I recently came across the concept of Linguistic Idealism, and being a recent reader of Wittgenstein this is a truly novel concept for me. Linguistic Idealism is a sort of dependence on language, or the Gamest-of-all-games that we (humans) play. It is a sort of epiphany that all statements we make about the world are statements within the customs of language which results in a criticism of the validity of those statements with respect to correspondence to an external reality. The criticism of statements' validity stems from the fact that they rely on language, a somewhat arbitrary set of customs and rules which we follow when we communicate. Philosophers such as Sellars have gone as far as to say that all awareness is linguistically mediated. If we step back, can we say anything at all about perception?

I'm currently reading a book on Wittgenstein called "Wittgenstein's Copernican Revolution: The Question of Linguistic Idealism."

Its all about the data

2009-06-29T22:00:00.002-05:00

I'm at Google this summer (Google summer internship round #2) because its where the data is. If you want to recognize objects from images you need to learn what objects look like. If you want to learn what an object looks like you need to have many examples of that object. You then feed those instances into an algorithm to figure out its essence -- what it is about that object's appearance that makes it that object. Google has the data and Google has the infrastructure to process that data, so I'm there for the summer.

A Shift of Focus: Relying on Prototypes versus Support Vectors

2009-06-19T14:52:00.011-05:00

The goal of today's blog post is to outline an important difference between traditional categorization models in Psychology such as Prototype Models, and Support Vector Machine (SVM) based models.

When solving a SVM optimization problem in the dual (given a kernel function), the answer is represented as a set of weights associated with each of the data-centered kernels. In the Figure above, a SVM is used to learn a decision boundary between the blue class (desks) and the red class (chairs). The sparsity of such solutions means that only a small set of examples are used to define the class decision boundary. All points on the wrong side of the decision boundary and barely yet correctly classified points (within the margin) have non-zero weights. Many Machine Learning researchers get excited about the sparsity of such solutions because in theory, we only need to remember a small number of kernels for test time. However, the decision boundary is defined with respect to the problematic examples (misclassified and barely classified ones) and not the most typical examples. The most typical (and easy to recognize) examples are not even necessary to define the SVM decision boundary. Two data sets that have the same problematic examples, but significant differences in the "well-classified" examples might result in the same exact SVM decision boundary.

My problem with such boundary-based approaches is that by focusing only on the boundary between classes useful information is lost. Consider what happens when two points are correctly classified (and fall well beyond the margin on their correct side): the distance-to-decision-boundary is not a good measure of class membership. By failing to capture the "density" of data, the sparsity of such models can actually be a bad thing. As with discriminative methods, reasoning about the support vectors is useful for close-call classification decisions, but we lose fine-scale membership details (aka "density information") far from the decision surface.

In a single-prototype model (pictured above), a single prototype is used per class and distances-to-prototypes implicitly define the decision surface. The focus is on exactly the 'most confident' examples, which are the prototypes. Prototypes are created during training -- if we fit a Gaussian distribution to each class, the mean becomes the prototype. Notice that by focusing on Prototypes, we gain density information near the prototype at the cost of losing fine-details near the decision boundary. Single-Prototype models generally perform worse on forced-choice classification tasks when compared to their SVM-based discriminative counterparts; however, there are important regimes where too much emphasis on the decision boundary is a bad thing.

In other words, Prototype Methods are best and what they were designed to do in categorization, namely capture Typicality Effects (see Rosch). It would be interesting to come up with more applications where handing Typicality Effects and grading membership becomes more important than making close-call classification decision. I suspect that in many real-world information retrieval applications (where high precision is required and low recall tolerated) going beyond boundary-based techniques is the right thing to do.

On Edelman's "On what it means to see"

2009-06-16T15:19:00.014-05:00

I previously mentioned Shimon Edelman in my blog and why his ideas are important for the advancement of computer vision. Today I want to post a review of a powerful and potentially influential 2009 piece written by Edelman.

Below is a review of the June 16th, 2009 version of this paper:
Shimon Edelman, On what it means to see, and what we can do about it, in Object Categorization: Computer and Human Vision Perspectives, S. Dickinson, A. Leonardis, B. Schiele, and M. J. Tarr, eds. (Cambridge University Press, 2009, in press). Penultimate draft.

I will refer to the article as OWMS (On What it Means to See).

The goal of Edelman's article is to demonstrate the limitations of conceptual vision (referred to as "seeing as"), criticize the modern computer vision paradigm as being overly conceptual, and show how providing a richer representation of a scene is required for advancing computer vision.

Edelman proposes non-conceptual vision, where categorization isn't forced on an input -- "because the input may best be left altogether uninterpreted in the traditional sense." (OWMS) I have to agree with the author, where abstracting away the image into a conceptual map is not only an impoverished view of the world, but it is not clear whether such a limited representation is useful for other tasks relying on vision (something like the bottom of Figure 1.2 in OWMS or the Figure seen below and taken from my Recognition by Association talk).

Building a Conceptual Map = Abstracting Away

Drawing on insights from the influential Philosopher Wittgenstein, Edelman discusses the difference between "seeing" versus "seeing as." "Seeing as" is the easy-to-formalize map-pixels-to-objects attitude which modern computer vision students are spoon fed from the first day of graduate school -- and precisely the attitude which Edelman attacks in this wonderful article. To explain "seeing" Edelman uses some nice prose from Wittgenstein's Philosophical Investigations; however, instead of repeating the passages Edelman selected, I will complement the discussion with a relevant passage by William James:

The germinal question concerning things brought for the first time before consciousness is not the theoretic "What is that?" but the practical "Who goes there?" or rather, as Horwicz has admirably put it, "What is to be done?" ... In all our discussions about the intelligence of lower animals the only test we use is that of their acting as if for a purpose. (William James in Principles of Psychology, page 941)

"Seeing as" is a non-invertible process that abstracts away visual information to produce a lower dimensional conceptual map (see Figure above), whereas "seeing" provides a richer representation of the input scene. Its not exactly clear what is the best way to operationalize this "seeing" notion in a computer vision system, but the escapability-from-formalization might be one of the subtle points Edelman is trying to make about non-conceptual vision. Quoting Edelman, when "seeing" we are "letting the seething mass of categorization processes that in any purposive visual system vie for the privilege of interpreting the input be the representation of the scene, without allowing any one of them to gain the upper hand." (OWMS) Edelman goes on to criticize "seeing as" because vision systems have to be open-ended in the sense that we cannot specify ahead of time all the tasks that vision will be applied to. According to Edelman, conceptual vision cannot capture the ineffability (or richness) of the human visual experience. Linguistic concepts capture a mere subset of visual experience, and casting the goal of vision as providing a linguistic (or conceptual) interpretation is limited. The sparsity of conceptual understanding is one key limitation of the modern computer vision paradigm. Edelman also criticizes the notion of a "ground-truth" segmentation in computer vision, arguing that a fragmentation of the scene into useful chunks is in the eye of the beholder.

To summarize, Edelman points out that "The missing component is the capacity for having rich visual experiences... The visual world is always more complex than can be expressed in terms of a ﬁxed set of concepts, most of which, moreover, only ever exist in the imagination of the beholder." (OWMS) Being a pragmatist, many of these words resonate deeply within my soul, and I'm particularly attracted to elements of Edelman's antirealism.

I have to give two thumbs up to this article for pointing out the flaws in the current way computer vision scientists go about tackling vision problems (in other words researchers too often blindly work inside the current computer vision paradigm and do not often enough question fundamental assumptions which can help new paradigms arise). Many similar concerns regarding Computer Vision I have already pointed out on this blog, and it is reassuring to find others point to similar paradigmatic weaknesses. Such insights need to somehow leave the Philosophy/Psychology literature and make a long lasting impact in the CVPR/NIPS/ICCV/ECCV/ICML communities. The problem is that too many researchers/hackers actually building vision systems and teaching Computer Vision courses have no clue who Wittgenstein is and that they can gain invaluabe insights from Philosophy and Psychology alike. Computer Vision is simply not lacking computational methods, it is gaining critical insights that cannot be found inside an Emacs buffer. In order to advance the field, one needs to: read, write, philosophize, as well as mathematize, exercise, diversify, be a hacker, be a speaker, be one with the terminal, be one with prose, be a teacher, always a student, a master of all trades; or simply put, be a Computer Vision Jedi.

Exemplars, Prototypes, and towards a Theory of Concepts for AI

2009-06-12T01:45:00.003-05:00

While initial musings (and some early theories) on Categorization come from Philosophy (think Categories by Aristotle), most modern research on Categorization which adheres to the scientific method comes from Psychology (Concept Learning on Wikipedia). Two popular models which originate from Psychology literature are Prototype Theory and Exemplar Theory. Summarizing briefly, categories in Prototype Theory are abstractions which summarize a category while categories in Exemplar Theory are represented nonparametrically. While I'm personally a big proponent of Exemplar Theory (see my Recognition by Association CVPR2008 paper), I'm not going to discuss the details of my philosophical stance in this post. I want to briefly point out the shortcomings of these two simplified views of concepts.

Researchers focusing on Categorization are generally dealing with a very simplified (and overly academic) view of the world -- where the task is to categorize a single input stimulus. The problem is that if we want a Theory of Concepts that will be the backbone of intelligent agents, we have to deal with relationships between concepts with as much fervor as the representations of concepts themselves. While the debate concerning exemplars vs. prototypes has been restricted to these single stimulus categorization experiments, it is not clear to me why we should prematurely adhere to one of these polarized views before we consider how we can make sense of inter-category relationships. In other words, if an exemplar-based view of concepts looks good (so-far) yet it is not as useful for modeling relationships as a prototype-view, then we have to change our views. Following James' pragmatic method, we should evaluate category representations with respect to a larger system embodied in an intelligent agent (and its ability to cope with the world) and not the overly academic single-stimulus experiments dominating experimental psychology.

On another note, I submitted my most recent research to NIPS last week (supersecret for now), and went to a few Phish concerts. I'm driving to California next week and I start at Google at the end of June. I also started reading a book on James and Wittgenstein.

Time Travel, Perception, and Mind-wandering

2009-03-30T14:51:00.003-05:00

Today's post is dedicated to ideas promulgated by Bar's most recent article, "The proactive brain: memory for predictions."

Bar builds on the foundation of his former thesis, namely that the brain's 'default' mode of operation is to daydream, fantasize, and continuously revisit and reshape past memories and experiences. While it makes sense that traversing the internal network of past experiences is useful when trying to understand a complex novel phenomenon, why exert so much work when just 'chilling out' a.k.a. being in the 'default' mode? Bar's proposal is that this seemingly wasteful daydreaming is actually crucial for generating virtual experiences and synthesizing not-directly-experienced, yet critically useful memories of alternate scenarios. These 'alternate future memories' are how our brain recombines tidbits from actual experiences and helps us understand novel scenarios before they actually happen. It makes sense that the brain has a method for 'densifying' the network of past experiences, but that this happens in the 'default' mode a truly bold view held by Bar.

In the domain of visual perception and scene understanding, the world has much regularity. Thus the predictions generated by our brain often match the percept, and thus accurate predictions rid us of the need to exert mental brainpower on certain predictable aspects of the world. For example, seeing a bunch of cars on a road along with a bunch of windows on a building pre-sensitizes us so much with respect to seeing a stop sign in an intimate spatial relationship with the other objects that we don't need to perceive much more than speckle of red for a nanosecond to confirm its presence in the scene.

Quoting Bar, "we are rarely in the 'now'" since when understanding the visual world we integrate information from multiple points in time. We use the information perceptible to our senses (the now), memories of former experiences (the past), as well all of the recombined and synthesized scenarios explored by our brains and encoded as virtual memories (plausible futures). In each moment of our waking life, our brains provide us with a shortlist of primed (to be expected) objects, contexts, and their configurations related to our immediate perceptible future. Who says we can't travel through time? -- it seems we are already living a few seconds ahead of direct perception (the immediate now).

My 2nd Summer Internship in Google's Computer Vision Research Group

2009-03-29T11:22:00.005-05:00

This summer I will be going for my 2nd summer internship at Google's Computer Vision Research Group in Mountain View, CA. My first real internship ever was last summer at Google -- I loved it.

There are many reasons for going back for the summer. Being in the research group and getting to address the same types of vision/recognition related problems as during my PhD is very important for me. It is not just a typical software engineering internship -- I get an better overall picture of how object recognition research can impact the world at a large scale, the Google-scale, before I finish my PhD and become set in my ways. Being in an environment where one can develop something super cool and weeks later millions of people see a difference in the way they interact with the internet (via Google's services of course) is also super exciting. Finally, the computing infrastructure that Google has set up for its researchers/engineers is unrivaled when it comes to large scale machine learning.

Many Google researchers (such as Fernando Periera) are big advocates of the data-driven mentality, where using massive amounts of data coupled with simple algorithms has more promise than complex algorithms with small amounts of training data. In earlier posts I already mentioned how my advisor at CMU is a big advocate of this approach in Computer Vision. This Unreasonable Effectiveness of Data is a powerful mentality yet difficult to embrace with the computational resources offered by one's computer science department. But this data-driven paradigm is not only viable at Google -- it is the essence of Google.