Eric Emer UROP Faculty Supervisor: Bei Xiao Term: Fall 2011 09/28/11 Perception Science: Understanding Material

Recognition Project Overview If you go to a donut shop, you will have no problem selecting your favorite donut, but any state-of-the-art robot will fail it. This is because humans are very good at material recognition, namely, deciding what an object is made of. Material recognition is important for decision making in daily life such as deciding whether food is raw or cooked, shopping for clothes, designing furniture, deciding how to drive or walk on icy road. Material recognition is also an important problem in computer and robotic vision. A successful algorithm of material-based image search will be useful in matching material properties of clothes while doing online-shopping. A self-guiding robotic agent that has a good material recognition system can recognize icy road while driving. In computer graphics, learning how humans perceive materials will help develop perceptual-driven rendering algorithms. However, most previous work on recognition has been done on object recognition. Little is known about material recognition. Our research will be in the field of material recognition, and what produces its accuracy. The immediate goal is to construct a large database of materials, and gather experimental data regarding how people perceive them. The method for constructing the database will be by taking digital photographs. The methods for gathering experimental data will be by running subjects through tests in the lab, and online through Amazon Mechanical Turk.

The research will be conducted under the faculty supervision of Bei Xiao, Ph.D. There may also be side projects administered by Derya Akkanyak Yellin. The research will be conducted in several locations. Running subjects will take place in 46-4115, and the offices and nearby lab rooms of Bei Xiao and her colleagues. Taking photos of everyday objects will take place around the city of Boston. Programming aspects will probably take place in a quiet location of my choice.

Personal Role & Responsibilities I will do the following work for the project. First, I will take high-quality photos of everyday objects under natural lighting in the Boston area. Most databases existing so far focused on object recognition. This proposed dataset would focus on capturing the variety and details enabling material perception. Second, to get myself familiar with visual psychophysics, I will help scheduling and recruiting human subjects to run the existing material discrimination experiments programmed in MATLAB. I will also help analyzing the data using MATLAB. Third, I will write programs for designing onlineexperiment via Amazon Mechanical Turk. This will involve using Javascript, Python, and Amazon Mechanical Turks API to design the experiment, collect the responses and analyze the data.

Goals The goal of the research project is to understand the information in images that allows humans to make material judgment. In particular, we will ask: how do images convey information about the identity and properties of materials seen in the images?

To do this, we will collect a large dataset of objects that contain a variety of materials in the real world and use human visual psychophysics to measure material discrimination and categorization. We will collect the psychophysical data both in the laboratory running subject one at a time and online using crowd-sourcing method via Amazon Mechanical Turk.

Personal Statement This research project interests me greatly. The challenges of computer perception are enormous, and understanding the mechanics and science behind human perception of materials would constitute a tremendous first step to this confronting challenge. I am also interested in the opportunity to use new technologies/ventures such as developing and optimizing an HIT designed for crowd-sourcing. This UROP project can help me learning a few new things. Firstly, I will learn visual psychophysics, an important method in studying human cognition. Secondly, I will have the opportunity to learn techniques in crowd-sourcing and improve my ability in computer programming such as learning MATLAB and Javascript. Thirdly, I will become familiar digital photography and learn more about computational photography. Most importantly, I will have the chance to complete a project that would potentially contribute to knowledge for a very unexplored field.