Project Activities

Please reiterate the goals and objectives of your efforts, and summarize the research and education activities you have engaged in that aim to achieve these objectives. Include experiments you have conducted, the simulations you have run, the collecting you have done, the observations you have made, the materials you have developed, and major presentations you have made about your efforts.

1. Basic Dome Research

One of the core goals of the project was to develop hardware/software infrastructure for dome applications, in order to make digital fulldomes a viable commercial platform for a variety of consumer applications. To this end, the teams from the Advanced Graphics Lab, the ARTS Lab, and the Santa Fe Complex combined efforts to work several projects to develop infrastructure that would allow us to explore the full potential of digital fulldomes. Development of a software API for dome development Team leader: Dr. Joe Kniss (Advanced Graphics Lab) Partners: AGL, ARTS Lab Dr. Joe Kniss led a team of graduate students (Jeff Bowles, Matthew Dosanjh, Vahid Normoofidi, Andrei Buium) to develop an OpenGL software platform that would allow real-time rendering on curved surfaces. The basic idea of the software was to take a typical OpenGL application and intercept the rendering code to render to an off-screen buffer instead of the screen, as it would normally render. This off-screen buffer was then distorted by virtually mapping it to a spherical structure on the graphics hardware, which was used by the GPU to compute what each projector in the dome should display in order for the dome visualization to appear correct to a user standing in the center of the dome. The result was a framework that allowed us to run real-time rendering applications in dome environments without distortion, and all one needed to do was to modify the existing code to have this new rendering path. The project, called DomeGL, was a success and it enabled the students to port several applications to the dome quickly and easily. Once the core code of DomeGL had been developed, the graduate students worked on a variety of different applications to demonstrate the feasibility of the dome platform for effective visualization, such as interactive particle systems, games, etc. A few examples of these are shown in Figs. 1 and 2. The students also interfaced to various control devices, such as the Nintendo Wiimotes, in order to control the display. The next steps of DomeGL development are to try to make this a stand-alone piece of code that would automatically intercept OpenGL commands from a legacy application in order to enable dome rendering for existing applications without modifying the code.

Fig. 1. Researchers view the ARTS Lab dome running a DomeGL program. DomeGL allows real-time applications to be seamlessly projected onto the dome and automatically corrects for the curvature of the dome.

Fig. 2. Another view of the dome running a physically-based particle simulation. The framework was fairly general, so porting new applications to the dome became a straightforward process.

Portable dome development Team leader: Stephen Guerin (SF Complex), David Beining (ARTS Lab) Partners: SF Complex, ARTS Lab, Lumenscape One of the drawbacks of digital fulldomes as raised in the brainstorming workshop is that they can be expensive and not many people have access to this technology. To this end, the teams from SF Complex and ARTS Lab worked on low-cost, portable dome solutions that would enable more people to experience dome technology. The Santa Fe Complex (in association with Joe Abraham Dean from Lumenscape) worked on a clothbased dome that could be easily transported and set up in about a day (Fig. 3). This portable dome has market potential since it allows users to set up dome environments at a relatively small cost. Joe Dean and others are looking into the marketing of such dome structures. The ARTS Lab team (David Beining, Jon Strawn, and others) worked on a smaller dome for immersive visualization that was made of wood and other low cost materials (Fig. 4). Both domes were portable and were presented at several venues. For example, the ARTS Lab portable dome was showcased at the Mesa Del Sol opening for the new digital media center.

Fig. 3. A cloth dome designed and built at the Santa Fe Complex. The fabric and structure is very lightweight, making the dome very portable for deployment in remote venues. This allows people who normally would not have access to a full dome to experience full dome content.

Fig. 4. A low-cost dome is built in the ARTS Lab from wood and diffuse screen segments. This dome is more portable and could be moved to various locations such as the Mesa del Sol building or the Santa Fe State Capitol. These kinds of low-cost dome solutions could help introduce dome technology to the masses.

2. Research Beyond the Dome

One of the concerns raised at the brainstorming workshop was that dome technology was fundamentally limited by the expensive hardware of a physical dome, which was not available to a wide range of users. For this reason, the team in parallel began do explore how dome content could be displayed on arbitrary surfaces, moving beyond the dome. The basic idea is that by allowing dome content to be displayed in an arbitrary environment (such as someones living room), dome content would become more prevalent because it would break the chicken-and-egg cycle that normally afflicts this kind of media: no one makes dome content because no one has a dome to view it on, and no one buys domes because there is not enough content. This idea posed several technical challenges which had to be addressed by the research team. Calibration of projectors in multi-surface environments Team leaders: Dr. Pradeep Sen (Advanced Graphics Lab), Stephen Guerin (SF Complex) Partners: AGL, SF Complex Domes present an ideal surface onto which to project content because their shape is well known. This allows software such as DomeGL to correct for the aberrations introduced when projecting onto the curved surface in order to display content correctly. When projecting content onto arbitrary surfaces, such as the corner of a room, the problem becomes much harder. The purpose of this part of the project is to calibrate the projector and camera automatically, so that the image viewed from the point-of-view of the camera looks correct despite the distortions introduced by the room environment. This problem was tackled by both teams in AGL (Dr. Sen working with graduate student Vahid Noormofidi) and at the SF

Complex (Stephen Guerin working with Cody Smith, Steve Smith, Bjrn Swenson, August Swanson, Skyler Swanson, Scott Wittenberg). This technique could also be used to produce a correct-looking image after bouncing the projected light off of a spherical mirror onto some surface, in order to extend the field-of-view of the projector for example. Finally, because the surfaces could be complex, the process of calibration had to happen automatically. Fig. 5 shows an example of the typical setup for this problem.

Fig 5: A diagram of the typical setup for projector-camera calibration to allow projection onto arbitrary surfaces. In this typical configuration, a computer controls a camera and projector pair. The projector is placed either directly at some surface or reflected off of a spherical mirror, which can be used to enlarge the projection angle to almost 180 degrees. Since the mirror significantly distorts the image, the distortion needs to be removed through a calibration process such as the one proposed. In order to get the correct calibration, we first illuminate a set of predefined binary patterns onto the surface from the projector which are then captured by the camera. Fig. 6 and 7 show examples of what these patterns look like. These patterns are chosen in a way such that each projector pixel is given a unique binary code, which allows us to find the correspondence map between the projector and camera using custom software developed by both the AGL and SF Complex teams. This software was developed independently by both teams (with some discussion between them) so they came up with different solutions for some of the problems that come up as described in more detail below. The basic idea is that by projecting enough of these patterns, we can find a high-resolution mapping between the projector and camera that allows us to distort images in the projector space so that they appear correctly from the pointof-view of the camera.

Fig. 6: Projecting some binary patterns on surface. These patterns represent horizontal and vertical bars from the perspective of the projector, but because they are landing on the corner of the room they are distorted from the cameras perspective. The goal of this work is to remove the distortion.

Fig. 7: Binary code patterns projected onto the corner of a room. Each pixel in each picture represents either a 0 or 1 in the camera-to-projector map. In the AGL implementation, a PointGrey Grasshopper camera (model # GRAS-02K2M/C) was used to do the acquisition. This camera has a resolution of 640 x 480 at 60 frames per second, in fairly high-quality gray-scale. The SF Complex implementation, on the other hand, used a Sony PS3-eye webcam with a similar resolution of 640x480 at 60 frames per second. In the case of the Sony cam, the lens was retrofitted with a fish-eye lens with roughly 160 degree viewing angle. After projecting the binary patterns, the computer has to find correspondences between pixels in the camera and pixels of the projector. In other words, it needs to know which projector pixels it needs to illuminate in order to produce a desired camera image. The nave approach is simply to make a reverse look-up table for every projector pixel that indicates what camera pixels a specific projector pixel maps to, which would work if we were using very high resolution cameras. Since both the teams at AGL and at SF Complex were using low resolution cameras (640 x 480 resolution), this creates a lot of holes in the mapping where it is unclear where the projector pixels map to. For example, one camera pixel will see projector pixel (x, y), and the adjacent camera pixel next to it would see projector pixel (x + x, y + y), where the ( x, y) offset can be substantial (i.e., much greater than 1) because of the low resolution of the camera. Therefore there are many projector pixels (the ones in between (x, y) and (x + x, y + y)) that have not been properly tagged.

To fill in these holes in the correspondence map, there are a several approaches that can be used to interpolate between values. The two teams did different things at this point. The AGL team used the texture hardware to do the interpolation, since graphics hardware can automatically resample texture signals during the rendering process. To do this, they expressed the correspondence map as a texture in the graphics card and used the GPU itself to perform the resampling and find the correspondences for projector pixels in between the camera pixels. Since the correspondence map was stored in the hardware, it could be used as a fast look up table to distort images in real-time by simply sending the images to the GPU and then doing a real-time indirect look-up using the correspondence map. This simple solution was easy to implement and allowed the team to distort the imagery in real-time in a very fast way. Results from the AGL implementation is shown in Figs. 8 and 9.

Fig. 8: A test image of a black square on a white background, viewed by the camera. Note that both the projector and the camera are very tilted with respect to the projected surface, which means that normally the image of the square from the point-of-view of the camera would look very distorted. The proposed algorithm corrects for this perspective and produces a square which looks correct from the point-of-view of the camera.

Fig. 9: A more complex example where we stream video frames to the graphics card and do real-time distortion to project them correctly. On the left is the image from the point of view of the projector, which is distorted because we are correcting for the image from the point-of-view of the camera. On the right is the camera image. At this stage we are not concerned about the borders of the image (which are corrected by blending in neighboring projectors, see section below) but we are making sure we are producing a correctly rectified image. The fact that we can do this with a video stream is a transformative technology in terms of

The SF Complex team used a different solution to fill in the missing samples. They interpolated the correspondence map by using Delaunay triangulation (a way to tessellate a set of points into well-behaved triangles) and then interpolated using Barycentric coordinates. This is a common approach for nonuniform signal interpolation. An example of the tessellation provided by the Delaunay triangulation is shown in Fig. 10. Figs. 11 and 12 show some sample results acquired by the SF Complex team.

Fig. 10: A sparse set of (red) projector coordinates connected to their nearest neighbors by a triangular mesh. The points with no data (black) are later filled in with interpolated values. This triangulation had unexpected benefits later on, when the triangulation is displayed in camera view.

Fig. 11: (left) a warped grid projected onto a non-flat surface. The grid lines look orthogonal to each other because the distortion-correction has been applied. (right) the warped image displayed by the projector.

Fig. 12: Preliminary tests involving the calibration of three projectors. The results are still a bit blurry. One benefit of finding the closest neighbor triangulation as proposed by the SF Complex team is that it can help identify reflections. For example, a projector pixel might illuminate a part of the wall that then reflects onto a different part of the scene, so the same projector pixel is seen by two very different regions in the camera. To do this, we can view the triangulation in camera view instead of projector view. When this is done the points are still connected to their closest neighbors in projector space but they appear on the screen in camera space. Reflections show up in the camera view as really long skinny triangles. It is possible to remove reflections by simply removing the points connected to the long skinny triangles. Although this process does lose some information, it can help improve the quality of the final image.

Fig. 13: The removal of reflected coordinates using a tessellation. Note in the top right there are lots of reflections on the left wall, visible as long skinny triangles.

Stitching and edge-blending across multiple projectors Team leader: Stephen Guerin (SF Complex) Partners: SF Complex, Ambient Pixel The process for multiple projectors calibrates each projector separately and then stitches the correspondence maps together. In theory, one could handle an indefinite number of projectors, but in our implementation we are currently limited to three projectors because the Matrox triple head2go device that is being used to split out the video in the Santa Fe Complex implementation only has three outputs. The display looks like a triple wide screen to the computer, and it sends of the display to each projector. Another potential option in the future would be to use multiple graphics cards instead.

Fig. 14: (left): Picture going into the triple head2go device without blending. This is what the projectors will display. (right) the picture taken by the camera, which shows what the user would see from this position. Note the bright region in the middle which means that we need to blend the displays together. As shown in Fig. 14, when one looks at the display stitched from multiple projectors they immediately notice that the overlapping region is significantly brighter than the non-overlapping regions. This can be compensated for in the software, by finding how far each overlapping pixel (in the camera view) is from its closest non-overlapping pixel and then weighting its brightness accordingly.

Fig. 15: (left) Two un-blended projectors, note the brightness in the overlapping regions in the middle. (right) Two projectors blended using the brightness of the image. Note this leaves a noticeable dark spot in the result.

There are numerous established approaches for edge-blending overlapping projectors to create a single continuous display surface (Raskar et al, 1998; Bourke 2004). Many works assume edge-blending is occurring on a simple geometric surface (plane, dome, curved wall, etc) for which a parametric geometry can be fitted. A second common assumption is that the projectors have simple horizontal or vertical overlaps as depicted in Figure 16.

Fig. 16: Traditional, horizontally-aligned projectors

As we project multiple projectors at arbitrary angles onto non-uniform surfaces like the pleated LumenScape Dome or arbitrary room surfaces, we must warp the projected imagery in the projectors to compensate for complex geometric surfaces, as described in the previous section. This means that we must blend the overlapping projector images in cases when their areas of intersection can be extremely non-uniform as depicted as an example in Figure 17:

Figure 17: Overlapping Unaligned Projectors

Previous work on edge-blending across different projectors has used edge-blending transition masks to gradually blend from one image to another. When the images are aligned horizontally, this is a relatively straightforward calculation and it generates a mask that looks something like that shown in Fig. 18.

Fig. 18: Alpha mask to cross-fade between projectors

It is common to generate the transition mask like this using a S-shaped curve. AmbientPixels current algorithm uses a sigmoid function as is typically found in activation functions in neural networks. The general form of this function is:

(1)

where a is a scaling term that manipulates the steepness of the transition mask. Fig. 19 below compares two functions when a is set to -2 and -7. Note that the sigmoid ranges between 0 and 1 with a midpoint yvalue of 0.5 when x = 0.5 and that the function has asymptotes at 0 and 1.

Fig. 19: Sigmoid currently used by Ambient Pixel for generating transition masks
An alternative transfer equation is used by Bourke (2005) that uses a conditional if statement is shown in Eq. 2.

(2) One advantage of this transfer function is that the scaling term P allows the function to scale all the way back to a linear function as shown in Fig. 20.

Fig. 20: Transition function that allows for linear relationship (Bourke, 2005)
Regardless of the transition function used, the key challenge is how to calculate the transition masks when the areas of overlap are not simply horizontal or vertical. Consider the overlapping projectors shown in Fig. 21.

Fig. 21: Three overlapping projectors

For pixels on a surface that are illuminated by only one projector, we want that pixels output to be full strength, or 1.0. For pixels that have more than one projector illuminating it (as registered from a camera), we will need to generate a weighted sum of the projector pixels making sure they always add to 1.0 intensity so that perceptually this region is the same intensity than the unblended regions. Our approach is to calculate for each pixel in an intersecting region the minimum distance to a pixel in each projector that is only illuminated by that projector. In other words, we are finding the minimum distances to the borders of the intersecting areas. Figure 22 shows gray pixels that indicate the intersecting regions and sets their gray scales based on how far they are from a projector boundary.

Fig. 22: Individual projector images with distance coloring

Once these distances are stored, we sum them up for each pixel and then have each projector pixel contribute its RGB value based on its distance / total sum of distances. This fractional value between 0 and 1.0 is applied as an input X value to the sigmoid in Fig. 19 to get the amount of contribution from this pixel. Although we are currently doing the weighted composite in RGB space, it is probable that we would have a better perceptual blend if we did this in HSV space. Once these transition masks are calculated, they can be stored as alpha channels for real-time compositing and warping using the GPU and run in tools like WarpPlayer and a QuartzComposer Composition allowing for further chroma and luminance adjustments.

Making Immersive Lighting a Reality with Smart Rooms Team Leader: Dr. Pradeep Sen (Advanced Graphics Lab) Partners: AGL, Rensselaer Polytechnic Institute (RPI) One of the challenges of making any technology economically feasible is to find ways that it can be accepted by the mass public by making it easy to integrate into their normal lives. The kinds of technologies which we are discussing so far face hurdles because they are transformative in nature, and by definition this means that most people do not see the direct application of such technologies into their normal lives in the immediate future. As was raised in the brainstorming workshop, many people do not

know why digital fulldomes (or some of the other immersive technologies) would be useful to them in their lives. In order for this PFI to be a success, this is an important matter that needs to be addressed. For this reason, Dr. Pradeep Sen and his team at AGL has been investigating how to ease into these immersive technologies by leveraging work done on Smart Lighting. We believe that if we start integrating our technologies along with Smart Lighting (which is already starting to have economic impact) it might be easier to evolve the technology to the point where everyone has immersive-capable displays in their living room or office. To give some background, Smart Lighting rooms are rooms that are equipped with a set of novel lighting sources (often LED or laser projection systems) and variety of light-sensitive sensors positioned in some configuration around the room. This is very similar to the multi-surface display idea, except that in Smart Lighting the light sources can be heterogeneous, as opposed to only projectors in the multi-surface approaches. Dr. Sen is an investigator on an NSF Engineering Research Center grant (#0812056) on Smart Lighting, being led by Dr. Robert Karlicek at RPI, so in this part of the project he collaborated with the team over at RPI who have the infrastructure to do these kind of experiments. Specifically, Dr. Sen has been studying how Smart Lighting rooms can be used to create immersive lighting experiences for users. The basic idea is that the Smart Lighting rooms have a combination of light sources and sensors, very similar to the camera-projector pairs described in the calibration section. Using very similar techniques as described earlier with shining binary patterns, it is possible to measure the full light transport matrix for these complex environments, which tell us how the light from the projectors arrives at the cameras. This is a more general form of the correspondence map we described earlier. In this case, since light transport can be modeled as a linear process (by ignoring the wave and quantum effects of light transport), we can write an equation of the form: y = Au (3)

to describe the light transport, where u is the light leaving the light sources, y is the light measured at the camera and other sources, and A is a matrix that defines how each light source communicates with each camera pixel or light sensor. By this formulation, it is possible to invert the process and determine which lighting configuration is needed to achieve a desired signal at the camera. Again, this is similar to the inversion described in the calibration section to produce a mapping between the projector pixels to the camera pixels, but it is much more general and offers many new possibilities. For example, by formulating the problem in this way, we can introduce constraints into the least-squares solver such as minimizing the norm of u which would find the lowest-power solution to the problem. This is extremely interesting because it helps motivate the use of these immersive lighting environments to a typical user. For example, as opposed to the typical applications we think about (such as active visualization, entertainment, etc) the user might simply want to illuminate the room while using the amount of power possible. Solving with the constraint of also minimizing the norm of u gives us the lowest power usage for a desired lighting configuration. We can also use the lighting in this way to track users or create other forms of interaction between the user and the lighting. This whole room can be controlled by a computer similar to what was done in the calibration work, using the system shown in Fig. 23.

Fig. 23: Block diagram of the lighting control system

To demonstrate the feasibility of these ideas, the team has put together a smart lighting room at RPI (shown in Fig. 23) which can serve as a test bed for experiments. Using this testbed, they have been able to demonstrate that they can achieve desired lighting configurations (in essence projector calibration) using the formulation in Eq. 3. This is very exciting, because it means that it is the first demonstration that these kinds of technologies can be integrated into real-room environments in a cost-affordable manner.

Fig. 24: Smart Lighting room for experiments put together at RPI

New input devices for navigating virtual environments Team Leader: Dr. Joe Kniss (Advanced Graphics Lab), David Beining (ARTS Lab) Partners: AGL, ARTS Lab Finally, in this last section we describe work led by Dr. Joe Kniss on novel interfaces for exploring these immersive environments, whether they be on arbitrary surfaces or domes. One of the challenges faced by the technology is that the new realm of immersion creates real problems in terms of interaction (a mouse does not work very well any more). For this reason, Dr. Kniss and a team of graduate students (Jeff Bowles, Matthew Dosanjh, Vahid Normoofidi, Andrei Buium) and some of the ARTS Lab staff (David Beining and Jon Strawn) have been working on innovative interfaces to allow people to navigate these immersive environments. The first interface developed was a flying skateboard interface, which features a suspension-mounted skateboard upon which the user stands. By moving their body in different directions, the user can control the direction of motion within the virtual environment. An example of the skateboard interface in use is shown in Fig. 25.

Fig. 25: Dr. Joe Kniss (AGL) demonstrates the skateboard interface that he built along with a set of graduate students and the ARTS Lab team. The skateboard interface allows more intuitive navigation of immersive environments.

The second interface developed was a hydraulically-actuated platform interface, which extends the ideas of the skateboard interface by being able to provide the user with feedback. An example of this interface is shown in Fig. 26.

Fig. 26: Dr. Joe Kniss (AGL) demonstrates a hydraulic-actuated platform interface that he built with a set of graduate students and the ARTS Lab team. The platform allows the user to get some feedback about the virtual environment.

3. Dome for Education and Workforce Development

An important goal for our project was to leverage the dome technology to help educate New Mexicans and establish a high-tech workforce that could utilize these novel technologies. This side of the project had several thrusts, which are listed below. Construction of the IAIA digital fulldome Team leaders: Ethan Bach (IAIA), David Beining (ARTS Lab) Partners: IAIA, ARTS Lab Over the past year, the teams from IAIA and ARTS Lab have built and set up a digital fulldome at IAIA, which is now in operation and being used to train the next generation of dome users how to work with this new technology. The dome can be raised or lowered, and can also be tilted automatically depending on the kind of visualization it is presenting. This makes the dome very flexible and the space underneath very configurable for different purposes. Photographs of the completed dome structure are shown in Figs. 27 and 28.

Fig. 28: A wide-angle picture of the IAIA dome suspended in the air. The IAIA dome is designed to be rotated and lifted up and down, which allows it to be used in a wide variety of configurations and for a wide range of immersive applications.

Fig. 29. The IAIA dome tilted at a 90-degree angle, which is ideal for applications which require immersive navigation.

Teaching Fulldome Production Courses Team leaders: David Beining (ARTS Lab) Partners: ARTS Lab The team from ARTS Lab led by David Beining has been offering dome-related courses. This current semester, the team is offering a course focusing exclusively on immersive media and interactive visualization. The purpose of this course is to teach students at UNM about the exciting applications that are being enabled through fulldome technology, and the students have the opportunity to put together dome content themselves as part of the course. More information can be found at the course webpage at: http://imivs11.wordpress.com/

4. Economic Development
Team leader: David Beining (ARTS Lab) Partners: ARTS Lab, AGL, SF Complex Economic development is a driving force for the PFI project. David Beining from the ARTS Lab has been helping to lead this effort by being the evangelist for dome technology. Several presentations on dome technology have been given to senior level state officials (such as Bill Richardson, former governor) and to high tech companies who are interested in immersive media (such as Hewlett Packard, as shown in Fig. 30).

Fig. 30: The ARTS Lab team meets with representatives from HP to discuss the potential of dome technology.