CC BY
6Put on your glasses and press right mouse button
AR-based user interaction using laser pointer tracking
1 12 Christian Tonn , Frank Petzold , Dirk Donath
1 2
' Bauhaus University Weimar, Germany
1 2
http://uni-weimar. de/jpai | http://infar. architektur. uni-weimar. de Designing and planning in existing built contexts
The use of computers in architectural practice is widespread and complements traditional tools such as drawings and physical models. Computers can support the design process in three dimensions with the help of a building information model (BIM), however, the output devices are still generally 'traditional' 2D devices, such as screens or plotters. Rapid technological advances are however enhancing the possibilities of architectural design, for instance three-dimensional VR and AR environments. Immersive and semi-immersive projection displays, such as CAVEs™ and workbenches are already used to support virtual reality applications in many professional domains, including the field of architecture. The visualization of data using such displays requires dedicated rooms for setting up non-mobile screens, and allows one to interact with purely virtual information only.
The current software and hardware market in this field is characterized by a variety of individual products. Most of them are adaptations of CAAD-Systems or specific computer-supported solutions for new building or adaptations of products from other fields, e.g. VR/AR supported design in the automobile sector or SAR applications (Bimber and Raskar, 2005).
The Framework
Based upon the analysis and the identification of deficiencies, an IT concept was developed with the aim of providing a set of tools for the computerassisted surveying and designing of buildings. The individual tools provide a continuous, evolutionary, flexible and dynamically variable system which address aspects ranging from the initial site visit to 1:1 designing on-site. The concept of the system follows a modular principle. Different modules for information capture, design and planning can be combined as required. The individual modules form a continuous, extensible, flexible and in real-time dynamically adaptable system, which covers all aspects from the initial site visit to detail planning. Each tool was developed for an individual aspect taking into account its role and the requirements of the entire planning process. This concept (see fig. 1) was developed in an earlier research project "Collaborative Research Center 524" and adapted and expanded for this framework (Petzold and Donath, 2004; Donath, Petzold et al., 2001).
Application clients - building surveying
Figure 1
Schematic diagram of the software framework
igning and planning
Figure 2
Examples of application clients and service clients
Selected aspects have been realised as prototypes on an experimental platform and demonstrate the system feasibility. The experimental platform consists of a server and extendable series of different tools in the form of clients (see fig. 2). The server is used to store data permanently and centrally and interacts with the various clients. Redundant data storage within the clients reduces the level of continuous data-transfer load on the server (Petzold et al., 2008).
The project "spatial augmented reality for architecture" is focussing onto one aspect of the research area "Planning and Building in Existing Built Environments" - the support of surveying and designing on site with and within existing buildings.
Figure 3
Current PTZ camera prototype
Figure 4
Initial interaction on site
The input device
As part of the research project, we developed a laser-pointer tracking technique that utilises a Pan-Tilt-Zoom camera (see fig. 3), which enables the laser pointer to be used as an input device in architectural environments (Kurz et al., 2007). A zoom camera captures the bright point of the laser pointer on the surface of the building and its position in 3D-space then calculated from the dimensions of the room, which are known. If the point moves outside of the zoom-camera's field of vision, its movement is either continued automatically or its position is determined using a second fish-eye
context camera. In initial pilot testing using the laser-pointer tracking system, gestures were used to activate particular functions, for example a "time gesture", where the point remains on the same spot for more than 3 seconds, was used to activate the distance measurement function at this point (see fig. 4). In practice this proved to be a little too imprecise - it's difficult to hold still for so long. One solution is to add buttons to the laser pointer.
For this we chose to use a Nintendo Wiimote®. In addition to several buttons, this device also has a movement sensor, an IR-camera and Bluetooth interface. In addition, WiiUse, the freely available C++ control library (Michael Laforest 2007, http://www.wiiuse.net/), made it possible to instantly use the devices features for our own purposes. The wireless Bluetooth connection also makes it possible to move around freely within a radius of approximately 10 metres, which is sufficient for our purposes. A further client was developed and integrated into the software framework that allows other application clients to access the Wiimote® functionality. In the end it proved easier to dismantle the laser pointer and add this to the Wiimote® rather than adding buttons to the laser pointer (see figs. 5, 6).
The on-site interaction concept
Working directly on site with the building substance around one is in certain respects fundamentally different to working at a desk and screen. When working on a CAAD-model on a 2D-display, one has an on-screen user interface with menu bar, toolbars and palettes, and one or more views of the CAAD model. Unlike the screen and desktop, when working within a building, the display surfaces are themselves a section of the architecture itself, which can also have a complex surface geometry. There is no frame around the screen, and the Start-Menu in the bottom left corner disappears among the overlay of projections.
Working on site also places new demands on the presentation quality and on interaction. When one works at a scale of 1:1, one also wants to experience the design, colour and materials of the space at true scale. Permanent menus and user interface destroy this illusion. Other forms of interaction such as gesture-recognition, speech input and context menus interfere less with the projection of the information.
Pie-menus
One possible approach to keeping the projected area free of distracting widgets is to use contextual pie-menus that appear as required. Context pie-menus are already familiar as they are used for browser plug-ins and in various operating systems. The available options and tools are arranged as icons around a circle centred on the dot of the laser beam (see fig. 7). By moving the laser pointer towards a certain point on the circle, the appropriate option is selected or a sub-menu pops up. With the help of text descriptions for the icons (see fig. 8), the designer quickly gets used to using the laser pointer to select a certain option. In this way, selecting a tool also helps the user to establish a laser pointer gesture.
Figure 7 Figure 8
Pie-menu in use on site Desktop view of the pie-menu with additional
text descriptions
According to Callahan et al. (1988), pie-menus are easier to use than rectangular linear context menus, as each option is the same distance away from the centre. One need only move the pointer into the right pie-section of the circle to select the function.
Application in the Software Framework
An "InteractionKernel" was implemented as a software library that translates the various input data from the input device (laser-pointer tracker, Wiimote® or mouse) and displays the necessary UI element in the 3D CAAD model. All application clients in the software framework can use these libraries to enable them to be controlled remotely. The "InteractionKernel" also controls which application client (e.g. "Colored Architecture" or "Designing with Images") currently has input focus in the network. The "InteractionKernel" contains a central pie-menu from which all functions can be started that is accessed from the middle mouse button or the Wiimote® "Home" button. This menu provides access to all functions in the application client and the option to switch to another application client. The pie-menu also serves to activate the interaction modes, the cursor symbol denoting the currently active function. The mouse or Wiimote® button applies the function. The following section describes a selection of possible on-site planning applications.
Colored Architecture
The software prototype "Colored Architecture" (CA) was developed to support the design process for colours and materials (Tonn 2005 and Tonn 2008). The tool addresses the deficits of digital colour and material design and supports digital planning with colour and materials from the initial design, through the planning phase to specification. The digital support of colour and material design in CA uses and adapts existing strategies, instruments and representations, e.g. alternative variants, colour studies and colour relationships such as harmonies and contrasts. To reliably assess and evaluate the results of colour and material choices, integrated radiosity visualisation is employed as it is able to represent interactions between different surfaces such as reflections.
The following examples show work undertaken in the windowless ARVis-Lab at the Bauhaus-Universität Weimar. The hardware setup consists of two wide-angle video projectors mounted on the ceiling (see fig. 9), the Pan-Tilt-Zoom camera (see fig. 1), a computer that drives the projectors and a mobile laptop on which the application clients run. The pie-menu can be used to start the different functions of the "Colored Architecture" software prototype (see fig. 7). In addition to a drawing function that can be used, for example, to trace elements on a wall such as a crack or damages (see fig. 10), and a measuring tool (see figs. 14-15), a number of colour design functions are available. By way of example, a colour picker context menu has been implemented in order to be able designate the colour of surfaces or areas thereof on site at true scale (see figs. 11-12).
Figure 9 Figure 10
Hardware setup in the ARVis-Lab Tracing the line of a crack in the building
fabric
Figure 11 Figure 12
On-site colour design using colour context Colour design variants
menu
Designing with Images
The "Designing with Images" software prototype allows one to capture optical impressions and to work with them in the design process (Schneider et al., 2007). Typical approaches to designing with images in this way are most well-known from image editing programs for the creation of photomontages. These are currently limited to two dimensions, but by extending the same principle into the third dimension, collages can be experienced spatially. Planning with images and photos allows the architect to compose photorealistic spaces in the early phases of the design process and is especially useful for interior design, where photorealism is particularly desirable (see fig. 16).
Figure 13 Figure 14
Changing the size and position of the window Checking the new width of the window
Figure 15
Check the door and window design
Figure 16
Experience of the spatial impression
The images can be inserted into the scene via drag and drop and adapt their position to the surfaces of the model. In a similar manner to traditional image editing programs it is possible to resize and rotate these images (see fig. 13). Where images are laid on top of other images they stack making it possible to design in layers. Furthermore it is possible to replace images, which makes it easy to rapidly compare different variants.
Conclusion and outlook
In this paper we have presented an augmented-reality based user interaction method that uses laser-pointer-tracking and is suitable for use on site. In addition to an "InteractionKernel" for use in the distributed software framework, we have examined the use of pie-menus in different examples as a means of controlling and selecting functions. The integration of speech control for adding text is also envisaged. Similarly, an IR-marker tracking method is planned so that the position of the viewer can be determined for a 3D stereo projection.
A further aim of the research project is the development and integration of a motorised Pan-Tilt-Projector into the software framework. Through the use of a mobile projector, it should be possible to partially augment an entire room. The projection is then continually adapted to the rotation angle of the projector. A camera attached to the Pan-Tilt-Projector can follow the position of the laser dot of the input device so that through the shifting projection surfaces, the entire design of the room can be experienced.
*
filf. 1 # y l*
Figure 17
Discussing a design
Figure 18
The actual location in the ARVis-Lab
Augmented reality technology opens up great potential for supporting the architectural design process. The ability to make and present design decisions at true scale on site "for real" adds an entirely new dimension to architectural design (see figs. 17-18). The long-term aim of the research project is to take the practice of architectural design to a new level.
References
Bimber, O. and Raskar, R.: 2005, Spatial Augmented Reality - Merging Real and Virtual
Worlds, A K Peters Ltd., Wellesley / USA. Callahan, J., Hopkins, D., Weiser, M. and Shneiderman, B.: 1988, An empirical comparison of pie vs. linear menus, in: J. J. O'Hare, Ed., Proceedings of the SIGcHI Conference on Human Factors in Computing Systems, ACM, New York, NY, 95-100. Hommerich, C., Hommerich, N. and Riedel, F.: 2005, Zukunft der Architeken - Berufsbild und Märkte. Eine Untersuchung im Auftrag der Architektenkammer Nordrhein-Westfalen, available at: www.bakcms.de/bak/daten-dakten/architektenbefragungen/
Zukunft_der_Architekten_Endbericht.pdf: Jan 2006. Kurz, D., Häntsch, F., Grosse, M., Schiewe, A. and Bimber, O.: 2007, Laser Pointer Tracking in Projector-Augmented Architectural Environments, in: IEEE International Symposium on Mixed and Augmented Reality (ISMAR'07), Nara, Japan , pp. 19-26. Petzold, F., Thurow, T. and Donath, D.: 2001, Geometrieerfassung und -Abbildung in der Bestandsaufnahme, in Richard Romberg, Manuel Schulz (ed.), Forum Bauinformatik 2001, VDI Verlag Düsseldorf, pp. 260-267. Petzold, F. and Donath, D.: 2004, The Building as a Container of Information - the Starting Point for Project Development and Design Formulation, in Proceedings of the ICCCBE-X, Weimar, Germany, pp. 164-165. Petzold, F. and Donath, D.: 2005, Digital Building Surveying and Planning in Existing Buildings - Capturing and Structuring Survey Information, in A. Amir (ed.), Proceedings of ASCAAD 1st International Conference, Dhahran, Saudi-Arabia, pp. 73-87. Petzold, F., Tonn, C. and Schneider, S.: 2008, Scale 1:1 - Design Support Site With and Within Existing Build Context, in AMIT 2008 2(3), available at http://www.marhi.ru/AMIT/2008/2kvart08 /Frank_Petzold/article.php Schneider, S., Tonn, Ch. and Petzold, F.: 2007, Designing with images - Augmented reality supported on-site Trompe-l'œil, in: Em'boby'ing Virtual Architecture - ASCAAD, Alexandria, Egypt, pp. 275-290. Tonn, Ch.: 2005, Computergestütztes dreidimensionales Farb-, Material- und Lichtentwurfswerkzeug für die Entwurfsplanung in der Architektur, Diploma Thesis, Bauhaus-Universität Weimar. Tonn, Ch.: 2008, Colored Architecture, Software Prototype, Chair for Computer Science in Architecture, Bauhaus-Universität Weimar.
