International Journal of Emerging Trends & Technology in Computer Science (IJETTCS)

Web Site: www.ijettcs.org Email: editor@ijettcs.org, editorijettcs@gmail.com Volume 2, Issue 3, May June 2013 ISSN 2278-6856

Development of an Intelligent Virtual Environment for Augmenting Natural Language Processing in Virtual Reality Systems
Dr. R.S. Kamath1 and Dr. R.K. Kamat 2
Department of Computer Studies Shahu Institute of Business Education and Research Kolhapur, India Department of Electronics Shivaji University, Kolhapur, India
2 1

been done to design an interface that will bridge the gap between the NLP and VI. The objective of the research work presented here is to develop an IVE suite intended for digital scenario visualization which will serve as a natural language interface for VR systems. The significant advantage of the integration is the shortening timeframe for the construction of virtual world due to the natural advantages of the NLP. The aforesaid convergence aspect forms the core of the research work presented through this paper. The IVE suite developed in the course of this work is a natural language text driven system to visually simulate the modeled operation in 3D virtual space. This suite has been designed to receive inputs in the form of English words. NLP module of this suite extracts commands out of given text. Text parser analyzes the inputted text and generates Rule base. Further the Rule base maintains sequence of the actions to Keywords: Intelligent Virtual Environment (IVE), be performed. Subsequently, the interface directs all these Virtual Reality (VR), Natural Language Processing visualization details to the renderer. Renderer displays the (NLP), Artificial Intelligence (AI) object with additional features as per input specified by the user. The IVE suite portrayed in this paper is 1. Introduction developed in Visual C Plus (VC++). The paper is divided Throughout the progression of the modern computing, amongst several sections. After introducing the theme, researchers are striving hard to nurture the systems the relevant literature review pertaining to the VR, NLP centering on the human being. This prevalent trend has led and AI has been taken. This is followed by the description to the development of some of the most innovative and of the functioning modalities of the development. culmination of many converging technologies such as Subsequently the advantages and future scope has been Natural Language Processing (NLP), Virtual Reality (VR) described in the concluding remarks. However it is and Artificial Intelligence (AI) congregating as Intelligent worthwhile at the outset to take a stock of recent Virtual Environment (IVE). The formal notion of the development related to the technologies enabling the IVE. Intelligent Virtual Environment as per the reported literature depicts unification of Artificial Intelligence and 2. Enabling Technologies For IVE Virtual Reality [1, 6]. It offers a much more sophisticated approach to user interaction leading to cover the new 2.1 Advances in VR needs of Human Computer Interaction [2, 9]. VR in this context achieves a feeling of immersion in the simulated Virtual Reality is the integration of computer graphics world [3]. Promoting the users ability to communicate in and various input and display technologies to create the natural language in the VR paradigm has been in the illusion of immersion in a computer generated reality. It forefront for quite some time [4] and the same has served is the use of computer technology to create the effect of an interactive 3D world in which the objects have a sense of as the motivation of the present work. spatial presence [21]. It makes the user to feel effectively However, despite the above mentioned efforts in immersed in a responsive virtual world. It achieves a integration of the technologies for extending the users promising way of interacting with computer applications with their natural communication, not much has so far and simulations. VR is a method of human-computer Abstract-We present development of a software suite
embodying Intelligent Virtual Environment (IVE) that provides Natural Language Interface for directly manipulation in the Virtual Reality (VR) paradigm. The work is motivated by the increasing convergence of Artificial Intelligence (AI) and the Virtual Reality archetypes supplemented and complemented with the emergence of the Natural language Processing (NLP). IVE suite portrayed in this paper enables the user to intermingle with the VR system in order to visualize scenarios in the most natural manner. The aforesaid suite captures text written in plain English as an input and translates it to a three dimensional (3D) object. After reading and interpreting the scenario, the system creates the virtual scene and the intended pertinent animation. The capability of natural language interface for VR systems is the most significant achievement of the present work which brings novel perspective in the field IVE.

Volume 2, Issue 3 May June 2013

Page 198

International Journal of Emerging Trends & Technology in Computer Science (IJETTCS)

Web Site: www.ijettcs.org Email: editor@ijettcs.org, editorijettcs@gmail.com Volume 2, Issue 3, May June 2013 ISSN 2278-6856
interaction provides a natural, intuitive way of interacting with computer applications. Some current applications of VR include simulation of surgery for instructional purposes, a modeling system to manipulate molecules easily in 3D space, a VR-controlled robot arm, a building designed in VR before it was built in physical reality, and various games [23]. Researchers are exploring and developing other uses of VR in such areas as medical imaging, data management, modeling, scientific hypothesis testing, and prototype design. It is changing the way people work with computers to an intuitive, more direct method. 2.2 NLP integrating with AI NLP is a subfield of Artificial Intelligence and linguistic devoted to make computers understand statements written in human languages [7]. Here we have taken English as a natural language. It encompasses natural language understanding and natural language generation. It is a form of human-to-computer interaction where the elements of human language, spoken or written, are formalized so that a computer can perform value-adding tasks based on that interaction [11, 12]. NLP is an area of research and application that explores how computers can be used to understand and manipulate natural language text or speech to do useful things [5]. At the core of any NLP task there is the important issue of natural language understanding. Developing a program that understands natural language is a difficult problem. Numbers of natural languages are large, they contain infinitely many sentences. Also there is much ambiguity in natural language. This makes creation of programs that understands a natural language, a challenging task. 2.3 Intent of present Research - Amalgamation of NLP and VR At first glance, the kind of interactivity found in NLP seems different from that of VR. NLP focuses on cognitive interactions involving language. NLP's components are phonemes, lexicons, syntactic and semantic systems, and knowledge representation [13, 14]. Interaction in VR, on the other hand, people walk through a virtual environment, grab and move objects. The terminology associated with VR involves actions and physical materials: objects, textures, events and behaviors. Indeed, we can have a better understanding of interactivity in VR if we view it in light of the work being done in NLP motivated for this amalgamation [8, 10]. We have expanded the benefits of NLP and VR by the construction of IVE suite in this research [9]. The main endeavor of our research is to develop an NLP interface for VR systems. With the use of NLP, computers understand natural languages giving users a more convenient way to communicate. This tool parses the text inputted by the user and obtains the desired display. By combining scenario parsing and Virtual Reality modeling Volume 2, Issue 3 May June 2013 a brand new area of research and development unfolds. IVE suite tries to use these new technologies and unite them under a common application, aiming the construction of Virtual & Interactive fields.

3. Insight Of IVE Suite Design Aspects

Present section describes the design aspects of the software suite. This interface accepts statements in natural language and sends details for processing and displays desired output. The block diagram shown in figure 1 explains the conceptual model of IVE suite developed in this research. The suite mainly consists of three segments, 1. Natural Language Processor This phase devoted for the processing of inputs furnished in natural language. Where everything is stored in a structure close to natural language, trying to simulate the way relevant information is stored in our minds. 2. NLP to VR Interface This segment takes care of the actual process of forwarding the Knowledge retrieved from the inputted scenario, to the creation of final Virtual Animation. 3. Virtual World The final stage of IVE, by which computer displays model on the screen with relevant attributes. It provides fully functional 3D system thereby enabling the user enter into the world of Virtual Reality. It has a direct interface to VR through stereo glasses [16]. A Red-Green eyewear is used as an external interface to perceive the 3D effect in passive stereo mode

Figure 1 Conceptual Model of IVE Suite

4. System Architecture
The purpose of the current research is to demonstrate and explore some of the capabilities of a NLP interface to a VR system. The architecture of IVE suite developed in this research is shown in figure 2. The required scenario is inputted by the user in natural language. Text parser analyzes the inputted text and generates Rule base. Rule base maintains sequence of the actions to be performed. Library is the repository CAD model requisite to generate virtual world. Interface directs all these visualization details to the renderer. Renderer displays the object with additional features as per input specified by the user. A 3D display is a prime part of this system that accounts for virtual reality. This section of the paper explains the detailed discussion of methodology carried out in the course of this research and execution of this IVE suite with example. Page 199

International Journal of Emerging Trends & Technology in Computer Science (IJETTCS)

Web Site: www.ijettcs.org Email: editor@ijettcs.org, editorijettcs@gmail.com Volume 2, Issue 3, May June 2013 ISSN 2278-6856

Figure 2 IVE Suite Architecture 4.1 User Input This suite is able to receive inputs in the form of natural English. User inputs the details of virtual scene in natural language. Thus IVE understands natural language giving users a more convenient way to communicate. Lets consider an example scenario, Display Sphere, Apply Material Bronze, Display-mode Wireframe, Isometric View, Apply Texture, 3D Vision A simple user-interface provides an easy way to input the scenario with parameters is shown in the figure 3.

Table 1: Attribute Value pairs for display of object Attribute Values Material Gold, Silver, Chrome, Emerald, Perl, Copper, Brass, Bronze Background Color, Image Outlook Wireframe, Vertices, Line View Front, Back, Top, Bottom, Left, Right, Isometric Tools Texture, Fog, Walkthrough VR 3D Vision Table 2: Information extracted by Parser Attribute Values Object Sphere Material Bronze Outlook Wireframe View Isometric Tools Texture VR 3D Vision 4.3 Rule Base Consists set of rules denotes the actions the renderer can take while working with the scenario. These actions when interpreted results to the production of virtual scenes by invoke of functions call. Rules are generated as per the sequence inputted by the user. It is referred to generate sequence of actions. Part of rule base generated in this example is, If (Material == Bronze), applyBronze() If (Outlook == Wireframe), applyWireframe()

Figure 3 Snapshot of user input form 4.2 Text Parser Parser is a program accepts as input a sequence of words in a natural language and breaks them up into parts to be managed by other programming. It analyzes input and extracting relevant information. It is able to translate the natural language statements into appropriate actions for the system. While parsing inputted text; parser gathers object and its necessary features and is passed to rule base for further processing. Building the text parser involved the creation of an application-specific dictionary and lexical semantics. It translates inputted text into messages for the application software to produce the appropriate actions. Parser maintains a special type of knowledge base, which represents and stores the knowledge. Table 1 shows sample of knowledge base designed in this research for the application of additional properties to the already generated scene. Table 2 shows extracted information by the parser for the example considered in section 4.1. Volume 2, Issue 3 May June 2013

If (View == Isometric), applyIsometric() If (Tool == Texture), applyTexture() 4.4 IVE Library It is a container for set of CAD models maintained in the form of ASCII formats. Initially model is designed under CAD system. Modeling software can export these designs into ASCII format. A ready-made set of models taken from CAD software such as I-Deas, Catia in order to provide the display backdrop. Virtual Reality Modeling Language (VRML) file format is considered in this research. VRML is a scene description language [17]. It encloses all the visualization details of the model. This text file defines description of the object in terms of triangulated data. The datasets are organized from point by point followed by polygons vertices references. All the details of the object such as vertices and edges of the polygon; surface color, material, lighting, shininess etc are represented in triangulated fashion [26]. Figure 4 shows part of objects data in VRML format. Library maintained with a wide variety of VRML files helpful for the display Page 200

International Journal of Emerging Trends & Technology in Computer Science (IJETTCS)

Web Site: www.ijettcs.org Email: editor@ijettcs.org, editorijettcs@gmail.com Volume 2, Issue 3, May June 2013 ISSN 2278-6856
of virtual world. It maintains dictionary of known words and expressions on which interface is based. 4.5 NLP to VR Interface Interface forwards the Knowledge retrieved from the inputted scenario, to the creation of final Virtual Animation. It fetches required object from the library and directs all the details extracted during parsing to renderer for the display. Permit walkthrough of the model to examine the inner details of the object Enable to view more than one model through either tiling or cascading the windows. As per the example mentioned in section 6.1, the renderer displays the model shown in figure 5, 6 and 7.

Figure 5 Display of Sphere

Figure 4 Part of VRML file 4.6 Renderer Display of Model Rendering, the process by which computer displays the model on the screen. Renderer fetches the selected model from the library and renders the object. This module reads the triangulated data contained in VRML file and performs display of it [19]. As per the specification of rule base the sequence of actions are applied to the rendered model. We have used OpenGL tool for displaying the dataset on the screen [24, 25]. Data sets of the model are passed to OpenGL for rendering. OpenGL is a software interface to graphics hardware. It constructs 3D rasterization information from geometric primitivespoints, lines & polygons. It has set of commands to specify the objects and operations needed to produce interactive three-dimensional applications. Along with display of model, renderer supported with additional features such as: Provide transformation options like Rotation, Panning and Zooming Provision of editor for selection of different material properties Options for rendering the model in various modes i.e. solid, wireframe and point Give different views of the model i.e. front, back, left, right, top, bottom and isometric Provide texture mapping, Volume 2, Issue 3 May June 2013

Figure 6 Bronze Material is applied

Figure 7 Wireframe view of the Sphere 4.7 From Natural Language to Virtual Environment We are in the process of exploring NLP interface for VR systems. Presence is closely related to the sensation of immersion. It can be described as the feeling of being in the same space as the Virtual Environment, which gives a sense of the reality of objects in the computer-generated scene and the users presence with those objects. The parameter depth is provided along with height and width for the display of model to perceive 3D view. The technique incorporated for the 3D viewing is generating two views of the scene, one from each eye position [18]. The position of the eyes is specified in terms of eye separation parameter. Having separate Page 201

International Journal of Emerging Trends & Technology in Computer Science (IJETTCS)

Web Site: www.ijettcs.org Email: editor@ijettcs.org, editorijettcs@gmail.com Volume 2, Issue 3, May June 2013 ISSN 2278-6856
drawing buffers for the left and right eyes does 3D display. Both the images are drawn on the screen. The third dimension is visualized by wearing stereo enabled glasses. We have used Red-Green anaglyph to view the 3D scene [20, 22]. Left and right eye images are combined into a single image consisting of greens for the left eye portion of the scene, reds for the right eye portion of the scene, and shades of yellow for portions of the scene occupied by both images The viewer wears a pair of glasses with red over the left eye and green over the right eye. Left eye sees one image and the right eye sees another image. Combination of this in brain gives the 3D effect. This enables the rotation of model to view the model from any angle. Figure 8 is a snapshot of the model in this IVE suite. number of additional capabilities that would increase the power and usability of this interface considered as scope for future work: We have considered only syntactic analysis while designing Parser. Capability of NLP can be extended by incorporating semantic analysis and pragmatic analysis Scope of Learning for extending Knowledge Base While developing NLP interfaces for VR systems it is required to understand how people conceive of, talk about, and use 3D space. Development of improved discourse tracking capabilities and appropriate error detection and correction strategies are considered for future work. A natural language generation module is incorporated to handle user feedbacks. VR is achieved in our research by facilitating 3D vision. Research can be extended by providing interface to other VR devices.

REFERENCES
[1] L.A. Dobrzaski, R. Honysz, Artificial intelligence and virtual environment application for materials design methodology, International Scientific Journal, Vol. 45 Issue 2, 2010 [2] Jean-Luc Lugrin, Marc Cavazza, Sean Crooks and Mark Palmer, Artificial Intelligence- Mediated Interaction in Virtual Reality Art, IEEE Inteligent Systems, 2006 [3] Samir Sadek Hosny, Sherif Morad Abdel Kader, Integrating Intelligent Mixed Reality in Architectural Education, Al Azhar University Engineering Journal, Vol. 7 No. 3 2004 [4] Tamer M Wasfy, Ahmed K Noor, Rule Based Natural Language Interface for Virtual Environments, ELSEVIER Advances in Engineering Software 33 (2002) 155-168 [5] Prakash M Nadkarni, Lucila Ohno-Machado, Wendy W Chapman, Natural language processing: an introduction, J Am Med Inform Assoc 2011 [6] Ruth Aylett Michael Luck, Applying Artificial Intelligence to Virtual Reality: Intelligent Virtual Environments [7] Stephanie S. Everett, Kenneth Wauchope, Manuel A. Perez, A Natural Language Interface for Virtual Reality System [8] Curry I. Guinn and R. Jorge Montoya, Natural Language Processing in Virtual Reality Training Environment [9] Stephen N. Matsuba, Speaking Spaces: Virtual Reality, Artificial Intelligence and the Construction of Meaning [10]Milos Cernak1, Adrian Sannier, Command Speech Interface to Virtual Reality Applications, Technical Report, Iowa State University Page 202

Figure 8 Spheres in Virtual Environment

5. Significance And Conclusion

Natural Language Interface and generation of virtual environment are the characteristic features of this solution which makes this unique. Here user can submit request in natural spoken English. Then system generates sequence of actions in the virtual environment. This IVE suite can be used in various fields such as manufacturing industries, architectural designs, education, training etc., because of its distinctive feature. The major objective of this research was the development of IVE. This research verifies the feasibility of integrating NLP and VR leading to a range of technological advancements. NLP interface shortens the timeframe for the construction of virtual world. VR is of great significance in providing better understanding and visualization capabilities as an effective tool. The capability of natural language interface for VR systems makes this research to bring new perspective in the field IVE. User can express queries naturally by typing for which computer system generates corresponding virtual world. Renderer represents the actual environment where the story described in the scenario is taking place.

6. Scope For Future Work

Almost every module and functionality we have described is currently implemented in this IVE suite. There are a Volume 2, Issue 3 May June 2013

International Journal of Emerging Trends & Technology in Computer Science (IJETTCS)

Web Site: www.ijettcs.org Email: editor@ijettcs.org, editorijettcs@gmail.com Volume 2, Issue 3, May June 2013 ISSN 2278-6856
[11]Stephanie S. Everett, Kenneth Wauchope, Manuel A Perez, A Spoken Language Interface to a Virtual Reality system [12]Scott McGlashan, Tomas Axling, A Speech Interface to Virtual Environments [13]K.R. Chowdhary, Natural Language Processing [14]Gobinda G. Chowdhury, Natural Language Processing [15]Hilary McLellan, Magical Stories: Blending Virtual Reality and Artificial Intelligence [16]Alpaslan, Z.Y., & Sawchuk, A.A. (2004). ThreeDimensional Interaction with Autostereoscopic Displays. Proc. Stereoscopic Displays and Virtual Reality Systems X I Symposium, Proc. SPIE, Vol. 5291, San Jose. [17]Carey, R. & Bell, G., The annotated VRML 97 ReferenceManual http://accad.osu.edu/~pgerstma/class/vnv/resources/i nfo/AnnotatedVrmlRef/appd.htm. [18]Kamath, R.S. & Kamat, R.K. (2010). An Effective Stereo Visualization System Implementation for Virtual Prototyping. International Journal on Computer Science and Engineering, 2(7), 2316-2321. [19]Kamath, R.S. & Kamat, R.K. (2012). Development of Virtual Reality Interface for Architectural Visualization. International Journal of Computer Science and Knowledge Engineering, 6(1), Page 2731. [20]Seth, A., Smith, S.S. Shelley & M. Jiang, Q. (2005) A Low Cost Virtual Reality Human Computer Interface for CAD Model Manipulation. Spring. [21]Steinicke, F., Bruder, G. Hinrichs, K., Jerald, J. Frenzy, H. & Lappey, M. (2009). Real Walking through Virtual Environments by Redirection Techniques. Journal of Virtual Reality and Broadcasting, 6(2). [22]Wormell, D., Foxlin, E. & Katzman, P. (2007). Improved 3D Interactive Devices for Passive and Active Stereo Virtual Environments. 13th. Eurographics Workshop on Virtual Environments. [23]Zelle, Z. M. & Figura, C. (2004). Simple, Low Cost Stereo Graphics : VR for Everyone. SIGCSE04, March 37, Virginia, USA. [24]R.S. Wright, L. Benjamin and H.Nicholus, OpenGL Super Bible Comprehensive Tutorial and Reference 4th Edition, Addision-Wesley [25]Mason Woo, Jackie Neiderr, Tom Davis. OpenGL programming guide The official guide to learning OpenGL [26]http://www.techiwarehouse.com/cms/engine.php?pag e_id=bb0d3668 - VRML Tutorial, Techi Warehouse Her Doctorate research was in Computer based Visualization. Dr. Kamath has published ten research papers in national and international journals. She has vast experience of around nine years in teaching and research. R.K.Kamat holds M.Sc, M.Phil from Shivaji University Kolhapur and Ph.D from Goa University. He is a Professor in Shivaji University. He has immense skill of around 20 years in teaching and research. He is a recipient of various honors. He has supervised many M.Tech and Ph.D theses. Dr. Kamat has published more than 50 papers in national and international journals, presented papers in conferences. He has served on numerous conferences program committees.

AUTHORS
R.S.Kamath is an Assistant Professor at Chh. Shahu Institute of Business Education and Research, Kolhapur. She has obtained her M.Sc from Mangalore University and Ph.D from Shivaji University in Computer Science. Volume 2, Issue 3 May June 2013 Page 203