Development and Production

Overview
Dislocation was developed throughout 2005. The work was constructed in several distinct stages. Some initial tests with software were undertaken to see if the concept was even remotely feasible. Concept development then took place regarding the nature of the phantoms that would inhabit the gallery space, what emotional responses did I want to induce in the viewer , and what action would take place to achieve this.

Multiple shots were taken in a chromakey studio for each particular scenario, so that depending on how many individuals were present in the galley, a suitable variation was delivered. The backgrounds of the 100 or so pieces of footage were subsequently removed so they could be composited over the live video feed in the exhibition space. An 8 channel sound mix was constructed for these phantom video's so that the sound would spatially follow the phantom as they moved around the exhibition space. Software was developed in Max /MSP, Jitter and Soft VNS to bring all the live video elements, phantom composites and spatial sound together in a manner that was responsive to the audiences activities in the room. This is outlined in further detail below (taken from this paper)....

-Design Considerations

In order for the work to succeed a number of factors had to be addressed.
• Create a seamless illusionary environment that looked and sounded real.
• Control the audience within this environment for the illusion to succeed and be sustained.
• Inhabit this environment with suitable virtual characters and scenarios that could produce pronounced emotional responses within the audience.

Perceptual Shift
In a perfect world Dislocation would fulfill Sutherland's ideas for the ideal computer display.

" One must look at a display screen as a window through which one beholds a virtual world. The challenge to computer graphics is to make the picture in the window look real, sound real, and the objects act real."

Unfortunately the world is not perfect (yet). Many endeavors in the field of mixed reality focus on the use of 3D graphics to allow unconstrained views of objects so they seem to 'act real'. The primary goal for illusion in Dislocation was creating a tangible and realistic hybrid world that looked real. Pre recorded video sequences combined with spatial audio were used to achieve this.
The installation was designed so that the audience engaged with the work via an interface that was unencumbered by technology. This illusionary freedom enabled the viewer to inhabit and respond to the virtual world just as naturally as they inhabit real physical space.

Visual Systems
All aspects of the environment were designed to lead the viewers into particular situations, without conveying the feeling of being obviously manipulated. This subtle influence was utilized to instill in the viewer a sense of freedom within the space, reducing any sense of abnormality in the environment with the aim of enhancing the illusionary impact. If the audience has the impression that they are being manipulated then they will be more guarded and wary and not as susceptible or responsive to the illusion.

One challenge to the successful establishment of illusion was the intersection of phantoms and viewers within the physical space (when displayed through the portals). In an attempt to minimize collisions between phantoms and humans several techniques were used. Many of the phantoms were recorded to inhabit the rear area of the room.

The entrance was positioned in a way to lead individuals towards the portals rather than having the audience enter the space and wander around randomly, thus reducing the chance of collision with the virtual entities. The entrance was closely situated to the front of the room to encourage users to move directly to the portals, the prominent focus of attention within the minimal room. The portals themselves were positioned beneath eye height and set back from the wall so that to view the screens properly, the audience

Audience enterance on the right side of room within a couple of meters of the portals/screens

had to move quiteclose to the portals and lean forward. This provided two benefits.

It further controlled the location of the live audience within the room and additionally, by making the audience peer directly into the portals at such a close proximity, removed much of the individual's peripheral vision. To this end, the portals acted as the viewer's only visual reference on reality within their surrounds thus immersing them further into the mixed reality they inhabited. Certain contingencies were also implemented due to the fact that one can never fully predict or control the activities of the audience. In the instance when viewers did stray from the bounds of the portals, it was necessary to find a way to smooth over aberrations in the illusion when phantoms and humans collided. Instead of the viewer seeing a phantom walking through an audience member, a tracking and collision detection system was designed in which the live visual feed presented on the screens was temporarily distorted and the moment of impact. This gave an impression of a glitch in the system or breakdown in technology. The illusion is sustained and the audience has the impression that they inhabiting an unstable electronic environment.

Examples of the collision detection system and realitme distortion of the composited image.

Aural Systems
Sound is a powerfully emotive tool and particularly well suited to the creation of illusion. Sound is a pivotal aspect in Dislocation on two levels. Firstly it sets the overall ambience and secondly it is used as a device to heighten the sense of the visual illusion that the viewer is experiencing. Multi channel atmospheric sound was used to create a low level of tension within the room and broadly shape the audiences initial emotional state. The ambient base also facilitated blending of the live and phantom sounds within the room, acting as an intermediary acoustic zone that both worlds fluidly shifted between.

Sound design techniques from conventional cinema such as off screen sound were applied to the world of mixed reality installation. The presence of the phantoms could be heard prior to their entry into the room, just as one would perceive the impending entrance of a live audience member. These initial spatial audio cues were utilized to setup the forthcoming visual illusion that was about to unfold. Upon visible entry into the environment, sounds generated by the phantoms were spatialised

Development image indication the placement of the speakers in the work...

via the eight channel speaker array so that virtual footsteps would move around the room in conjunction with the phantoms movements, mobile phones rang next to you as the phantom reached for their phone and shrieks, laughs, barks and kisses all spatially drifted around with their virtual counterparts. Although the illusion could have succeeded without the use of spatial audio, there is no doubt that this added further depth to the illusion, ingraining the veracity of situation deeper within the viewers mind.

Virtual Presence
The framework of the system permitted a nearly limitless array of possibilities for the development of the hybrid phantom presence. Given that human beings respond strongly to the presence and behavior of other humans in their immediate surroundings, developing an illusion of the presence of others was a powerful way of engendering emotional reactions in the participants. Emotional response could have been achieved via other means such as architectural changes to the space or even abstraction of the environment, but the ability of humans to readily connect with other humans (both real and virtual) appeared to be a logical starting point for these investigations.

A number of video sequences of virtual characters appearing in the space were developed to explore the potential resonance between the live subject and the virtual characters. Beyond the successful implementation of illusion, the choice of scenarios was the next most critical decision. Several broad categories of emotion were
considered as starting points for the scenario development. These encompassed curiosity, discomfort, happiness, confusion, and fear and were manifested through over 100 pre recorded video sequences.

3 examples of live human/phantom image composite via portals. Click for large size....

Two subsets of phantom interactions were developed with an aim of evoking these responses in the audience. These took the form of passive and active interactions. Passive activities included phantoms simply inhabiting the same environment as the viewer, a virtual gallery visitor passing through or a transient conversation between two phantoms. Active interactions were based around direct connections with specific
audience members. For instance, if an individual was located at portal number one, a phantom would enter and precede to address the viewer, begging for money, whispering to them, or being aggressive and physically threatening. These virtual entities were critical to forming the substance of the hybrid world. They not only provided the initial perceptual shift but also sustained audience engagement through emotionally compelling situations and the construction of narratives.

4.Technical Realisation - Illusion on a Shoestring.
Dislocation was developed with consumer grade technologies and programmed within Max/MSP. In order for the illusion to succeed the phantom overlay had to mimic the quality of the CCTV video feed being transmitted to the portals that served the function of the viewer's reality. The compositing solution was achieved by the use of Chroma Keying.

Actors were videoed within a green cyclorama and digitally separated from their background, ready to be composited within the live exhibition space. As the aim of the work was to create the most realistic and compelling illusion possible, approximately 130 scenarios and permutations were recorded that were tailored to various audience situations within the exhibition space. A database and tracking system was then used to present the most appropriate scenarios for a given audience. For instance, if individuals were at portal number 1,3 and 4, a phantom could be presented who, mimicking an audience member, would enter the room, walk up and peer into portal number 2, giving the impression that they were standing right next to you and engaging with the work.

The above images (click on image for video) demonstrate the three different approaches to phantom/human interaction. Scenario 1 on the left is an example of passive interaction in which the phantom inhabits the same space as the audience, however does not have direct contact with them. In this instance individuals could be present at portals 1-4. The example in the middle demonstrates an active scenario. In this instance the phantom is directly interacting with a audience member who is present at portal number 2 (This particuar scenario was shot 4 different times, one for each portal, so that it could be delived under a variety of circumstances). This form of interaction ranges from simply inhabiting the personal space of the individual to verbal action. The third example present on the right demonstrated the use of a phantom acting as an audience member (who approaches and looks into portal # 1) and is subsequently approached by another phantom who interacts with them. In this instance audience members may be situated at any of the portals ranging from 2-4.

Tracking the Real and Virtual
This form of basic intelligence within the software engine was developed through analysis of the video images from the camera at the rear of the room. The camera not only provided the video stream that was processed and presented on the portals, but also provided critical data on the activities of individuals in the space that could be interpreted and applied to the system. Minor flaws in the visual system would cause the illusion succeed or fail. An essential aspect of the development of the work was to have a clear understanding of the activities of the live individuals in the space and the subsequent location of the phantoms in the selected scenario. The reason for this was twofold. It was necessary to ascertain the location of the audience at the portals in order to tailor specific scenarios to the immediate configuration of individuals in the space. It was also critical in identifying if phantoms and humans collided in the space, thus destroying the illusion. Without this level of environmental information, only a rather superficial mixed reality could be presented, lacking the required depth necessary to create a sustained and engaging audience experience.