Frolicking with Phantoms: Illusion in Mixed Reality
Alex Davies
This
paper explores the induction of specific emotional effects upon
individuals through illusion in mixed reality installation environments,
using the authorÕs work Dislocation, (2005) as a case study. The paper reports on the design and
production of Dislocation, concentrating on the development of techniques for
creating illusion. These approaches can be used to produce compelling
works that are not only technically and aesthetically sophisticated,
but also provide a sustained engaging experience for the user. It
reflects on the effectiveness of these techniques from observations
of audience interaction in gallery settings.
islocation
[i]
is a gallery-based interactive installation in
which realtime video, audio and audience locational data are employed
to create the illusion of additional characters inhabiting the installation
space.
Over the last few decades there has
been considerable activity in the fields of mixed, virtual and augmented
reality research both in visual and aural implementations. GrauÕs
examination of the field in Virtual Art indicates that there is a tendency towards immersing
the viewers within the virtual world with a suggestive impression
of Ôimmersing oneself in the image space, moving and acting there
in Ôreal timeÕ and intervening creativelyÕ
[ii]
. These systems transport the user into an artificial realm distinctly removed from reality,
or overlay artificial imagery over real physical space.
[iii]
.By contrast Dislocation brings the virtual elements seamlessly into the physical
reality of the participants, rather than transporting the audience
into the virtual. In this sense the virtual and the real become
indistinguishable. Although works such as Nigel HelyerÕs Sonic
Landscapes examine this approach
in aural form, there has been ltittle development in integrated
audio-visual systems
[iv]
. This shift of focus presents particular
demands and challenges. How can such an illusion be successfully
achieved? What is the significance of this shift in audience experience?
Can the transition of virtual characters into the real world be
used as a powerful and compelling emotional tool?
1.Illusion There
are two distinct forms of illusion that can be applied to create
a fusion of the real and virtual. The first is evident in cinema
and theatre whereby the audience, through suspension of disbelief,
can validate the presence of a fantastical creature inhabiting the
same space as a human. Although the viewer is immersed within this
fantasy world, this form of illusion takes place on a separate plane
to the audienceÕs lived reality. Many models of interactive systems
that rely on suspension of disbelief for immersion to occur, such
as Char DaviesÕ Osmose
[v]
and CAVE based VR
[vi]
also operate within screen space rather than
reality.
Dislocation does not work with such
cinematic or theatrical techniques as the phantoms not only infiltrate
the participantÕs world but do so unexpectedly. This has more in
common with the second form of illusion; that used in stage magic,
whereby the audience is involuntarily made to believe in the impossible
inhabiting physical reality. In order to bring phantoms into the
real world of the audience, the element of surprise is critical
as Lokuge states:
The notion of willing suspension of disbelief applicable to theatrical
performances is not apt in creating illusions. Since magic inherently
relies on violating peoples expectation, asking audiences to willingly
succumb to the effect of the illusion is purposeless
[vii]
(Lokuge 1995)
By violating expectations,
destabilizing moments can be created in which viewers involuntarily
exist within a hybrid world of physical reality and illusion 2.Dislocation
– Installation Structure
The exhibition space itself is empty,
apart from the small inset screens at the front of the room (portals).
Concealed from the participants, a camera is embedded in
the rear wall, and an array of eight speakers is mounted within
the walls around the floor of the room.
The empty room is composed to create a feeling
of normality, without any preconceptions of what may occur. As a
counterpoint to the visual minimalism, an eight channel audio composition
is presented in the room enveloping the audience.
In Alex DaviesÕs Dislocation, four
small mounted monitors are set back in one wall of an enclosed installation
space. You need to approach them closely to see what
they are showing. It takes a moment to realise that what you can
see is your own back, and those of your neighbours peering at
the adjacent monitors. The screen flickers slightly, as if there
is a minor disruption in transmission, and someone else enters
the gallery, nearer to the camera, talking on a mobile phone.
The sense of their presence behind you is spine-tinglingly palpable,
as is the illicit feeling that you are eavesdropping on their conversation.
But glance over your shoulder and you find
the room is empty.
3.Design Considerations In
order for the work to succeed a number of factors had to be addressed.
á
Create a seamless illusionary environment that looked and sounded real.
á
Control the audience within this environment for the illusion to succeed
and be sustained.
á
Inhabit this environment with suitable virtual characters and scenarios
that could produce pronounced emotional responses within the audience.
3.1
Perceptual Shift In
a perfect world Dislocation would fulfill SutherlandÕs ideas for the ideal computer
display.
One must look at a display screen as a window through which one beholds
a virtual world. The challenge to computer graphics is to make the
picture in the window look real, sound real, and the objects act
real.
[ix]
(Grau 2003)
Unfortunately the world is not perfect
(yet). Many endeavors in the field of mixed reality focus on the
use of 3D graphics to allow unconstrained views of objects so they
seem to Ôact realÕ. The primary goal for illusion in Dislocation was creating a tangible and realistic
hybrid world that looked real. Pre recorded video sequences
combined with spatial audio were used to achieve this. The installation was designed so that the audience engaged
with the work via an interface that was unencumbered by technology. This illusionary
freedom enabled the viewer to inhabit and respond to the virtual
world just as naturally as they inhabit real physical space. 3.2 Visual Systems
All
aspects of the environment were designed to lead the viewers into
particular situations, without conveying the feeling of being obviously
manipulated. This subtle influence was utilized to instill in the
viewer a sense of freedom within the space, reducing any sense of
abnormality in the environment with the aim of enhancing the illusionary
impact. If the audience has the impression that they are being manipulated
then they will be more guarded and wary and not as susceptible or
responsive to the illusion.
One
challenge to the successful establishment of illusion was the intersection
of phantoms and viewers within the physical space (when displayed
through the portals). In an attempt to minimize collisions between
phantoms and humans several techniques were used. Many of the phantoms
were recorded to inhabit the rear area of the room. The entrance
was positioned in a way to lead individuals towards the portals
rather than having the audience enter the space and wander around
randomly, thus reducing the chance of collision with the virtual
entities. The entrance was closely situated to the front of the
room to encourage users to move directly to the portals, the prominent
focus of attention within the minimal room.
The
portals themselves were positioned beneath eye height and set back
from the wall so that to view the screens properly, the audience
had to move quite close to the portals and lean forward. This provided
two benefits. It further controlled the location of the live audience
within the room and additionally, by making the audience peer directly
into the portals at such a close proximity, removed much of the
individualÕs peripheral vision. To this end, the portals acted as
the viewerÕs only visual reference on reality within their surrounds
thus immersing them further into the mixed reality they inhabited.
Certain
contingencies were also implemented due to the fact that one can
never fully predict or control the activities of the audience. In
the instance when viewers did stray from the bounds of the portals,
it was necessary to find a way to smooth over aberrations in the
illusion when phantoms and humans collided. Instead of the viewer
seeing a phantom walking through an audience member, a tracking
and collision detection system was designed in which the live visual
feed presented on the screens was temporarily distorted and the
moment of impact. This gave an impression of a glitch in the system
or breakdown in technology. The illusion is sustained and the audience
has the impression that they inhabiting an unstable electronic environment.
3.3 Aural Systems
Sound
is a powerfully emotive tool and particularly well suited to the
creation of illusion. Sound is a pivotal aspect in Dislocation on two levels. Firstly
it sets the overall ambience and secondly it is used as a device
to heighten the sense of the visual illusion that the viewer is
experiencing. Multi channel atmospheric sound was used to create
a low level of tension within the room and broadly shape the audiences
initial emotional state. The ambient base also facilitated blending
of the live and phantom sounds within the room, acting as an intermediary
acoustic zone that both worlds fluidly shifted between.
Sound
design techniques from conventional cinema such as off screen sound
were applied to the world of mixed reality installation. The presence
of the phantoms could be heard prior to their entry into the room,
just as one would perceive the impending entrance of a live audience
member. These initial spatial audio cues were utilized to setup
the forthcoming visual illusion that was about to unfold. Upon visible
entry into the environment, sounds generated by the phantoms were
spatialised via the eight channel speaker array so that virtual
footsteps would move around the room in conjunction with the phantoms
movements, mobile phones rang next to you as the phantom reached
for their phone and shrieks, laughs, barks and kisses all spatially
drifted around with their virtual counterparts. Although the illusion
could have succeeded without the use of spatial audio, there is
no doubt that this added further depth to the illusion, ingraining
the veracity of situation deeper within the viewers mind. 3.4 Virtual Presence
The framework of the system permitted
a nearly limitless array of possibilities for the development of
the hybrid phantom presence. Given that human beings respond strongly
to the presence and behavior of other humans in their immediate
surroundings, developing an illusion of the presence of others was
a powerful way of engendering emotional reactions in the participants.
Emotional response could have been achieved via other means such
as architectural changes to the space or even abstraction of the
environment, but the ability of humans to readily connect with other
humans (both real and virtual) appeared to be a logical starting
point for these investigations.
audience
members. For instance, if an individual was located at portal number
one, a phantom would enter and precede to address the viewer, begging
for money, whispering to them, or being aggressive and physically
threatening. These virtual entities were critical to forming the
substance of the hybrid world. They not only provided the initial
perceptual shift but also sustained audience engagement through
emotionally compelling situations and the construction of narratives. 4.Technical
Realisation - Illusion on a Shoestring.
Figure 3 – Production Still –
Phantom ChromaKey Shoot
recorded
that were tailored to various audience situations within the exhibition
space. A database and tracking system was then used to present the
most appropriate scenarios for a given audience. For instance, if
individuals were at portal number 1,3 and 4, a phantom could be
presented who, mimicking an audience member, would enter the room,
walk up and peer into portal number 2, giving the impression that
they were standing right next to you and engaging with the work.
4.1.Tracking the Real and Virtual
This
form of basic intelligence within the software engine was developed
through analysis of the video images from the camera at the rear
of the room. The camera not only provided the video stream that
was processed and presented on the portals, but also provided critical
data on the activities of individuals in the space that could be
interpreted and applied to the system. Minor flaws in the visual
system would cause the illusion succeed or fail. An essential aspect
of the development of the work was to have a clear understanding
of the activities of the live individuals in the space and the subsequent
location of the phantoms in the selected scenario. The reason for
this was twofold. It was necessary to ascertain the location of
the audience at the portals in order to tailor specific scenarios
to the immediate configuration of individuals in the space. It was
also critical in identifying if phantoms and humans collided in
the space, thus destroying the illusion. Without this level of environmental
information, only a rather superficial mixed reality could be presented,
lacking the required depth necessary to create a sustained and engaging
audience experience. 5.
Outcomes - Human Interaction in a World of Illusion
Dislocation,
much like many practical investigations in the field was primarily
developed in a speculative mode informed by prior research experience.
The public experimental phase of the research was undertaken in
presentations at Experminenta Vanishing Point, Blackbox Melbourne 2005 and Experimenta Under the Radar, FACT, Liverpool UK 2006.
During these presentations audience behavior within the installation
was recorded to video and provides the basis for the following observations.
5.1
Reflection The
first and most basic form of interaction is that of the viewer interacting
with their own reflection, their presence in the screens. Even in
the absence of any manipulation of the image, viewers found this
in itself compelling enough. This was accentuated in the case of
multiple individuals being present at a given time as will be examined
later. Merely the minor shift in perspective is enough of a transformation
to provide the audience with as Rokeby states:
a wayward loop of consciousness through which ones own image of ones
self and ones relationship to the world can be examined, questioned
and transformed
[x]
(Rokeby 1996)
Although
this perceptual shift provided within the video system was incidental,
it informs the subsequent interactions the viewer has with the phantoms.
Some users report that this rear perspective of their environment
was in itself uneasy and disconcerting, thus placing the viewer
in receptive a psychological state for the forthcoming phantom arrivals.
5.2
Human and Virtual Presence The
second layer of interaction inherent in the work is that of the
viewer and the phantom presences in the space. This takes several
forms. Initially there is a fleeting moment when seeing a phantom
inhabit the space, and hearing their movements within the room,
that illusion becomes reality.
Figure 4 – Audience Interaction,
FACT UK 2006
triple
takes whereby a phantom would enter, the viewer would establish
that they were a virtual entity within the space and continue to
view scenarios as they played out. Another audience member would
walk in to the space and the initial viewer would think they were
in fact another phantom, only to be ever further astonished to turn
around and see a physical form behind them. This blurring of the
boundaries between the real and the virtual is the point in which
viewers are particularly immersed in the work, their perception
consumed by the system. 5.3
The Unexpected Initially
I had envisaged that the more psychologically intense scenarios
such as direct violent confrontation with the viewer would have
the most significant emotional impact. It became evident that this
was often not the case. The scenarios that were surprisingly effective
were the passive interactions. These were primarily based upon phantom
activities that took place within the space, but often had not direct
link to the presence of the individual viewers within the environment.
In these instances, scenarios such as virtual gallery visitors entering
and leaving the space, phantoms walking into the space to have their
mobile phones ring, and other banal aspects of everyday life provided
some of the most pronounced effects of the audience. The sheer normalcy
of the situation and reflection of the everyday made the illusion
more tangible and potent.
An
unusual recurrent theme of audience interaction was alarmingly based
upon violence. For a small proportion of the participants, the only
form of expression or ability to connect with the virtual presence
was limited to kicking and punching the phantoms. This was particularly
popular with children who are notorious for assaulting interactive
arts. Witnessing adults however, throwing punches at non-confrontational
phantoms or kicking their legs back from the portals as the phantoms
approached was mildly disturbing. The response was not incited by
fear but seemingly out of an inability to connect in any other manner.
5.4
Multi User Interaction and Sustained Engagement A clear distinction became apparent
between single user behavior and the behavior of groups. Although
the initial illusion achieved the same impact upon individuals and
groups, subsequent activity differed considerably. Individuals appeared
much more likely to focus on the phantoms, remaining at the portals
as the various scenarios unraveled. After the initial disbelief,
the work was experienced more like a mutated form of cinema. When
two or more individuals were present in the space there was considerably
more interaction between virtual phantoms and humans alike. Audience
members, mediated through the screen-based environment became more
playful and performative. The portals provided a novel medium for
viewers to stage their own undertakings. Viewers used the unusual
visual perspective to run, jump, dance, kick, hug, and display nearly
every other form of human exchange in between, to the scrupulous
eye of the portals. Even in the absence of additional phantom
presences, Dislocation provided a framework that was unorthodox enough to
generate unexpected social interaction.
As the illusion slowly unravelled,
the strings revealed, the smoke and mirrors drifted away and the
virtual world unfurled in the audienceÕs perception, a distinct
shift in experience of the work became apparent.
I watched people dissolve in delight
over and over again, drag in unsuspecting companions and relish
the moment of being duped. Even after this moment of realisation,
the images of the ghostly others occasionally behaving in inappropriate
ways remains compelling. The small audience shivers at each new
arrival like ouija board conspirators.
[xi]
(Muller 2005)
Engagement and interaction are sustained
after the illusion has faded through the playful and performative
potential of the space. The addition of phantoms into the environment
shifts the initial interactions between the viewers present by providing
additional playmates to interweave into their own personal narratives.
Individuals leave the portals for closer interchange with the phantoms
whilst the other viewers watch on as the performance unfolds.
DislocationÕs ability to sustain audience
interest beyond the primary illusion also lies in the anticipation
of what will unfold. Viewers savor the momentary loss of control
over the world they inhabit and the charged tension of the unexpected.
6.ConclusionDislocation attempts to create a mixed reality environment in which the virtual world inhabits the viewersÕ physical reality. By utilising convincing techniques of illusion to distort the perception of individuals, compelling works can be created that encourage sustained user interaction and engagement on several levels. In light of the public presentation outcomes, the work clearly succeeded due to the pronounced and visceral reaction of audience members. Further investigation into the nature of emotional response to virtual human presence in these environments (ie, the unexpected success of the banal and everyday scenarios) would be pertinent for future progress. The phantoms not only provided individual narratives but also gave rise to unexpected social interactions between humans. The surprising ability of these virtual entities to mediate and influence the behaviour of the audience clearly shows scope for further applications that the work initially did not set out to investigate.
Though Dislocation succeeded in both looking and sounding real, there was a deficit in the ability of the virtual forms to Ôact realÕ. Due to the use of pre-composed video sequences only a superficial degree of audience interaction is possible. This can be solved by the use of 3D graphics but to the detriment of Ôlooking realÕ. Due to the current level of sophistication in computer technologies, further exploration into methods that provide a higher degree of exchange between realistic video representations and humans is necessary and would extend the potential and depth of mixed reality installations.
[i]
Dislocation
was produced with the assistance of Experimenta through the New
Visions Commissions.
[ii]
Grau, O (2003) Virtual
Art, From Illusion to Immersion, Cambridge, MIT Press P.4
[iii]
Azuma, R (1997) Survey
of Aug Reality In Presence: Teleoperators and Virtual Environments
6, 4 pp355-385. [iv] Robert, C (2004) Sonic Landscapes http://www.gmat.unsw.edu.au/snap/new/sonic_demo.htm
[v]
Wilson, S (2002) Information
Arts, Cambridge, MIT Press, p700
[vi]
Shaw. J Weibel , P (2003)
Future Cinema ZKM, MIT Press pp 492-497
[vii]
Lokuge, I (1995) Dynamic
Magical Environments, Media Arts and Sciences MIT p11
[viii]
Muller, L (2005) Evocative
Objects, Strange Selves, Realtime Issue 70 http://www.realtimearts.net/rt70/muller_vanishing.html
[ix]
Grau,
op. cit., p162.
[x]
Rokeby, D (1996) Transforming
Mirrors http://homepage.mac.com/davidrokeby/mirrorsmirrors.html [xi] Muller, op. cit. |
