The Mediaturgy of Videomapping

AUGMENTED REALITY OR AUGMENTED SPACE?

Anna Maria Monteverdi

INTRODUCTION

Augmented Reality means overlaying virtual, digital elements on reality. It refers to the innovative multimedia techniques which make reality interact with digital constructions and reconstructions, thus modifying, enhancing and enriching the perceived world. It includes interactive projections, lighting techniques, virtual architecture and communication and audio-visual instruments. Its applications are numerous, ranging from overlapping virtual components onto reality (geolocalisation and referencing, logical and physical mapping) to simulation of virtual reality elements (simulating time-space interactions) and artistic applications through performances, video projections and installations. The best definition of A.R. is by Azuma (1997): “Augmented Reality enhances a user’s perception and interaction with the real world” (p.355).

Virtual reality offers great opportunities for the world of museums, galleries and archives. VR is an important tool to interact with 3D models and a fundamental help in making culture more accessible to the wide public and in documenting, recording and conservation of the Cultural Heritage. Oculus¹ Rift DK2 and the Leap Motion controller allow new experiences in cultural tour inside museums, archaeological sites.

The user puts the oculars on his head and when he looks along, he can see the world in 3D; oculus device uses custom display and optics technology designed for VR, featuring two AMOLED displays with low-persistence. The technology enables incredible visual clarity for exploring virtual worlds and for using VR gaming. The headset is tracked by IR LED constellation tracking system for precise, low latency 360-degree orientation and position tracking. Features an integrated VR audio system designed to convince your ears that you are truly there. 

Also videomapping can offer new modality of augmented perception and a new experience of seeing cultural heritage. Enormous projections with images, lights and writings by LED are parts of the urban landscape and they constitute the basic arsenal of advertising.

The mediatization of the public space

Lev Manovich states that the radical change that regards culture in the era of information society also concerns space and its systems of representation and organization, becoming itself a media: just like other media – audio, video, image and text- nowadays the space can be transmitted, stored and retrieved instantly. It can be compressed, reformatted, converted into a stream, filtered, computerized, planned and managed interactively (Manovich311). The urban screens follow the trend of a mediatization of the public spaces, bringing in open and collective places, an element traditionally used in enclosed spaces in its various forms and genres (cinema, television and computer). On one hand we consume virtual spaces through electronic devices (cell phones, GPS, huge screens) that change the perception of space and time and social relations, on the other hand we walk environments dense of screens, which create a new urban landscape.

Paul Virilio says about it:

Today we are in a world and reality split in two; on the one hand there is the contemporary, real space (i.e. the space in which we are now, inside the perspective of the Renaissance); on the other there is the virtual space, which is what it happens within the interface of the computer or the screen. Both dimensions coexist, just like the bass and treble of the stereophonic or stereoscopic (Virilio 10-11).

To characterize the urban scene in recent years is the phenomenon of “giantism”. They are called “hypersurfaces”, “interactive media facades” those walls or architectures, either permanent or temporary, designed to accommodate bright and colourful surfaces and big video projections and screens. Enormous projections with images, lights and writings by LED are parts of the urban landscape and they constitute the basic arsenal of advertising. Urban screens, architectural mapping, facade projections, 3D projection mapping, display surfaces, architectural Vj sets are some of the definitions used for describing this technique and the context is that of so called “Augmented Reality” but according to Lev Manovich it is more correct to speak about “Augmented Space” because there is a superposition of electronic elements in a physical space: a relational space in which persons could have a new type of dialogue.1

After ten years, this technique born for a specific commercial and advertising purpose as a digital signage or for the launch of a new brand, has become flexible for create live and interactive performances, live painting, laser painting, videoart using the big dimension of the architectural surface. In videomapping we could unite together graphic art, animation, light design, performances, music and also interactive art and public art. The borders of theatre change and become widen: the environment is no longer the background, is the focus of the artwork. Based on these advanced experiences, artists have created augmented reality video works, stage designs provided with video mapping of great realism and huge projections: videomapping is a new media art, a media-performative art.

Paul Virilio talks of “Gothic Electronic” in reference to this aspect of contemporary architecture linked to the role of the images:

I think that the revolution of the Gothic age, which has been so important in Europe in every field, also corresponds to the new role from the image as a building material. . . . The most interesting images are those of the windows, as in Gothic art they corresponded to information that is more intangible, that transmitted by light in extraordinarily refined creations such as large rosettes (Virilio 6).

The aesthetics of the beautiful, of the astonished, the one that Andrew Darley in his book Digital Culture defines “aesthetic of the surface”, it is behind these spectacular artistic forms related to videomapping in the work of Urban Screen, NuFormer, Macula, Apparati Effimeri, Visualia, AntiVJ, Obscuradigital.

We are facing a new machine vision in a theatrical sense: the videomapping projections are based on the same principle as the “ineffabile vision” of the XVI century,those paintings created on the basis of anamorphosis forcing to the extreme the linear perspective of the Renaissance. In the work based on an anamorphic way technique, reality can only be perceived through a distorted mirror, while the videomapping is nothing else than a mask that deforms a reality that doesn’t exist at all. Art history has given us not only the linear perspective but also other view, the so called “broken perspective”, the concatenation of plans, the multiple points of view that pose the problem of depth in painting, the imaginary level of the third dimension. The suggestion, the fictional construction of the space, the union of the backclothes and the first floor and the resulting artificial illusion are the basis of monumental frescos by Vasari, Tiepolo, Veronese and above all, of Michelangelo’s Sistine Chapel that joins architecture and paintings..

Italian painters of the late XV century such as Andrea Mantegna (1431–1506) began painting illusionistic ceiling paintings, generally in fresco: he employed perspective in painting and techniques such as foreshortening in order to give the impression of greater space to the viewer below.

This type of trompe l’œil as specifically applied to ceiling paintings is known as “di sotto in sù” (“from below, upward”). The elements above the viewer are rendered as if viewed from true vanishing point perspective. Well-known examples are the Camera degli Sposi in Mantua.

We said videomapping is an art of illusion, an art of perception: so we can make a confrontation between two different kinds of OCULO2: Andrea Mantegna studied a special fresco decoration that would invest all the walls and the vaulted ceilings, adapting to the architectural limits of the environment , but at the same time breaking through the walls with illusionistic painting, as if space was expanded far beyond the physical limits of the room; nowadays the well known technological group Urban screen with the videomapping entitled “320 degree” that uses the circular surface of the Gasometer of Oberhause has tried to obtain a similar effect as a visual game of shapes and lights in which the observer experiences the sens of void and of the deph.

As Thomas Maldonado remember us, Western civilization basically became a producer and consumer of trompe-l’oeil:

Our has been called a culture of images . . . This definition would be more true, if we add that it is a civilization in which a particular type of images, images made by trompe-l’oeil, reach, thanks to the contribution of new technologies, a prodigious yield realistic . . . Confirmation is stronger, today, thanks to the computer graphics, especially when you consider its latest developments in the field of the production of virtual reality (Maldonado 48).

Following the history of the theatre and searching similar attempts in that ancient art to give the illusion of three dimensional space, we can mention the techniques of painted background in perspective: we can quote the drawings of Baldassare Peruzzi for “Calandria” (1514), the painted scene-prototype by Serlio (the comic, tragic and satirical scene, 1545), the theorical books: Perspectivae books by Guidubaldo (1600), Andrea Pozzo (1693) and Ferdinando Galli Bibbiena (1711), and above all, La pratica di fabrica machine ne’ teatri byNicola Sabbatini (1638).

For the contemporary theatre, we can quote two examples of scenographes created with a simple technique of illusion of three dimensional space and of perfect integration of body and stage or dispositif:

– The “digital landscape” used by Robert Lepage and created by stage designer Robert Fillion for Andersen Project (2005), a concave device that includes images in video projections which, thanks to an elevation of the central structure, seemed to have concrete and lively form and succeds in interacting with the actor, literally immersed in this strange cube built around him;

– The dispositive used by Italian group Motus for L’Ospite inspired by Pasolini’s Teorema: it is a monumental scenography looming and crushes the characters, made up of a deep platform inclined, enclosed on three sides and composed of three screens for video projections. This video triptych produces the illusion of an huge house without the fourth wall.

VIDEOMAPPING PROJECTIONS IN THEATRE

The use in the theatre of the videomapping concerns not only the sets (the stage, the volumes in which images could be projected) but also objects, actors, costumes and the entire empty space.

The group Urban Screen produces a curious experiment with videomapping projection for What’s up? A virtual site specific theatre, made in Enschede (Netherlands) in 2010, where the actors were moving images projected forced into a box that picked up their intimacy, built into the walls of the building in a surreal atmosphere. Or Jump in! where the façade of the building becomes a sort of “free climbing wall” for actors who jump, climb, hide among the windows. For the opera Idomeneo, King of Crete by Mozart, Urban Screen has used an architecture of lights in mapping that adheres to the volumes of stage on which the singers acted; these structure is a simple white furniture composed by several block unite together and similar to the cliffs with the reference to the god Neptune, one of the character of the opera. Projections adhere to the structure, thanks to a software created expressely by Urban Screen called Lumentektur.

 

Apparati Effimeri an Italian technological group, has created a dramatic short cameo, in baroque style, for Orfeo ed Euridice (2014), directed by Romeo Castellucci: they used videomapping for reconstructing the scene a “grove of greenery” embedded in a scene completely empty, from where emerges an ethereal female figure, showing a sad atmosphere and a bucolic landscape worthy of a painting by Nicolas Poussin. We are in Arcadia, the scene unfolds and the video projectes on the scene moving images and 3D effects of high accuracy, leaving the illusion of wind on the leaves, the lights on the water and the evening shadows. The “baroque technologic style” of the set generates the amazing sensation reserved for large frescoes of the past and it is perfectly in syntony with the myth and with the emotions of the drama which focuses on the image live from an hospital, of the women in coma which represents in the theatrical fiction, the “double” contemporary of Euridice.

https://www.youtube.com/watch?v=rQmaUSAC-pg

 

Robert Lepage has used videomapping in his most ambitious project: the opera direction for tetralogy of Wagner (The Ring) produced by the Metropolitan of New York (2014). Protagonist of the scene is the huge machine designed for the entire tetralogy, a true work of mechanical engineering, made up of 45 axes movable independently of each other and surging and rotate 360 ° thanks to a complex hydraulic system which allows a large number of different forms, becoming the plug of a dragon, a mountain or the horse of the Valkyries. On the surface of the individual axes are projected 3D images in videomapping showing trees, caves, the waters of the Rhine, the lights of the Walhalla.

 

IAM PROJECT: AN EUROPEAN PROJECT FOR VIDEOMAPPING

IAM (International Augmented Med) is an international cooperation project in which 14 partners from 7 mediterranean countries work together to demonstrate and promote the use of Augmented Reality, videomapping and innovative interactive multimedia techniques for the enhancement of cultural and natural heritage sites. The Partnership is led by the City of Alghero, Italy. Other participating countries are: Egypt, Jordan, Lebanon, Palestine, Spain, Tunisia. It is funded by the European Union’s ENPI CBC Med programme and runs from October 2012 to October 2015.

The winners of the grants awarded in the framework of the IAM project are one Croatian and one French candidate. These subgrants are managed by project partner Generalitat de Catalunya (Government of Catalonia region – Spain, Department of Culture).From 1 to 3 October 2014 the Spanish project partners of the I AM project are hosting the I AM festival, in Girona – Catalonia, Spain, under the title “JornadesAPP” (App Days).

Three days of round tables, discussions, demonstrations, presentations, video-mappings, mobile apps and more in various locations in the historic city of Girona.

The Spanish group Konic thtr proposed in Girona an audiovisual composition that combines symbolic elements of the building with historical elements related to the former use of the building as a hospital (now Regional Office of the Catalan Government), with visual elements related to the use of medicinal plants. Processus is the title and proposes a large scale architectural projection based on the history of the projected building. Life, death and spirituality that convert into the ADN of this XVII century building, important part of the archaeological heritage of the city of Girona.

The fundamentals of Video Mapping, according to KONIC stand on concepts that define, identify, structure and characterize the technique. The mapping has to co-live with the dramatic text, the actor’s actions, the choreography and the sound, taking part of the narrative of the piece and contribute to the tension and development of the piece through the visual evolutions. As stated by Koniclab:

We can understand it as a treated space, or as a dynamic stage set, that will bring dynamics, stories, actions along time, by its audiovisual evolution. The media co-exists with the actors and/or dancers and bring to the piece another layer in a horizontal hierarchy with the other elements composing it. We can think of a global work, in which the different disciplines and materials participate to the narration and composition of the work.The audience, from our experience, perceives the piece as a whole and is often surprised by the intimate and somehow magical dialogue established between light-image-sound-object or architecture- and the actors/ bodies, who will bring the human scale and the humane-fictional proportions.When the projection is interactive, it offers direct participation of the audience in the piece, who will then coexist with the actors and can then become a part of the fictional world, and make the step from social being to active part of the piece (Monteverdi).

In their opinion, the following elements should be taken into account when developing a stage play that incorporates this technique:

– The inter-relation image-object-volumetric support, convey the fact that the dramaturgy is focused on the image mapped onto the object. This is to say on the augmented object transformed into an object-image hybrid. The visual narrative is driven by this object-image articulation, and not only by the image. Hence, the object should ideally be related with the visual compositions projected onto it.

– The concept of Mapping = a skin made of visuals and light covering the volumetric object. A dynamic and flexible skin, which will adapt like a dress to the object onto which it is projected or visualised.

– The technology, in understanding that mapping is building a perceptual device composed of light, image, sound, software, hardware, space and time, architecture, actors and audience, and all these elements together create an experiential and relational whole.

Marko Bolkovic one of the two winners of IAM subgrants, with his group Visualia from Croatia, has presented in Girona a mapping for Casa Pastors of high visual impact made entirely in 3D entitled Transiency, inspired by the metamorphosis of life.

Marko Bolkovic said about his videomapping:

Video mapping can be made with any available material, such as video material, film. It’s necessary to adjust this material to a projected surface, which is done with special computer software. 3D video mapping is much more complicated and takes more time for its production: first of all, a model of the object where a projection will be made should be created, afterwards all the animation should be made in a special software provided just for that. The greatest difference between video mapping and a 3D video mapping lies in the final outcome – with 3D video mapping you get a realistic space effect on an even surface, and with usual, “ordinary” mapping there’s no such depth and realism. Visualia Group specialized in 3D video mapping. One of groups goals is to educate people about this rather “new art”, to show its potential and to present its great effect as realistic as possible. In this particular project the artists tried to tell a story about transiency of things, people and life itself, combining it with almost magical building transformation, so realistic and dreamy at the same time. First idea was to create video mapping with historical basis – the group wanted to show how the building was built and how it survived for 3 centuries. Unfortunately, due to a lack of material and information, the idea was dropped at the end.

The group than decided to follow its free will and artistic spirit and to simply bring life to Casa del Pastors using 3D softwares. Due to Group’s rich experience in 3D video mapping, they decided that the best way to create a really powerful and realistic effect for audience is to draw the whole building from the beginning. The front of the building (facade) was challenging enough: it is made out of bricks so it was a really one great creative challenge for the Group to model and revive the building. The Group also agreed that Casa del Pastors is a great place to “tell the story” about the transiency of time, things and life.. Everything passes and it is unreachable and unstoppable. It exists only in moments, which we’re aware of only when they’re already gone. “Transiency” became the name of Group’s “story”, which has to be told. How to present transiency through images and video animations? After discussing this, the artistic Visualia Group decided to start with creation process – clouds (air element) symbolize imagination from which emerges the water element (life), which is continued to a birth process (the tree) and it culminates with simpler form of life (caterpillar) becoming a higher, more complex form (butterfly) which is a symbol of a free art.

The 3D animation were made by Jean, and he divided this animation process in 4 different parts:

– Geometry – fragmentation and deconstruction

– Contrast – black and white matter with optical and hypnotic illusions

– Creation process – air, water, LIFE

– Colors – free art

Conclusion

Videomapping has a restrictive format, partly due to the aesthetic criteria of those leading these expressions and artistic interventions. There are certain artists that show videomapping as what we can define as behavioural or linear, showing architectural videomapping as a display full of effects produced with and upon the image and structure of the façade. This criterion, based on a restrictive approach of available effects, tends to deplete the poetic of the interventions by playing with an aesthetic model based on visual and sound effects that the public recognizes and expects.

As we have seen in the works by Kònic and Marko Bolković, the architectural mapping has to understand the function of the building onto which the artist works not only as a façade around the building but also as the context in which it is located, its history; bearing in mind that they are working with heritage buildings or historical sites. Could technologies enrich, enhance local heritage, in a new manner of communicating history, arts giving a hand to the economy, offering new and unexpected forms for visiting historical monuments, archaeological sites inspiring an interplay of different interests such as traditions, history, visual and youth culture?

We try to reflect on the rules of temporary or permanent media architecture, on their importance for the continued revitalisation of the area, on the new role of media for public spaces. We point out the shift in the application of these technologies from commercial, advertising purpose to cultural purposes.

There is a need for a strategy of sustainable urban media development that targets the intersections between the need of residents, tourists, government agencies, the commercial interest of companies and the ideas of architects and urban planners in the perspective of connecting cities, and also creating tourist attractions. We need to reflect on the relationship between the transformation of our cities and the identity-conferring power of digital architecture.

A building’s façade is more than just a wall and a framework for the entrance, it always reflects the functions of the building, it has symbolic patterns, signs or architectural elements that identify the urban typology.

If a place can be defined as relational, historical and concerned with identity, then with innovative digital media in public space we can create dynamic revitalization of places in empathy with the area, and the audience, making visible the past, its history and functions. Media façade should primarily have the task of representing the identity of the building, of the city seen as a large number of overlapping stories and connections, history, emotions, humanity.

Referring the book Invisible cities written by Italo Calvino (1972), which suggests a poetic approach to the imaginative potentialities of the cities, the urban screens and the digital mapping works can make visible the invisible and allow us to understand, create and experience the city with all its change, story, secret folds and diversity.

The mapping has to co-exist with the dramatic text, the actor’s actions, the choreography and the sound, taking part in the narrative of the piece and contributing to the tension and development of the piece through the visual evolutions. We can understand it as a treated space, or as a dynamic stage set, that will bring dynamics, stories, actions through time, by its audio-visual evolution. The media co-exists with the actors and/or dancers and brings to the piece another layer in a horizontal hierarchy with the other elements composing it. We can think of a global work, in which the different disciplines and materials participate to the narration and composition of the work.

The audience perceives the piece as a whole and is often surprised by the intimate and somehow magical dialogue established between light-image-sound-object or architecture- and the actors/ bodies, who will bring the human scale and the humane-fictional proportions.

When the projection is interactive, it offers direct participation of the audience in the piece, who will then coexist with the actors and can then become a part of the fictional world, and make the step from social being to active part of the artistic work. There are countless options to have the public involved in an interactive way. There is still much work to do, still a lot to be explored and understood, and the rapid evolution of the technologies involved can also help us to find new solutions and criteria for the artistic intervention in public spaces.

 

Notes

1. Urban screens, is a term coined by Mirjam Struppek considered a pioneer in the field of urban digital surfaces; in 2005 she curated the first Urban Screens Conference in Amsterdam paving the way for Urban screen conference in Manchester and in Melbourne in 2007 and 2008. She wrote an essay Urban potential of public screens for interaction about the mediatization of architecture.

2. An oculus (plural oculi, from Latin oculus, eye) is a circular opening in the centre of a dome or in a wall. Originating in antiquity, it is a feature of Byzantine and Neoclassical architecture. The oculus was used by the Romans, one of the finest examples being that in the dome of the Pantheon. Open to the weather, it allows rain to enter and fall to the floor, where it is carried away through drains. Though the opening looks small, it actually has a diameter of 27 ft (8.2 m) allowing it to light the building just as the sun lights the earth. The rain also keeps the building cool during the hot summer months (from Wikipedia).

Works Cited

Arcagni, Simone. Urban screen e live performance. 2009. Web. 10 May 2015.

Artieri Boccia, Giovanni. Preface. “La sostanza materiale dei media. Video culture digitali tra

virtuale e performance”. Videoculture digitali. Spettacolo e giochi disuperficie nei nuovi

media. By Andrew Darley. Milano: FrancoAngeli, 2006.

Darley, Andrew. Videoculture digitali. Spettacolo e giochi disuperficie nei nuovi media. Milano:

FrancoAngeli. 2006. Print.

Maldonado, Tomàs. Reale e virtuale. Milano: Feltrinelli, 1992. Print.

Manovich, Lev. Il linguaggio dei nuovi media. Milano: Olivares 2002.

Monteverdi, Anna Maria. Hypersuperface and Mediafaçade. Urban Screen. 2014. Web. 02 May

2015.

—, Drammaturgy of Videomapping. Interview to Koniclab. 2014. Web. 06 May 2015.

—, L’Orfeo di Castellucci, Musica celestiale per un angelo in coma vigile. 2014. Web. 08 May

2015.

Virilio, Paul. “Dal media building alla città globale: i nuovi campi d’azione dell’architettura e

dell’urbanistica contemporanee”,Crossing n°1, numero monografico

Media Buildings,Dec. 2000: 10-11. Print.