nokia Archives | Digital Cortex

Applying Baudrillard

For Jean Baudrillard (1983), “at any moment in the course of our modernity, a particular arrangement of signifying objects and images conditions the way we see the world” (Clark, 1995). “Each major transformation is accompanied by a feeling of disorientation and discomfort over the loss of the previous ‘reality’. This effects a recourse into the imagined certainties of the receding order to ground or stabilise that which is new. In this way, “reality loops around itself”, as “each phase of value integrates into its own apparatus the anterior apparatus as a phantom reference, a puppet or simulation reference”” (Baudrillard, 1988: 145, 121; cited in Clark, 1995). In these words, we see Baudrillard’s perspective can apply neatly to my analysis of Mobile AR. Taking up where McLuhan left us- a view of the Magic Lens constrained by its deterministic overtones- Baudrillard injects the much-needed element of an actively social construction of Mixed Reality, whilst grounding my work in his Postmodern thought on Virtuality.

I am interested in the view that iterations of reality, whilst overlapping and viewable through the Magic Lens, support and influence each other’s existence within a wider structure. I could live wholly in The Virtual, and bring to it conceptions of the reality from whence I came. We see a similar behaviour in Alternate Reality games such as Second Life (Linden Lab: 2003) or The Sims (Maxis: 2000) whereby developers program known physical world causalities, behaviours and actions despite the near-limitless formal opportunities offered by the medium. Users, when given freedom, will likely bring their own conceits and personal experiences to these alternate realities, thereby foregoing what else might be possible in favour of their own culturally-inherited drives and ambitions. The Magic Lens presents a wholly new canvas for the social construction of reality. The collaborative and democratic Mobile 2.0 ethos that Nokia hope to breathe into Mobile AR could falter if users bring too much of our present iteration of reality to it. The Magic Lens offers an opportunity to reshape The Real, not solely through tagging buildings or leaving messages floating in mid-aid, but through the lessons we might learn through engaging with each other in a new way.

Baudrillard focused his work on how we interface with information, and how we build it into our view of reality. He posited that The Media had hijacked reality, becoming a powerful force in the construction of hyper-reality, a social reality that has become more powerful than we exert control over. Through the Magic Lens, we might give form to some aspects of hyper-reality. The medium allows for virtual elements to co-exist with real objects occupying space in the user’s own hyper-reality. In this way, each user can choose which hyper-reality they want to exist in, whether it is one in which 3D AR avatars walk the streets and go about their virtual lives; or one where arrows and directions graphically point out where to go to fulfil a shopping list’s requirements. The Magic Lens makes a shift from mass-media control to personalised, user-focused context-based reality: Reality 2.0 if you will.

Assuming AR does present a new layer to reality, there are certain Baudrillardian imperatives that we will bring to this landscape. One such imperative links the physical properties of real-world space- gravity, mass, optics- to our new environment. To make sense of virtual elements in their context we will employ what we already know about the environment we are in. This means that the most prized virtual objects will exhibit expected behaviour, intuitive interactivity and will be visually suited to its surroundings. Similarly, an object’s location in space alters its perceived importance. I would argue that should a common Mixed Reality exist, governing bodies would write entire protocol for the positioning and size of virtual objects so that one contributor could not take up more than his worth. Important to consider is that even writing hypothetically I am bringing Baudrillardian imperatives to task, applying democracy to a non-existent world! Baudrillard’s “reality loops around itself” has a troublesome effect on my analysis. Let me instead take a fresh perspective, in my next section written from the perspective of Walter Benjamin…

Summary So Far

In summary, Mobile AR has many paths leading to it. It is this convergence of various paths that makes a true historical appraisal of this technology difficult to achieve. However, I have highlighted facets of its contributing technologies that assist in the developing picture of the implications that Mobile AR has in store. A hybridisation of a number of different technologies, Mobile AR embodies the most gainful properties of its three core technologies: This analyst sees Mobile AR as a logical progression from VR, but recognises its ideological rather than technological founding. The hardware basis of Mobile AR stems from current mobile telephony trends that exploit the growing capabilities of Smartphone devices. The VR philosophy and the mobile technology are fused through the Internet, the means for enabling context-based, live-updating content, and housing databases of developer-built and user-generated digital objects and elements, whilst connecting users across the world.

I have shown that where the interest in VR technologies dwindled due to its limited real-world applicability, Mobile Internet also lacks in comparison to Mobile AR and its massive scope for intuitive, immersive and realistic interpretations of digital information. Wearable AR computing shares VR’s weaknesses, despite keeping the user firmly grounded in physical reality. Mobile AR offers a solution that places the power of these complex systems into a mobile telephone: the ubiquitous technology of our generation. This new platform solves several problems at once, most importantly for AR developers and interested Blue-chip parties, market readiness. Developing for Mobile AR is simply the commercially sensible thing to do, since the related industries are already making the changes required for its mass-distribution.

Like most nascent technologies, AR’s success depends on its commercial viability and financial investment, thus most sensible commercial developers of AR technologies are working on projects for the entertainment and advertising industries, where their efforts can be rewarded quickly. These small-scale projects are often simple in concept, easily grasped and thus not easily forgotten. I claim here that the first Mobile AR releases will generate early interest in the technology and entertainment markets, with the effect that press reportage and word-of-mouth behaviour assist Mobile AR’s uptake. I must be careful with my claims here however, since there is no empirical evidence to suggest that this will occur for Mobile AR. Looking at the emergence of previous technologies, however, the Internet and mobile telephony grew rapidly and to massive commercial success thanks to some strong business models and advancements in their own supporting technologies. It is strongly hoped by developers like Gameware and T-Immersion that Mobile AR can enjoy this same rapid lift-off. Both technologies gained prominence once visible in the markets thanks to a market segment called early adopters. This important group gathers their information from specialist magazine sources and word of mouth. Mobile AR developers would do well to recognise the power of this group, perhaps by offering shareware versions of their AR software that encourage a form of viral transmission that exploit text messaging.

Gameware have an interesting technique for the dissemination of their HARVEE software. They share a business interest with a Bluetooth technology firm, which has donated a prototype product the Bluetooth Push Box, which scans for local mobile devices and automatically sends files to users in acceptance. Gameware’s Push Box sends their latest demo to all visitors to their Cambridge office. This same technology could be placed in public places or commercial spaces to offer localised AR advertising, interactive tourist information, or 3D restaurant menus, perhaps.

Gameware, through its Nokia projects and HARVEE development program is well placed to gain exposure on the back of a market which is set to explode as mobile offerings become commercially viable, ‘social’, powerful, multipurpose and newsworthy. Projects like HARVEE are especially interesting in terms of their wide applicability and mass-market appeal. It is its potential as a revolutionary new medium that inspires this very series.

Mobile Telephone

The Internet and the mobile phone are two mighty forces that have bent contemporary culture and remade it in their form. They offer immediacy, connectivity, and social interaction of a wholly different kind. These are technologies that have brought profound changes to the ways academia consider technoscience and digital communication. Their relationship was of interest to academics in the early 1990’s, who declared that their inevitable fusion would be the beginning of the age of Ubiquitous Computing: “the shift away from computing which centered on desktop machines towards smaller multiple devices distributed throughout the space” (Weiser, 1991 in Manovich, 2006). In truth, it was the microprocessor and Moore’s Law- “the number of transistors that can be fit onto a square inch of silicon doubles every 12 months” (Stokes, 2003) that led to many of the technologies that fall under this term: laptops, PDA’s, Digital Cameras, flash memory sticks and MP3 players. Only recently have we seen mobile telephony take on the true properties of the Internet.

The HARVEE project is partially backed by Nokia Corp. which recognises its potential as a Mobile 2.0 technology: user-generated content for mobile telephony that exploits web-connectivity. Mobile 2.0 is an emerging technology thematically aligned with the better established Web 2.0. Nokia already refer to their higher-end devices as multimedia computers, rather than as mobile phones. Their next generation Smartphones will make heavy use of camera-handling systems, which is predicated on the importance of user-generated content as a means to promote social interaction. This strategic move is likely to realign Nokia Corp.’s position in the mobile telephony and entertainment markets.

Last year, more camera phones were sold than digital cameras (Future Image, 2006). Nokia have a 12 megapixel camera phone ready for release in 2009, and it will be packaged with a processing unit equal to the power of a Sony PSP (Nokia Finland: non-public product specification document). MP3 and movie players are now a standard on many handsets, stored on plug-in memory cards and viewed through increasingly higher resolution colour screens. There is a growing mobile gaming market, the fastest growing sector of the Games Industry (Entertainment & Leisure Software Publishers Association (ELSPA) sales chart). The modern mobile phone receives its information from wide-band GPRS networks allowing greater network coverage and faster data transfer. Phone calls are the primary function, but users are exploiting the multi-media capabilities of their devices in ways not previously considered. It is these factors, technologic, economic and infrastructural that provide the perfect arena for Mobile AR’s entry into play.

Mobile Internet is the natural convergence of mobile telephony and the World Wide Web, and is already a common feature of new mobile devices. Mobile Internet, I would argue, is another path leading to Mobile AR, driven by mobile users demanding more from their handsets. Mobile 2.0 is the logical development of this technology- placing the power of location-based, user-generated content into a new real-world context. Google Maps Mobile is one such application that uses network triangulation and its own Google Maps technologies to offer information, directions, restaurant reviews or even satellite images of your current location- anywhere in the world. Mobile AR could achieve this same omniscience (omnipresence?) given the recent precedent for massively multi-user collaborative projects such as Wikipedia, Flickr and Google Maps itself. These are essentially commercially built infrastructures designed to be filled with everybody’s tags, comments or other content. Mobile AR could attract this same amount of devotion if it offered such an infrastructure and real-world appeal.

There is a growing emphasis on Ubiquitous Computing devices in our time-precious world, signified by the increased sales in Smartphones and WiFi enabled laptops. Perhaps not surprisingly, Mobile Internet use has increased as users’ devices become capable of greater connectivity. Indeed, the mobile connected device is becoming the ubiquitous medium of modernity, as yet more media converge in it. It is the mobile platform’s suitability to perform certain tasks that Mobile AR can take advantage of, locating itself in the niche currently occupied by Mobile Internet. Returning to my Mixed Reality Scale, Mobile AR serves the user better than Mobile Internet currently can: providing just enough reality to exploit virtuality, Mobile AR keeps the user necessarily grounded in their physical environment as they manipulate digital elements useful to their daily lives.

Gameware: A Case-Study in AR Development

I have been aided in this series by a connection with Gameware Development Limited, a Cambridge-based commercial enterprise working in the entertainment industry. Gameware was formed in May 2003 from Creature Labs Ltd, developing for the PC games market which produced the market leading game in Artificial Intelligence (AI), Creatures. When Gameware was formed, a strategic decision was made to move away from retail products and into the provision of technical services. They now work within the Broadcasting and Mobile Telephony space in addition to the traditional PC market. I use this business as a platform to launch into a discussion of the developments current and past that could see AR become a part of contemporary life, and just why AR is such a promising technology.

Gameware’s first explorations into AR came when they were commissioned by the BBC to develop an AR engine and software toolkit for a television show to be aired on the CBBC channel. The toolkit lets children build virtual creatures or zooks at home on their PCs which are uploaded back to the BBC and assessed:

A typical Zook, screenshot taken from Gameware's Zook Kit which lets children build virtual creatures

The children with the best designs are then invited to the BAMZOOKi studio to have their virtual creatures compete against each other in a purpose-built arena comprised of real and digital elements. The zooks themselves are not real, of course, but the children can see silhouettes of digital action projected onto the arena in front of them. Each camera has an auxiliary camera pointed at AR markers on the studio ceiling, meaning each camera’s exact location in relation to the simulated events can be processed in real time. The digital creatures are stitched into the footage, and are then navigable and zoomable as if they were real studio elements. No post-production is necessary. BAMZOOKi is currently in its fourth series, with repeats aired daily:

BAMZOOKi, BBC's AR game show where children’s zooks compete in a studio environment

BAMZOOKi has earned Childrens BBC some of its highest viewing figures (up to 1.2 million for the Monday shows on BBC1 and around 100,000 for each of the 20 episodes shown on digital Children’s BBC), which represents a massive milestone for AR and its emergence as a mainstream media technology. The evidence shows that there is a willing audience already receptive to contemporary AR applications. Further to the viewing figures the commercial arm of the BBC, BBC Worldwide, is in talks to distribute the BAMZOOKi format across the world, with its AR engine as its biggest USP. Gameware hold the rights required to further develop their BAMZOOKi intellectual property (IP), and are currently working on a stripped down version of their complex AR engine for the mobile telephony market.

I argue, however, that Broadcast AR is not the central application of AR technologies, merely an enabler for its wider applicability in other, more potent forms of media. Mobile AR offers a new channel of distribution for a variety of media forms, and it is its flexibility as a platform that could see it become a mainstream medium. Its successful deployment and reception is reliant on a number of cooperating factors; the innovation of its developers and the quality of the actual product being just part of the overall success the imminent release.

As well as their AR research, Gameware creates innovative digital games based on their Creatures AI engine. They recently produced Creebies; a digital game for Nokia Corp. Creebies is one of the first 3D games which incorporates AI for mobile phones. Gameware’s relationship with Nokia was strengthened when Nokia named them Pro-Developers. This is a title that grants Gameware a certain advantage: access to prototype mobile devices, hardware specifications, programming tools and their own Symbian operating system (Symbian OS) for mobile platforms. It was this development in combination with their experiences with BAMZOOKi and a long-standing collaboration with Cambridge University which has led to the idea for their HARVEE project. HARVEE stands for Handheld Augmented Reality Virtual Entertainment Engine.

Their product allows full 3D virtual objects to co-exist with real objects in physical space, viewed through the AR Device, which are animated, interactive and navigable, meaning the software can make changes to the objects as required, providing much space for interesting digital content. The applications of such a tool range from simple toy products; advertising outlets; tourist information or multiplayer game applications; to complex visualisations of weather movements; collaborating on engineering or architectural problems; or even implementing massive city-wide databases of knowledge where users might ‘tag’ buildings with their own graphical labels that might be useful to other AR users. There is rich potential here.

In HARVEE, Gameware attempt to surmount the limitations of current AR hardware in order to deliver the latest in interactive reality imaging to a new and potentially huge user base. Indeed, Nokia’s own market research suggests that AR-capable Smartphones will be owned by 25% of all consumers by 2009 (Nokia Research Centre Cambridge, non-public document). Mobile AR of the type HARVEE hopes to achieve represents not only a significant technical challenge, but also a potentially revolutionary step in mobile telephony technologies and the entertainment industry.

Gameware’s HARVEE project is essentially the creation of an SDK (Software Development Kit) which will allow developers to create content deliverable via their own Mobile AR applications. The SDK is written with the developer in mind, and does the difficult work of augmenting images and information related to the content. This simple yet flexible approach opens up a space for various types of AR content created at low cost for developers and end-users. I see Mobile AR’s visibility on the open market the only impediment to its success, and I believe that its simplicity of concept could see it become a participatory mass-medium of user-generated and mainstream commercial content.

What is AR and What is it Capable Of?

Presently, most AR research is concerned with live video imagery and it’s processing, which allows the addition of live-rendered 3D digital images. This new augmented reality is viewable through a suitably equipped device, which incorporates a camera, a screen and a CPU capable of running specially developed software. This software is written by specialist software programmers, with knowledge of optics, 3D-image rendering, screen design and human interfaces. The work is time consuming and difficult, but since there is little competition in this field, the rare breakthroughs that do occur are as a result of capital investment: something not willingly given to developers of such a nascent technology.

What is exciting about AR research is that once the work is done, its potential is immediately seen, since in essence it is a very simple concept. All that is required from the user is their AR device and a real world target. The target is an object in the real world environment that the software is trained to identify. Typically, these are specially designed black and white cards known as markers:

An AR marker, this one relates to a 3D model of Doctor Who's Tardis in Gameware's HARVEE kit

These assist the recognition software in judging viewing altitude, distance and angle. Upon identification of a marker, the software will project or superimpose a virtual object or graphical overlay above the target, which becomes viewable on the screen of the AR device. As the device moves, the digital object orients in relation to the target in real-time:

armarker2 — Augmented Reality in action, multiple markers in use on the HARVEE system on a Nokia N73

The goal of some AR research is to free devices from markers, to teach AR devices to make judgements about spatial movements without fixed reference points. This is the cutting edge of AR research: markerless tracking. Most contemporary research, however, uses either marker-based or GPS information to process an environment.

Marker-based tracking is suited to local AR on a small scale, such as the Invisible Train Project (Wagner et al., 2005) in which players collaboratively keep virtual trains from colliding on a real world toy train track, making changes using their touch-screen handheld computers:

crw_80271 — The Invisible Train Project (Wagner et al., 2005)

GPS tracking is best applied to large scale AR projects, such as ARQuake (Thomas et al, 2000), which exploits a scale virtual model of the University of Adelaide and a modified Quake engine to place on-campus players into a ‘first-person-shooter’. This application employs use of a headset, wearable computer, and a digital compass, which offer the effect that enemies appear to walk the corridors and ‘hide’ around corners. Players shoot with a motion-sensing arcade gun, but the overall effect is quite crude:

100-0007_img_21 — ARQuake (Thomas et al, 2000)

More data input would make the game run smoother and would provide a more immersive player experience. The best applications of AR will exploit multiple data inputs, so that large-scale applications might have the precision of marker-based applications whilst remaining location-aware.

Readers of this blog will be aware that AR’s flexibility as a platform lends applicability to a huge range of fields:

Current academic work uses AR to treat neurological conditions: AR-enabled projections have successfully cured cockroach phobia in some patients (Botella et al., 2005);
There are a wide range of civic and architectural uses: Roberts et al. (2002) have developed AR software that enables engineers to observe the locations of underground pipes and wires in situ, without the need schematics
AR offers a potentially rich resource to the tourism industry: the Virtuoso project (Wagner et al., 2005) is a handheld computer program that guides visitors around an AR enabled gallery, providing additional aural and visual information suited to each artefact;

The first commercial work in the AR space was far more playful, however: AR development in media presentations for television has led to such primetime projects as Time Commanders (Lion TV for BBC2, 2003-2005) in which contestants oversee an AR-enabled battlefield, and strategise to defeat the opposing army, and FightBox (Bomb Productions for BBC2, 2003) in which players build avatars to compete in an AR ‘beat-em-up’ that is filmed in front of a live audience; T-Immersion (2003- ) produce interactive visual installations for theme parks and trade expositions; other work is much more simple, in one case the BBC commissioned an AR remote-control virtual Dalek meant for mobile phones, due for free download from BBC Online:

A Dalek, screenshot taken from HARVEE's development platform (work in progress)

The next entry in this series is a case study in AR development. If you haven’t already done so, please follow me on Twitter or grab an RSS feed to be alerted when my series continues.