E3 2012 | Wonderbook: Augmented Reality Storytelling

From the creators of EyePet (Sony London Studio) comes Wonderbook, a new entry in the field of augmented reality applications for PlayStation 3. No fluffy virtual pets this time around, but books that come to life.

Actually, one book, atomically speaking, comprised of a few pages full of AR markers (see below). Open this book in front of the PS Eye connected to a PS3 running dedicated software and magical things happen. Literally so in the case of the debut content, which is Book of Spells, an an interactive storytelling application developed by Sony in collaboration with J.K. Rowling. Yep, that J.K Rowling.

Based on the Harry Potter universe, Wonderbook: Book of Spells turns the physical book (that will likely be bundled with the software) into a textbook for young wizards. Flipping through its pages, decorated with computer generated graphics, you’ll learn how to cast spells using the PlayStation Move as a magic wand.

Judging from the various online video demonstrations recorded by those who paid visit to the Sony E3 booth, this learning process varies depending on the spell but it generally entails discovering said spell (and the story behind it) by interacting with the pages in various ways, pronouncing the name of a spell (uses voice recognition), casting it by drawing a specific shape in the air using the wand, and eventually practicing with it by flicking or pointing the controller.

The first learning phases play out in the context of your actual living room, with various augmented reality graphics enriching the process as the pages of the Wonderbook come to life and react to your inputs. This results in a rather rich and dynamic showcase of AR visuals, ranging from animated drawings on the book pages to interactive 3D objects spawning upon them, or above them, or even “through” them. In one particular occasion, while learning the fire-making spell, a little gate opens on the book, reveling a cave running deep perpendicularly to the page. Tilting the book so that it faces the PS Eye camera, a fireball rises from the depths of the cave, comes out of the gate and eventually morphs into the flaming word “Incendio” above the book.

Practicing with a given spell triggers an interesting metamorphosis of both the playing field and the interaction methods as the video feed shrinks on the screen to become a portal-like window overlooking a paper-made recreation of an area of the Hogwarts school. From that portal, you are tasked with casting spells to interact with the environment before you. Examples include flicking the Move (Sorcery-style) to throw fireballs or pointing it to orientate a beam of shrinking magic (guess it’s “Reducio”) towards various creatures flying or crawling around the place. This sort of AR-based gameplay can also be found in some of the Start The Party: Save The World minigames by the way.

There are 20 spells overall to learn in Book of Spells, 4 for each one of its 5 chapters. At the end of a chapter, you’ll have to pass a final test where you’ll need to use all the spells learned in that chapter, switching between them via gestures.

From a technical standpoint, Book of Spells impresses in various ways. First of all, unlike other examples of AR applications such as The Eye Of Judgment (an amazing card game by the way, killed by an ill-conceived business model), here the alignment of computer generated graphics with the markers of the physical book appears to be rather robust, with little to none delay, at least as long as you don’t frantically move the thing around, I suppose. This is of course the result of the bigger and richer assortment of markers available for tracking the book compared to the tiny ones located on the smaller cards of The Eye Of Judgment.

Another cool aspect of the tech is how it manages to display computer generated visuals below portions of the live footage. As you are well aware of, the PS Eye is not a depth sensing device, and therefore it doesn’t allow for a realistic placement of virtual objects within a scene, resulting in physical real life objects always appearing behind computer generated graphics, regardless of their actual position along the Z axis.

Now that’s in theory. In practice, Wonderbook cleverly manages to fake some depth sensing via a few video processing tricks. The first one is showcased in the picture below. Here the player is asked to clean up the Wonderbook (after having extinguished a virtual fire) scrubbing its pages with his hands. Notice how the hand is visible even tho it should be hidden by the AR graphics (not referring to the burn marks, which are of course erased as soon as the hand moves onto them, but to the AR drawings below).

Below is another example of this feature, this time involving the cover of the book. See how the fingers overlay the virtual cover of the Wonderbook rather than being hidden by it.

How is that possible? Well, unless the guys at London Studio managed to magically turn the now 5 years old PSEye into a super-precise depth sensing camera, what I think is actually going on is that basically the graphics being drawn on the book are dynamically “cut-out” around the contour of the hand, probably using the same motion detection technique of the old EyeToy days, when just moving a hand would leave a trail on a steamed window.

Things get even more interesting when looking closely at the virtual magic wand. But first, a brief preamble: with regards to augmented reality graphics applied to the Move unit itself (as seen in games such as EyePet or Start The party), in order to avoid virtual objects to overlap the user’s hand or fingers, the handles of such objects are not rendered at all. In the case of a virtual sword, for example, only the blade is being rendered, letting the actual Move being the hilt.

Below is an example of this taken from EyePet & Friends. Notice how only the tip of the virtual object is being rendered.

Now, notice in the Wonderbook picture below how even tho the whole virtual wand is being rendered, the kid’s fingers are still visible.

In this particular case I guess we are looking at something more sophisticated than the above mentioned technique. What I think is going on is some sort of real-time chroma keying exploiting the strong contrast between the dark tones of the Move surface and the light ones of the hand. Basically, the rendering engine might be told to draw the virtual wand only on top of the dark tones. Since the software knows where the Move is and where to render the virtual wand, that information might very well allow for this effect to be applied exclusively to the relevant portion of the video feed (that is, the one defined my the contour of the virtual wand handle), without affecting the whole image.

Or maybe it’s much more simple than that. Perhaps it’s indeed just an application of the “EyeToy technique” exposed earlier. In any case, you can see this effect very clearly during the video demonstration of Wonderbook: Book of Spells embedded below, especially near the end of it, when the woman is holding the Move on her legs.

Whatever the technique behind this trick actually is, the end result is nevertheless quite neat, one showing a surprisingly high attention to detail. After all, why bother with rendering the whole virtual wand if just the tip of it would be enough to provide the illusion of holding a real magic wand? It’s not like kids would complain? Or would they? I’m not familiar with the Harry Potter fanbase, so I don’t know how much they value authenticity.

Anyway, what I’m sure kids of all kinds will care about is the quality of the execution. The EyePet series, with its legacy of excruciatingly long and frequent loading times, inconsistent hands motion detection and somewhat demanding ambient lighting requirements, didn’t really live up to the “happy family” expectations. More often than not, it led to frustrated kids constantly asking for parental assistance.

So here is hope London Studio has taken care of those issues because the potential of a successful family product is definitely there. J.K. Rowling’s name will likely work its sales magic with the first of “a thousand stories”, but for the Wonderbook platform to be “the next revolution in story-telling” rather than just another piece of Potter merchandise it needs to work as intended, to be easy to use, and more importantly, fast and reliable. Despite the awkward press conference demonstration, impressions from the E3 Sony booth have been generally positive, so I’m inclined to believe lessons have been learned.

Wonderbook: The Book of Spells will release in November in Europe and December in North America. Price is yet to be revealed.