In the last post – The Year of Augmented Reality – we dealt with AR/VR equipment and software, so let’s continue with content and communication. In one sense, AR content should be simple because the background is the real world: AR is just an overlay that makes the world more engaging. But that often means the content has to be more intricately designed than cinematic content.

Active 3D Game Figures

The figures in Pokémon GO aren’t as realistic today as the ones we’ll see in future AR systems. Just as the figures in video games have evolved from lines and circles in Pong to the semi-realistic images of today, AR figures while become more lifelike and realistic. It’s not difficult to imagine them reacting to the real world itself – hair flying in the wind, facial expressions changing when dogs bark, running from cars in the street, reacting to people. AR social gaming will allow us to share the figures that are parts of our games with other people so they can become parts of theirs as well.

Shared AR content is a lovely challenge for app designers and content creators. Imagine a Pokémon in front of you while another player coming toward you sees the same figure. The figure has to be rendered in 3D to both people. The 3D rendering is some sort of textured wireframe model that has a three dimensional location in the real world. As you move around it you see different aspects of the image, and it reacts to you and to the other person.

3D images are made out of geometric models with textured surfaces. There’s a lot of computation in their description, and communicating that description requires bandwidth. There are many applications for this application. As well as action/adventure games like Pokémon, it wouldn’t be too difficult to make something like a virtual zoo for kids.

Educational applications would be endless. When teaching a lesson on dinosaurs, the teacher could make them appear in the classroom as life-size 3D holograms complete with sound. That beats a picture in a textbook and a scaled down action figure. This should be commonplace in less than twenty years.

 Virtual and Augmented Surgery

Adding additional forms of interaction expands the use cases. Let’s say you’re a surgery professor teaching med students how to transplant hearts. This is highly skilled work where test subjects are somewhat scarce. But if we have active 3D figures, we can train on virtual patients. Imagine a haptic feedback scalpel that simulates the feel of slicing through skin and muscle and stops when it hits bone. Or a surgical laser similar to those used in remote surgery. The operating room scenario involves multiple people, so the imagery needs to be shared the way it is in the game.

The educational value would be enhanced with scripts that simulate problems that may arise the course of surgery. Training content writers could create particular organ defects, heart attack scenarios, ruptured blood vessels, inattentive nurses, and anything else that aids learning.

Before a new doctor attempts a real-world heart transplant, they may have performed the surgery in the virtual scenario dozens of times. Real-world surgery would be a specialized form of AR taught to the surgeon in mixed AR/VR scenario. Similar systems are already used for pilot and drone operator training and remote surgery.

Augmented Reality Drama

The Star Trek holodeck is the classic example of AR/VR drama. Some commenters say we’re hundreds of years away from creating a holodeck, but I suspect we’re much, much closer to workable versions that provide unique scripted entertainment. Perhaps the first versions of AR drama will be scripted movies in which the viewer sees the action from the point of view of a character without any control over the action. We could do that with little more than AR cameras on the actors while the drama is filmed.

From there, the story can be evolved with multiple plotlines selected from a limited set of choices afforded to the viewer/participant. In one scenario, you might be a cowboy with a chance to kill the bad guy. If you take him out, the plot moves one way, and if you fail it goes another. Westworld makes a lot more economic sense as an AR theme park than one with actual robots.

Creating AR Scenarios

Scripted entertainment and education scenarios would be expensive to make with today’s technology, but scripting tools will improve. Even if we count on predictable tool improvements, it will probably take many more writers and technicians than today’s movies, which means jobs.

Special effects technologies will need to improve, but Hollywood is already invested in CGI houses that create virtual creatures and scenes. In some sense, an augmented reality version of Westworld is simply a continuation of the technology assists that have always been part of filmmaking.

Making the viewer part of the action is new, however, so it’s going to take some fairly close coordination among all the vertical parts of the entertainment supply chain to make it happen quickly. If we stick to old ways, it may take hundreds of years to create a Holodeck. But I hope it doesn’t.

In the next post, I’ll look at some augmented reality applications in productivity.