Using Kinect to get kids off the couch

Studies show that embodied learning—learning by involving the whole body—helps kids grasp new concepts. Microsoft's Soho Productions studio was given a mighty task to do just that with Sesame Street using the Xbox + Kinect.

Kinect Sesame Street TV

Overview

For decades, Sesame Street has drawn its young audience in with audience participation—songs that encourage singing and dancing along, answering Elmo’s questions, or counting with Count von Count.

Kinect Sesame Street TV, a collaboration between Sesame Workshop and Soho Productions, aimed to do this on a whole new level. Using the Xbox + Kinect, kids are dropped into the middle of Sesame Street to jump, move, wave and play with Elmo & friends.

My Role

I hopped across the pond to London in 2011 and joined the small but mighty Creative Media team as a motion designer and video editor. I worked alongside game designers, user researcers, engineers, art directors and producers to create interactive video branching magic.

A promo I edited, giving you an idea of the type of interactivity for kids to experience.

Motion Design as User Feedback

With video as the medium, motion graphics functioned as a layer providing immediate feedback to the child in front of the screen. A character on screen might ask them to get up and jump, wave or shout in order to help them complete a task. Without visual or auditory feedback, kids lose interest, feel overwhelmed or may not feel involved in the story.

In this example, kids are asked to wave to the character on screen. In usability sessions, kids weren't sure what to do, or if they did wave, their wave wasn't long enough or wide enough for the Kinect to detect it. We needed a big, enthusiastic wave. After several experiments, I added a paw animation with particles in an arc shape to mimic the action the child should do—it worked!

Once the child waves, as detected by the Kinect, the specific shape that they've learned in the segment is reinforced through on-screen graphics.

Making Old Footage Interactive

An interesting challenge came with taking existing Sesame segments from past seasons and transforming them into an interactive experience. These were filmed on set with human cast members and muppets, sometimes featuring a celebrity guest (like Jason Schwartzman 💖).

The scenes were cut down to ~7 minutes and to keep kids engaged, the team decided to “hide” objects throughout the scene for kids to find (through voice and gesture detected by the Kinect).

There needed to be varying levels of difficulty in finding the objects to keep kids engaged, which came with a number of technical and usability challenges:

  • Blending graphics into complex video footage.
  • Balancing the frequency of the objects, the position and size of the object, the length of time on-screen and the level of visibility.

Left, the pumpkin is "hidden" and animates when found, right. These scenes involved meticulous camera tracking, rotoscoping and compositing in After Effects. Think of all the muppet fur! The objects fit in the scenes seamlessly, often blending in with backgrounds or hiding behind characters or objects.

Our User Research team tested segments on a weekly basis with preschoolers, revealing objects that were too difficult to find, too dark or too small or onscreen for too short of a time, so we could go back to the footage and iterate.

Adapting for Touch Screens

Another interesting challenge came when we began working on a touch version for Windows Surface. The challenge was to use the same content, but the interactions and voiceover prompts needed to be redesigned for tap and swipe rather than motion and voice.

I worked with our game designers and engineers to restructure and redesign the video content and graphics to correspond with the tap and swipe gestures.

An example of this was a segment that originally required the child to jump along with Cookie Monster, which was adapted to be a tap instead. The challenge was that we only had the original footage and assets, so this required taking a still image, slicing and dicing to animate it, and compositing it.

Results

Kinect Sesame Street TV was released in September 2012. Microsoft made some snazzy promo videos, it won a BAFTA Children’s Award and got positive reviews from game critics and Amazon critics alike.

It wasn’t quite the game-changer I had anticipated for such an inventive TV experience—owning a $300 Xbox is a barrier for a lot of families and 3-year-olds aren’t so great at navigating TV and Xbox menus themselves—but it was definitely a success for those of us working on it and we had a lot of fun and pride in doing so.