Opportunity

With the explosion of Pokémon GO, augmented reality made a huge debut in the consumer spotlight in 2016. (Our office is actually a Pokéstop.) Augmented reality, or AR, is a technology that superimposes computer-generated content onto a real-world environment, blending together physical and digital experiences into a composite view. [1] While augmented reality and virtual reality are usually clumped together in conversations about new technology, both are distinctly different experiences. Truly immersive VR, going beyond Google Cardboard, requires more sophisticated hardware and dedicated space. Augmented reality can be delivered through mobile devices and has more universal applications than VR, which lends itself towards gaming, film, and niche enterprise uses. According to Digi-Capital, a mobile internet and AR/VR consulting firm, the AR industry is estimated to be worth $83B by 2021, over three times that of VR. Furthermore, augmented reality’s addressable market is similar to that of smartphones and tablets (especially in terms of hardware price points), making the space highly competitive. [2]

Inspiration

With all of the excitement buzzing around augmented reality, we wanted to tinker with the technology and flex our creative muscles in Jackrabbit Labs. We started gathering inspiration by researching the different types of AR apps out there, especially mobile games. What we discovered was that most AR games currently on the market required players to print out a marker, a reference point that allows the phone’s camera to position the game.

The appeal of mobile AR for our team was how easy it was to start playing a game just about anywhere, regardless of environment or materials. The idea of having to print out and carry a piece of paper around was a design challenge that we wanted to tackle.

From a brainstorming session came the idea of translating cornhole – the popular lawn game – into an augmented reality experience, and that’s what we sought out to do.

Design Process

Our team started the design process by doing technical discovery, which involved downloading different SDK libraries and playing around with sample applications. We narrow it down to two libraries: ARToolKit and Vuforia. We decided to go with Vuforia because it allowed players to create user-defined targets. Our software architecture included OpenGL as the renderer, SceneKit as the physics engine and asset generator, and Vuforia as the AR framework.

(Read more in-depth analysis in our Technical Breakdown below.)

After the initial design and groundwork set by various members of our team, our iOS developer extraordinaire, Hunter, took the lead and built out the majority of the app.

Going back to the marker challenge, we were able to build a custom marker-making feature, i.e. the ability to create a user-defined target. That means that a player can grab any nearby object that’s flat with high contrast (such as a beach towel or magazine cover) and turn it into their very own marker. We haven’t encountered this type of solution anywhere else in regards to mobile games.

During our research, we learned that for most mobile AR games, such as ARBasketball, the screen view of the game experience was usually engaging and immersive, but the motion range was limited to a simple swipe gesture. For example, shooting a basketball translated to swiping upward on the phone to “shoot” the ball into the hoop. We thought that a swipe only motion broke the illusion of a blended experience (physical and virtual). For our cornhole game, we wanted to preserve the “normal” motion of tossing. We incorporated that into our features, so a player in ARC can toss upward like they would with a real bean bag. We also kept the simple swipe motion as an alternative tossing option.

Another feature we decided to build was a leaderboard to encourage competition and incentivize game play. If a player allows location access, a local leaderboard appears which tracks the top weekly scores within a 2.5 mile radius. If a player does not allow location access, they’ll see a global leaderboard which tracks the top weekly scores of other players who chose not to share their location.

ARC is now available for iOS and, as the first iteration of our labs project, we’d love to hear your feedback. Download the free game, submit feedback, and invite your friends!

Future Applications

Moving forward, we plan to continue improving ARC through user feedback and tinkering with more AR/VR technologies. Here are general trends we believe will shape the AR/VR industry and where we see ourselves getting involved.

  • Since AR isn’t controlled by platforms, we’ll continue to experiment freely on mobile – where most early AR development will take place. [4]
  • Competition for VR hardware is heating up, which will bring price points down and make development more accessible. [3]
  • Retail AR marketing, such as trying on clothes or seeing what furniture looks like in your home, will be here to stay. [5]
  • Pro sports are and will continue to be heavily impacted by AR/VR. We’re excited to see how the technologies will affect athlete training and how fans experience the games. [3]
  • Lastly, device makers are encouraging more content creation to populate their ecosystems. [3] Diverse content creators (such as musicians and visual artists) will lead to new experiences that we’ve never dreamed of.

Help us shape the future of mobile gaming by downloading ARC and letting us know what you think!

[1] “Augmented Reality in Retail Recommendations.” Accenture. Accenture, n.d. Web. 19 Dec. 2016.

[2] “After mixed year, mobile AR to drive $108 billion VR/AR market by 2021.” NEWS DigiCapital. Digi-Capital, Jan. 2016. Web. 18 Jan. 2016.

[3] Bardi, Joe. “5 Top Virtual Reality & Augmented Reality Trends 2016.” Marxent. Marxent, 22 Apr. 2016. Web. 19 Dec. 2016.

[4] Martin, Ellie. “5 Augmented Reality Trends.” Datamation. Datamation, 26 Oct. 2016. Web. 19 Dec. 2016.

[5] McKone, Dan, Robert Haslehurst, and Maria Steingoltz. “Virtual and Augmented Reality Will Reshape Retail.” Harvard Business Review. Harvard Business Review, 09 Sept. 2016. Web. 19 Dec. 2016.

Technical Breakdown

To create an augmented-reality experience, ARC combines two frameworks: Vuforia and SceneKit. The former uses a camera to track a real-world image, and the latter renders a three-dimensional scene that simulates a physical environment. With these two frameworks, ARC uses a tracked image to orient and scale an overlaid scene containing a centralized board and bean bags that can be tossed.

Vuforia for User-Defined Targets and Tracked Images

Although there are many frameworks capable of tracking real-world images, Vuforia uniquely offers two advantages: user-defined targets and extended tracking. First, whereas other frameworks restrict the number of trackable images to a predefined subset that users must print out at home, Vuforia allows users to define targets on the fly. In other words, Vuforia allows an ordinary image, such as a magazine cover, newspaper page, or beach towel, to be scanned and tracked. Second, whereas other frameworks can only track the contents of a particular image, Vuforia can also extend tracking to the contents of the surrounding environment. As a result, even when the tracked image leaves the captured frame of a camera, Vuforia can use the surrounding environment to infer the position of the tracked image. Because of these two features, when compared to similar frameworks, Vuforia promotes a more convenient and a more reliable augmented-reality experience.

SceneKit for 3D Graphics

SceneKit is an Apple framework that provides a high-level interface to render three-dimensional graphics. Nodes serve as the fundamental building blocks of scenes, and every scene has a root node from which all other nodes extend. These nodes can represent objects with geometrical shapes (e.g., planes, boxes, and spheres) and physical properties (e.g., mass, friction, and acceleration). Additionally, these nodes can represent lights and cameras so that rendered scenes can be illuminated from a particular angle and captured from a particular perspective. For ARC, every game renders a scene in which various nodes represent a transparent floor, a stationary board, a movable camera, and bean bags that can be tossed from the camera toward the board. Additionally, ambient lights are used to provide illumination from every angle within the scene.

Better Together, Combining Vuforia and SceneKit

ARC combines Vuforia and SceneKit to produce an augmented-reality experience specifically tailored for iOS. For the underlying technology to work correctly, we guide the user through a strict yet convenient on-boarding process to set up every game. First, with the camera and the accelerometer, we assist the user in defining a target. To successfully define a target, two conditions must be satisfied. The user must capture a high-contrast image that can be reliably tracked, and the user must also define a target that’s lying horizontally on the ground since that’s how the overlaid scene should be rendered. After satisfying those two conditions with the camera and the accelerometer, we collect the length of the tracked image. The collected length allows us to properly scale the distance of the camera node from the board node within the rendered scene. Finally, with the tracked image and collected length, we overlay a rendered scene and dynamically update the position and the rotation of a camera node within that scene so that they correspond to the tracked image. Ultimately, while looking at the screen, the board and bags appear to exist within the real world.