Creating an Augmented Reality (AR) Demo with the Vuforia Studio Pilot Program

Augmented Reality (AR) has been booming the past couple of months, especially with the deployment of Pokémon Go. If you're not familiar with it, Pokémon Go is a mobile app that encourages its users to walk around the real world in order to capture virtual Pokémon characters.

Aside from making a very addicting mobile phone application, AR is now being considered in industrial settings. It can be used to diagnose and troubleshoot machinery, or provide visual instructions for assembly or disassembly. For example, this video shows an AR program walking the user through the steps of changing a circular saw blade. This particular program was created with the Vuforia Studio Pilot Program.

Vuforia Studio Pilot Program

The Vuforia Studio Pilot Program integrates 3 distinct platforms: ThingWorx, Vuforia, and PTC Creo Illustrate. ThingWorx is crucial for transmitting data between devices, such as sensor data collected from a Raspberry Pi to a phone. PTC Creo Illustrate is a technical illustration software that allows you to create 3D sequences and animations with CAD data. Vuforia can then import data from ThingWorx and integrate it with the 3D models and animations from PTC Creo Illustrate to create an AR environment. With these three platforms, the Vuforia Studio Pilot Program is capable of creating some really cool AR experiences.

I created an AR experience for the 2016 AWC BestFest with the Vuforia Studio Pilot Program. If you're interested in using the techniques mentioned in this post to create your very own AR experience, sign up for the Vuforia Studio Pilot Program through here. You'll then get a service URL that contains your ThingWorx and Vuforia instance.

2016 AWC BestFest Beer Tank Demo

The 2016 AWC BestFest is a technology tradeshow held at Saint Arnold's brewery in Houston, Texas. The festival is an opportunity for technology partners to mingle, show off new technologies, and drink beer. Another DMC employee gave a seminar on "Smart Manufacturing" and how IoT platforms can improve the industrial process. We decided some AR demos with IoT platforms would help illustrate this point. Being at Saint Arnold's Brewery, I though it was fitting to have a demo centered around a beer tank.

For the demo, the user scans a specific "ThingMark" with their smartphone. These ThingMarks are essentially QR codes that are tied to unique augmented reality experiences. When you sign up for the pilot program, you get 100 different ThingMarks for your projects. 

Upon scanning the ThingMark, a 3D CAD model of a beer tank appears on the screen alongside with temperature and tank level values. The temperature and tank level values were simulated through a python script on a Raspberry Pi. Additionally, I created a step-by-step visual sequence that showed the user how to open the lid at the top of the tank. 

The following sections give a brief overview of how I accomplished it.   

Setting Up ThingWorks 

ThingWorx Foundation is an Internet of Things (IoT) platform that strives to create a "smart, and connected world". What does this mean exactly? In this smart world, any and all electronic devices would be able to communicate with each other and execute services. For example, your printer would order its own ink cartridges after detecting low ink levels instead of relying on you. Or, your alarm would adjust when it woke you up depending on whether the train was early or late (look at this DMC blog for a similar project on how to track CTA Buses with a Raspberry Pi). You can learn more about how DMC can help you with your IoT projects here.  

Creating ThingShapes, ThingTemplates, and Things

ThingWorx is an object-oriented platform centered around "Things". There are Things, Thing Templates, and Thing Shapes. Thing Shapes are the lowest level, and this is where you define properties, services, events, and subscriptions. So I created two Thing Shapes: TankTempShape, and TankFillShape. Here is where I first defined the temperature, fill level, name, and other properties I wanted to keep track of. I then created a Thing Template called TankTemplate that inherits from my two Thing Shapes. Finally, I created TankThing that inherits from my TankTemplate. The advantage to this object-oriented structure is that I do not have to redefine properties for each Thing I want to set up. If I had multiple beer tanks I wanted to track, then I could easily create several TankThing's and each one would inherit from TankTemplate and already have the proper configurations. Alternatively, you could define all your properties and configurations within your Thing and it would work fine, but you would not be able to easily scale up your model. Once I created my TankThing, I had to ensure that I could receive data from my Raspberry Pi by setting up an application key.

Creating an Application Key

Each time you access your ThingWorx instance, you must login with your user credentials. So when you make an HTTP request with the Raspberry Pi, you'll need to send along the username and password before you can alter the ThingWorx data. Alternatively, you can create an application key within ThingWorx for the specified user. Now when we make the HTTP request you can pass along the application key ID instead. Under the home tab, under security, there is a section for application keys. If there isn't one already created, create a new one for the user you want to use (I used Administrator). Make sure your user has the proper read and write access. When you save it, it should generate an application key ID. Save this ID, you will need it in the next section.

Raspberry Pi Data Simulation

While I chose to simulate data within a python script, you can collect data from a sensor and export that to ThingWorx instead. In order to do that, you'll need to utilize the GPIO library. You can use this reference to set up your inputs to collect sensor data. While I'm not going to go over my simulation code (wouldn't really help you unless you wanted to run through the exact same numbers as me), I'm going to show you how to set up the HTTP request.

Making an HTTP Request to Edit ThingWorx

In order to do this, you will need to import the JSON and Requests library. The Requests library is an awesome tool that makes HTTP requests really simple. Everything you need can be condensed into one line of code.

If you don't have the Requests library already installed, enter the following lines into the terminal to get them.

1.  sudo git clone git://github.com/kennethreitz/requests.git  
2.  cd requests (i.e. change directory into the folder that git has just added)
3.  sudo python setup.py install

Now you're ready for the HTTP request. The following lines of code only work for my specific ThingWorx and Vuforia instance. You'll need to replace the URL with your service URL, replace the blacked out portion with the application key ID from your ThingWorx, and replace TankThing with the name of your Thing. That will ensure that the Raspberry Pi is sending the data to the correct location.

Lastly, you'll need to edit the payload line with your own custom variables. The blue text corresponds to the properties within ThingWorx, and the black text corresponds to the variables within the python script.

Afterwards, save your python script and execute it from the terminal. You can test the script by going into ThingWorx and checking your Thing's properties. Hit the blue refresh button next to Values to see the values change. If it's not updating, check to make sure the URL, Thing name, application key, and payload terms are correctly typed. Also check that your python script is properly collecting/simulating values.

PTC Creo Illustrate

Import an Existing CAD Model

PTC Creo Illustrate is a great program for creating animations and visuals, however, you can't create CAD models. You can use other programs such as PTC Creo Parametric, PTC Creo Elements, SolidWorks, or AutoCAD to create your own model. Alternatively, you can download a file from GrabCAD or other free sites. Once you're ready to import, go to File -> Import and then select your CAD file. When you click Open you should get a window similar to the one below.

Now you can make edits to your model like rotating or translating your device, creating sections, sequences, and animations. For my particular demo, I created a sequence  that shows how to open the lid at the top of the beer tank.

Creating a Sequence

You can create a sequence to showcase anything from assembly, disassembly, movement, or internal components. Once you have your figure imported, click on the "Sequence" tab at the top of the window. You'll now see a slew of visual manipulation options.

Click "Add Step" to begin the process. You can now add whatever motions you desire. There are several effects that you can add to individual pieces such as fly in/out, fade in/out, shake, pulse, unscrew, etc. Play around with them to determine what you want. You can add several effects to one step and PTC Creo will do its best to combine them all into one fluid motion. If you want them to be separated, make sure to put them in different steps. Continue to add steps, translations, and effects until you have the desired sequence. You can then go to "Preview Sequence" to view your entire sequence.

Publishing Your File

Once you're happy with your sequence, you'll need to publish your program in order to get two files: .pvz file, and .pvi file. The .pvz file is your 3D model, whereas the .pvi file stores your animation information. 

1. Go to File -> Publish. 
2. Publish it as a new .pvz file.
3. Go to your new file and RENAME it as a zip file (do not send to a zip file, for some reason this does not seem to work).
4. Extract all files

After extraction, you should be able to see a .pvz file and a .pvi file in your new folder. These files will be imported into Vuforia Studio.

Vuforia Studio

Here is where all the augmented reality magic happens. Here you'll set up your 3D models, 2D and 3D widgets, ThingMark, and import ThingWorx data.   

Importing Your 3D Model and Animation

In order to use your 3D model and animations, you'll need to import your files into Vuforia.

1. On the left under Resources, click the blue plus sign.
2. Click Select File and find your .pvz file and click ADD.
3. Repeat for the .pvi file.
4. Close out of that window. 
5. Drag a 3D Model into your environment.
6. To the right under the "Properties" column, go to "Resource" and select your .pvz file.
7. Under "Sequence URL", select your .pvi file.

Configuring a Button to Control Your Sequence

Now we have a 3D model and sequence imported into Vuforia, but nothing is currently telling it to run. To fix this, I configured toggle buttons that appear on the smartphone screen that execute play/rewind/forward actions.

1. Select 2D view under Canvas.
2. Drag a Toggle Button onto any of the panels (your choice).
3. Under your Toggle Button properties, drag the blue cross icon next to "Click" all the way to the left to your 3D model icon under "Views".
4. A new window will appear where you can select what action that button will have. Selecting "Play" will cause the next step in the sequence to play.

Add as many buttons as desired. As shown in the images of the 2016 BestFest demo, I had four distinct buttons: play, play all, rewind, and forward. You can also change what your buttons look like by changing "Image When Pressed" and "Image When Not Pressed" under Properties. You can import your own button images following the procedure for importing the CAD model. Play around with the Toggle Button's properties to your heart's content.

Importing Data from ThingWorx

There are several widgets available for displaying values. You can use 3D labels and gauges, or 2D gauges and value displays. Once you've selected your widget, you'll need to connect the value to the ThingWorx Data. On the right-hand side, there is a tab called "External Data". Click on the plus sign and you'll be taken to a search engine as shown below.

1. Use the top left search engine to find your Thing by its name.
2. Click the blue right arrow. 
3. In "Filter Services", type "GetPropertyValues" and click the blue plus symbol when it pops up.  It should now be added to "External Data".
4. Close out of the search window.
5. Expand "External Data". 
6. Expand "Configurations".
7. Select "Invoke on Startup", and "Auto-Refresh".
8. Change the Auto-Refresh rate to something that fits your project (I used 1 second).
9. Go to "Current Selected Item" to get a list of all your available properties. 
10. Drag the blue cross icon from the desired property over to the blue circle beside "Value" on your 2D/3D widget. 

This will cause the widget to show the value of the property from the ThingWorx Foundation platform. Repeat the last two steps for all of your widgets. 

Publish Your Experience

In order to save and publish your experience, you'll need to associate a ThingMark to your experience.   

1. Under 3D view, drag a ThingMark widget (under "Views" in the left hand column) into your environment.
2. Under "Application Parameters", drag the blue cross icon next to ThingMark to the blue circle next to ThingMark under the Properties  column for your ThingMark widget. 
3. Change the number to match the number of the ThingMark you'll be using for the experience (it'll be the name of the file in your ThingMark folder given to you with the pilot program).
4. Adjust the ThingMark widget's location in your 3D canvas so that it is in the same manner it will be placed in real life.

Now go to "Experience" under Configurations. Give your experience a name in the "Title" box and retype your ThingMark number from before.

Now that everything is set up, it's time to save your experience and publish it. Click on "Save" in the top left corner, then click on "Publish". Once it has successfully published, your AR experience is ready to be viewed on a smartphone or tablet with the Vuforia View Enterprise application downloaded.

Vuforia View Enterprise 

Almost finished! The last step is to download Vuforia View Enterprise onto your smartphone or tablet. Once you have it downloaded, open up the application, go to settings, and type your service URL. Now print the ThingMark that you used in Vuforia Studio and scan it with your smartphone or tablet. Open up your experience and enjoy your new augmented reality!        

Learn more about DMC's Custom Hardware and Software development.