Building CMS Backed Augmented Reality Apps with CrafterCMS

From SnapChat to Pokemon Go to the helmets used by F-35 fighter pilots, Augmented Reality (AR) applications are gaining a ton of traction. Unlike virtual reality which attempts to completely replace actual reality with an alternative universe and sensory experience, augmented reality, leverages technology to add a sensory (typically visual) layer of interaction and data on top of our everyday experiences. Augmented reality is so exciting and opens up so many new opportunities because it enhances our physical experience, giving us a way to experience things in a way we never could before.

As with most applications today, AR applications frequently contain a lot of content and related digital media such as 3D models, images, and videos.  Often times textual/markup content and digital media are deployed statically along with the application. This approach creates unwanted dependencies for content updates on the developers and requires laborious deployment processes.

In a business context, this misalignment and lack of independence can slow activities way down and ultimately impact the bottom line.  To address this common problem, developers often build custom applications for non-technical users or leverage content management system (CMS) technology.  Building custom authoring tools double development time and creates an additional support burden on your development team. Leveraging a CMS is a great idea, but you need to make the right choice.  Many CMS architectures impose severe limitations on markup, application and API architecture and design. As we’ll see in this article, CrafterCMS provides you with the tools and flexibility you need to create innovative, rich experiences like Augmented Reality.

In this article, we’ll build a simple (but not too trivial) AR application. After that, we’ll update it so that non-technical users can upload new models and edit and deploy/publish changes to the scene through CrafterCMS without any technical background.

For our application, we’ll start with a simple AR application based on open source frameworks AR.js, ARToolkit, and THREE.js.  Given that this is a tutorial and we want to have some fun with it, we’ll use (USC 17 107) some freely available 3D Models from (http://roestudios.co.uk/project/3d-pokemon-models.) You may also use any other models as long as they can be loaded by THREE.js’s library of loaders.  Free models vary in availability and quality. I chose this library because of the number of available models.

Step 1: Get a Simple, Static Browser-based Augmented Reality Application Running

Before we put anything in the CMS, let’s first make a basic static AR example application work.  We won’t get into the nuts and bolts of the various AR and 3D web related libraries.  There is a lot of material and examples already available on the subject.  Instead, we’ll try to avoid a lot of abstraction and construct an example that is simple enough in what it does so that the code can be readily understood.  You can find the code here on my Github account.

• Clone or download this code as a zip
• Open code/simple-ar.html with a browser

In your browser, you should see a video of yourself (captured by  your camera if it’s available and you have authorized it to be used by the browser.)  Additionally, hovering on screen, there should be a 3D model of Pikachu.

Now that we have a VERY SIMPLE working example of an AR application working we can get started integrating content management capabilities with our CMS so that anyone can add new 3D models to the application.

Step 2: Get CrafterCMS Downloaded and Installed

The first step is to get the CMS downloaded and installed. CrafterCMS is free and open source software licensed under GPL v3.

2.1 Download the CrafterCMS bundle

• Download CrafterCMS here: https://craftercms.com/download
• Simply click the Download button.

2.2 Install CrafterCMS and start it up

• Unzip the download
• Execute startup.sh (or startup.bat if windows) inside the unzipped folder
  • Prerequisite: Java 8 JDK is installed, and JAVA_HOME is defined as an environment variable
• Open a browser to http://localhost:8080/studio. I recommend using Chrome. Chrome has the best support for desktop file drag and drop of all the browsers which will come into play later in this article.
  
  
• Login with username ‘admin‘ and password ‘admin.’
  
  

Step 3: Install the Application as a Project in the CMS

We now have the prerequisites done, and we’re ready to party! Let’s install the application into the CMS.

3.1 Create and setup a CMS project to manage your application

• In Step 1.2 you logged into the CMS as the admin user who has an admin role. On the screen, you will see a button on the left hand that says “Create Site.” Locate and click this button.
  
  
• Enter AR Tutorial as the application name.  CrafterCMS will generate a Site ID for you.
• Select “Empty” for your blueprint and click the Create button.  CrafterCMS will create a new project for your AR Tutorial application with a basic “Hello World” type configuration.
  
  

3.2 Upload your application to the CMS

• Click on Site Content in the menu at the top of the screen. This will open a panel on the left
• Click on the Static Assets label this will open the folder.
• Right click on the static-assets folder and choose Bulk Upload Assets
  
  
• In your code folder from step 2, you will find a static-assets folder.  Drag the contents of this folder onto the upload dialog in the browser. When it’s done, you will be able to clickDone.
  
  
• By uploading the compiled assets, we’ve loaded the app into the CMS.  Now we have to tell the CMS to render our single page.  To do this, we must update our out-of-the-box template provided to us by the blueprint.  In CrafterCMS, templates are coded in Freemarker and can contain any kind of markup (typically HTML 5) as well as Freemarker Directives that make it dynamic.
  

  3.3 Update the site template with the content from the HTML page

• To update the template: click “Site Content” in the menu at the top to hide the Site Content panel.   Then locate the Wrench in the upper right-hand corner on the toolbar.  Clickthe Wrench.  This will open the Preview Tools panel on the right-hand side of the screen.  
  
  
• Now click on In-context Editing to expand that specific tool.  You will notice this tool allows you to edit content, templates, and controllers.  We’re interested in the templates.  Click Edit Template.
  
  
• Replace the code in this template with the HTML found in your build directory at code/simple-ar.html.
  
  
• Note that the static code example assumes the application assets are at ./static-assets on the server.  In CrafterCMS, static assets like CSS, Java Scripts and images are served from /static-assets.  You will want to modify the paths (remove the ‘.’) in the HTML template to use the /static-assets folder as shown above.  Once done, click update.
  
• Your app is now running in the CMS!  Congrats!   With this complete, we can start working on making the application dynamic, editable by a non-technical user and updatable at any time without a deployment.

Step 4: Content enable the application

The first step in content enabling the application is determining what the content model is.  A content model is a definition and structure of the content.  We maintain the definition of the content structures within CrafterCMS by declaring Content Types.

To figure out what your Content Types should be, look for the NOUNS in the application that you want a non-technical user to be able TO easily update and publish at any time.  In our case, this is an individual 3D model of a Pokemon like Pikachu.

While the model in this simple application will be very simple, CrafterCMS is capable of modeling very sophisticated objects as well as relationships between objects to handle real world scenarios.

Before getting into the CMS, let’s figure out what the definition for our Pokemon type should be.  We do this by looking at the application and seeing what fields the app will require. Because this particular type of content is a 3D asset rather than traditional digital media, we need to understand a little about what goes into making a model and how it’s described. We’ll start by looking at the two models in the model directory.  By examining both the pikachu and snorlax model directories we can see that a model can contain a different number of files.  Complex models may contain a lot of files.

We want the act of adding these models to the CMS to be really easy.  Other than adding the files to the CMS there is nothing to configure about them. Further, a folder is a good grouping mechanism for keeping the files for a specific model together. Given this, we’ll assume that the first step for the author in adding a new model is to simply upload it by dragging the model folder into the static-assets model directory via bulk upload.  In other words, for the 3D models themselves, we’ll leverage folder structure and IA rather than Content Model to store and organize the raw 3D content assets.

The next thing to look at is how the model is loaded into, positioned and scaled in the world.  For this, we need to look at the code.  Specifically, the code that loads the 3D model:

By examining this we see there are a number of important attributes:

• We also see that every model has a file that defines its structure (.obj)
• And a file that defines how to apply its textures (.mtl)
• The Model Scale Multiplier
• The Object Rotation on the X axis
• The Object Position on the X axis
• The Object Position on the Y axis

If we think about it, some of these are directly related to the 3D model itself and belong to the Pokemon content model.  Other attributes like scale and position are used to place the model in the world.  It’s more flexible if we manage the scale and positioning attributes outside of the object.

OK! We have a list of fields we need for our Pokemon, let’s get started defining the Content Type in the CMS:

4.1 Declare a Pokemon 3D Model Content Type

• Open the Site Content panel.
• Locate and click the menu item for Admin Console. This will take you to the power user area in the CMS used to manage and perform various site administration tasks.
• On the left panel in the Admin Console, locate and click the menu option for Content Types. This tool is used to create new Content Types and edit existing ones.
• On the toolbar at the top of your screen, you will notice two options: Open Existing Type and Create New Type. Because a Pokemon is a new concept you want to introduce and define in the CMS, click on Create New Type. This action will pop up a dialog to prompt you for some basic information about the content type.
• Enter Pokemon for the display label. CrafterCMS will suggest a content type name automatically for you.
• The Type you want to choose is Component. In CrafterCMS anything that you are modeling which is NOT a page is a Component. The main difference in the CMS between pages and components is that page objects are automatically URL addressable while components are not.
  
• Once the dialog is configured, click Create. This will open the content type editor for your Pokemon Content Type. You can learn more about the content type editor in detail at https://docs.craftercms.org/current/by-role/developer/common/content-modeling/content-modeling.html
  
• The Content Type Editor lets you drag/drop editing field types from the right-hand controls list into your Content Type definition. You then click on the individual controls and configure them with Labels, properties, data sources, and constraints.
• For each field, you need to add (Object File, Texture file, etc.), look through available controls list on the right, find the appropriate editing control for that content and drag/drop it into the Pokemon properties area.
• After you add a field to the Content Type, click on it to configure it in the Properties Explorer.
  • Add a Title. This is the user-facing name the author will see alongside the input control.
  • CrafterCMS will automatically suggest a variable name which you can use later in templates and controllers to access the content values.
  • Scroll down in the Properties Explorer to see all the properties and constraints for the given field. This is how you mark fields as required, change their input length and make other adjustments.
    
• Because you want the user to be able to pick the object and texture fields, you will need to give them a file picker control.  In CrafterCMS this is called the Item Selector control. Drag one of these on the screen as shown above and give it a title of “Object File.”
• Now you need to tell the Item Selector where to find the content it’s going to allow the user to pick from (remember the authors will upload models into the /static-assets/model folder via drag and drop.)  To wire this up, you will need a Data Source. In CrafterCMS the way picker type controls get their values is via a Data Source.  There are several out of the box Data Sources, and it’s possible to define your own as well.  For our example, because we want to pick an existing file in the repo we’re going to use the data source called File Browse.  Scroll down within the Data Sources area until you see it then drag/drop it onto the Data Sources portion of the Content Type definition.
  Click on the added Data Source and begin to configure it as shown above.  Note that we’re simply identifying a path elsewhere in the CMS where we want to pick a file from.
• Once our Model Files Data Source is defined, let’s connect it to our Item Selector control for Object Files.  Click on the Object Files control and configure it in the property editor as shown below:
  
• Add another field for Text Files and connect it to the same data source.
• Add an input control for X Rotation and configure it as shown below:
• Perfect! Your Content Type definition is now complete.  Click Save at the bottom of the screen.Note that you are going to get a warning that the content type does not have a template associated with it. That’s ok.  Not every Content Type needs a template.  In our case, the app already knows how to render the content, and we don’t need another template.  Click Save on the dialog.  You’ll notice a small save indicator on successful save.

4.2 Create Pokemon Content Objects

Now it’s time to create some test content.  To return to your project Click the Crafter Logo on the toolbar in the top left corner of the screen.  This takes you to your workflow dashboard.  From here you can create content or navigate back to your application preview via the Site Content panel.

• Click on Site Content in the menu at the top of the screen. This will open a panel on the left
• Click on the Components label this will open the folder.
• Right click on components folder and Create Folder.
  
• Provide the folder name “pokemon”.
  
• By clicking Create, you will have added a pokemon folder to the components folder.  This is simply to help organize your content.
• Right click on the pokemon folder and Create Content.
• The CMS will prompt you for the type of content you want to create.  You will note the Pokemon option.  Choose Pokemon and click “Ok.”
• Once you click OK, you will be given the entry form for your Pokemon which you can start filling out.  Awesome! CrafterCMS automatically creates your Pokemon editing interface based on the Content Type Definition.
  
• Repeat this process as many times as you want, creating a new Pokemon each time. When you save the form, you will see the Pokemon item listed under the pokemon folder as whatever value you entered into the Internal Name field (which is only used by the CMS editing tools as a label in the tool.)
  • Note:  Included an X Access Rotation in the model object because based on the available models the models all start facing in different positions.  The same is true for scale.  You may wish to add a “normalizing” scale value in addition to the rotation.

Step 5: Configure the AR Page Controller to Show a Pokemon Loaded from the CMS

Now that you have content objects in the system it’s time to provide the CMS with the logic to determine which items to show.  To start, we’ll keep things simple.  We’ll provide the template with a static list of models to display.

5.1 Update the Controller

The controller contains the business logic.  In our case, the controller is for a “Page.”  In CrafterCMS you can have controllers for pages, components, services, jobs and other kinds of objects as well.  To get to the page controller:

• Locate the Wrench in the upper right-hand corner on the toolbar.  Click the Wrench.  This will open the Preview Tools panel on the right-hand side of the screen.
• Now click In-Context Editing to open the In-Context Editing tools.
• Now click Edit Controller.
• Now add the following code:

def pokemonModelIds = [ “/site/components/pokemon/snorlax.xml”]

templateModel.pokemonInstances = []

pokemonModelIds.each { id ->
def pokemonInstance = [:]
def modelItemId = id

pokemonInstance.model = siteItemService.getSiteItem(modelItemId)
pokemonInstance.xPosition = -2
pokemonInstance.yPosition = -2
pokemonInstance.zPosition = -2
pokemonInstance.scaleMultiplier = 1

templateModel.pokemonInstances.add(pokemonInstance)
}

• Now click update to save the controller.

The changes we made to the controller have added a new variable called pokemonInstances to our template model.

5.2 Update the Template

Now that our template has pokemonInstances available to it, we have to update it to take advantage of the values in that data structure.  We’re going to iterate over the items in the array and call the addModelToScene function for each item.

• Locate the Wrench in the upper right-hand corner on the toolbar.  Click the Wrench.  This will open the Preview Tools panel on the right-hand side of the screen.
• Now click In-Context Editing to open the In-Context Editing tools.
• Now click Edit Template.
• Find line 99 and make the following updates

<#list pokemonInstances as pokemonInstance>

<#assign texturePath = pokemonInstance.model.textureFile.item[0].key />
<#assign objectPath = pokemonInstance.model.objectFile.item[0].key />

loadModelIntoScene({
basePath: “${texturePath?substring(0, texturePath?last_index_of(‘/’)+1)}”,
mtlPath: “${texturePath?substring(texturePath?last_index_of(‘/’) + 1)}”,
objPath: “${objectPath}”,
scaleMultiplier: ${pokemonInstance.scaleMultiplier},
xPosition: ${pokemonInstance.xPosition},
yPosition: ${pokemonInstance.yPosition},
xRotation: ${pokemonInstance.model.xRotation}}, scene);
</#list>

• Now click update to save the controller.

AWESOME! Our template now renders models that we add through the CMS.

The only issue is, adding a new model to the scene requires us to edit the controller.  This is the job of a developer.  In essence, we’ve moved the responsibility of deciding what models to render from the Javascript to the groovy code, but we haven’t put the power of deciding what to show in the hands of the authors yet.  So let’s do that.

Step 6. Give the Author the Ability to Add Models to the Scene

6.1 Update the Scene Content Model

To give the author, a non-technical user, the ability to update the scene, we have to first update our content model.  In this project, the content model that controls the scene is called Entry.  To edit the Entry Content Model:

• Open the Site Content panel.
• Locate and click the menu item for Admin Console. This will take you to power user area in the CMS used to manage and perform various site administration tasks.
• On the left panel in the Admin Console, locate and click the menu option for Content Types.
• Now click Open Existing Types and select Entry.
• Click the Body Content Field.
• Now click the dark X in the upper right-hand corner to remove the Body Content field.  It’s not used.
  
  
• Drag a Repeat Group object on to the Page Settings section.
• Configure the Repeat Group with a title of Models
• Then Drag a Node Selector control into the repeat group.
• Configure the node selector with the title Model
• Drag a Child Content data source onto the Data Sources area and configure it as shown below.
  
  
• Wire/Configure the Models Node Selector control to use the Models data source by clicking on the Model control and configuring the Item Manager property in the property sheet on the right.
  
  
• Add additional fields for X Position, Y Position, Z Position and Scale Multiplier as shown below.
  • Note the use of the input control and the configuration in the properties panel on the right.
    
    

Once you have completed this, you can click edit on the main toolbar whenever you are looking at Home scene in Preview, and you will be able to, through a simple user interface, add models to the scene.

• Click the Crafter Logo in the upper left corner.
• Click on Site Content in the menu at the top of the screen. This will open a panel on the left.
• Open Pages and then click Home to load the Preview
• Now click Edit in the toolbar at the top.
  
  
• Click Add on Models to add your first model.
• Now click Add on the Model element and choose “Browse for Existing.”
  
  

This will open a browser to the pokemon folder where you have been creating Pokemon content objects.  You can click Add & Close to select an item and add it to a model in the scene.

Repeat this process as many times as you need to add the items you want in your scene.

• Note you want to alter the X Position for each model that you add so that they do not overlap when they render.

6.2 Update the Scene Controller to Use the Scene Content Model

Now that we’ve modified the Scene content model and we’ve updated our content accordingly, we need to tell our controller to look at it and populate the template variables accordingly.

• Locate the Wrench in the upper right-hand corner on the toolbar.  Click the Wrench.  This will open the Preview Tools panel on the right-hand side of the screen.
• Now click In-Context Editing to open the In-Context Editing tools.
• Now click Edit Controller.
• Now Update the code with following:
  

  def pokemonModelIds = contentModel.get(“models/item”)
  templateModel.pokemonInstances = []
  pokemonModelIds.each { item ->
  def pokemonInstance = [:]
  def modelItemId = item.get(“model/item/key”).text

  pokemonInstance.model = siteItemService.getSiteItem(modelItemId)
  pokemonInstance.xPosition = item.get(“xPosition”).text
  pokemonInstance.yPosition = item.get(“yPosition”).text
  pokemonInstance.zPosition = item.get(“zPosition”).text
  pokemonInstance.scaleMultiplier = item.get(“scaleMultiplier”).text

  templateModel.pokemonInstances.add(pokemonInstance)
  }
  
  
  

• Click Update to save your work.

6.3 Update the Scene Template to Use the Scene Content Model

Now that our template has pokemonInstances updated to include the values updated by the author via the content model, we have to update the template to use the new values.

• Locate the Wrench in the upper right-hand corner on the toolbar.  Click the Wrench.  This will open the Preview Tools panel on the right-hand side of the screen.
• Now click In-Context Editing to open the In-Context Editing tools.
• Now click Edit Template.
• Find line 99 and make the following updatesloadModelIntoScene({
  basePath: “${texturePath?substring(0, texturePath?last_index_of(‘/’)+1)}”,
  mtlPath: “${texturePath?substring(texturePath?last_index_of(‘/’) + 1)}”,
  objPath: “${objectPath}”,
  scaleMultiplier: ${pokemonInstance.scaleMultiplier},
  xPosition: ${pokemonInstance.xPosition},
  yPosition: ${pokemonInstance.yPosition},
  xRotation: ${pokemonInstance.model.xRotation}}, scene);
  
  

• Click Update to save your work.

Step: 7 Play Around!

Congratulations! Now anyone with access to the CMS can quickly and easily add new models and configure it into the scene.  Download additional 3D models and play around with them!

Taking It Further

Our example is very basic.  We’re simply adding user configured 3D Models on top of a live video feed. This is just the start!

Real world AR use cases typically include animation, interaction and use visual triggers like QR codes, GEO Location or pure play vision to place the augmentation into the scene.  We excluded these additional capabilities from the article because they obscure the main point of the article which is making the scene configurable by non-technical users. That said, adding triggers is very easy to do. In fact, the CMS makes triggers even more powerful by giving you the tools to understand what device the user is using, where they are (geolocation) along with other contextual information.

Further, you have the full personalization capability of the CMS to help you make your AR experiences dynamic and personalized. How awesome is that?!

Conclusion

Web-based augmented reality frameworks that help developers build AR experiences are making it easier than ever for companies to leverage AR as a way to communicate, educate, entertain and assist users in a rich fashion not previously achievable without huge investments.

Today most AR work is done exclusively by technical users, and the experiences are statically designed and deployed.  This is extremely limiting.  It significantly reduces the value of the development, makes it challenging and costly to update and significantly reduces the RoI on the effort. Smart developers will separate code from content so that non-technical users can quickly and easily update the application’s content without involvement from technical resources for code updates or deployments.

Integrating your CMS with your AR is a strong pattern for solving this problem — but architects beware, not all CMS platforms are the same.  The wrong CMS will interfere with the development process and force interface and implementation details.  CrafterCMS, a modern, open source, Java-based CMS, offers the right architecture, technology, and flexibility for the job. CrafterCMS integrates with your development process and tools and puts your app in control of the interfaces and implementation code while providing content authors with the tools they need to update and publish the application independently.