VR and AR technology is a rapidly growing platform for many reasons. Leveraging the ability to provide an immersive experience to its users, the technology helps users become actively engaged in its content. By marrying the real and digital worlds, AR can alleviate some of the shortcomings of traditional forms of education and deliver lessons to its users in an engaging and effective manner.

However, cost is a significant disadvantage for VR and AR today. Many VR and AR experiences like the HTC Vive or Microsoft HoloLens require expensive devices and supporting equipment like high performance computers. Yet, the experiences that users of these devices have access to can be quite revolutionary. Google Tilt Brush is one such application that has many uses from design and creativity to art and architecture, but requires expensive VR rigs and accompanying accessories.

Our application, SketchAR is an introductory 3D sketching tool built for K-12 classrooms. Designed with a simple interface and compatible with Google Cardboard, SketchAR aims to minimize the user learning curve and make 3D drawing experiences available to all. SketchAR can be deployed on any smartphones and does not require a specific VR Kit, making the tool low-cost and accessible.

Functionality

Requirements

• A compatible smartphone (we have implemented this using an iPhone)

• Smartphone compatible VR kit (i.e. Google Cardboard)

• Ground Image Target (Stones)

• Wand Image Target (Wood Chips)

• Palette Image Target (Tarmac)

Set up

The user should place the ground image target where the user wants to draw. The palette image target should be placed within the user's reach and the wand image target should be mounted onto a small 250mL drink can (e.g. a Red Bull can).

Directions of use

The wand is an isotonic device that is used to carry out tasks. The user will use this object to interact with the virtual menu and select one of four modes: Sketch, Navigation, Save Viewpoints and Help. To select a mode, the user must collide the wand with the virtual object corresponding to the mode the user wants to select.

The virtual menu is composed of:

• Sketch  

• Navigation

• Save Viewpoints

• Help Mode

The Sketch mode’s menu provides the user the ability to draw, change colors of their brush and erase parts of their drawings. To enable the wand to serve as a brush, the user must press down the magnet select button in the Google Cardboard. To erase parts of the drawing, the user must place the wand on the eraser button then touch any part of the drawing that the user wish to erase.

Image: Arrow points to the magnet select button on the Google Cardboard
 

The Navigation mode’s menu allows the user to zoom in and out of their sketches and change views. The users can have both an exocentric and egocentric view of their drawings, and move around the drawing in all directions so as to be able to reach distant areas that they may be constrained to reach physically.

With the Save Viewpoints mode’s menu options, users can save up to nine viewpoints that users can revert to. This option gives a user further flexibility and ease to create drawings in areas that aren’t within the user’s reach.

The Help mode displays instructions on the palette, informing the user on how to use SketchAR.

With the menu and palette that is displayed onto the palette image target, users can begin virtually sketching in their physical environment.  

Design Implications

SketchAR has been designed keeping in mind Jakob Nielsen’s 10 general principles for interaction design, also commonly known as the usability heuristics.

Visibility of system status

Visual feedback is important for users to be aware of how their actions affect the system and SketchAR incorporates several features to display an accurate system status to users.

The user’s wand image target displays a cube above the object when it is correctly tracked. When the user uses the wand to select a mode by colliding the wand with the object representing the modes, UI elements appear that correspond to the features of that mode.

The cube changes color, depending on which color has been selected in paint mode. If the eraser has been selected, the cube displays the same pink material as the eraser. This helps keep the user informed of what mode the paint brush/wand is in.

Another detail implemented is the noticeable enlargement of icons as the wand touches them. This tells the user that the system has registered that their wand is selecting a button. Therefore, the user knows that their selection has not been registered if the button does not become larger while the cube on the wand is positioned on it.

In save mode, the buttons that are not active are greyed out. This signifies that nothing will happen if the user touches his/her wand to it. The buttons that are active are colored in a bright blue, signalling that they contain a saved scene.

The navigational map placed in the bottom right corner shows users their progress by presenting an orthographic view from above, akin to a God mode in games. Users can keep track of their progress and use the map to orient themselves within the system.

Match between system and the real world

The SketchAR user interface is straightforward to interact with since the 3D objects that serve as its UI elements have real world context. An artist’s palette represents the Sketch mode that allows users to create sketches/paintings, a camera represents the Navigation mode that allows the user to view and navigate around the entire sketch, and a floppy disk represents the mode to save viewpoints. Furthermore, in paint mode, where the user touches the wand to a block of color mimics the gesture of dipping a brush into paint on a palette.

Additionally, clear labels describe the functionality of various features. For example, the saved viewpoint buttons are clearly labelled with numbers, so that users know how to select the first scene they saved, the second scene they saved, and so on. The eraser has the label ‘ERASE’ and is colored pink, just like many erasers that our users will be familiar with. The magnifying glasses have ‘+’ or ‘-’ signs, for zoom in and zoom out functions, in a similar way that cameras use these symbols for zooming in and zooming out.

User control and freedom

The ability to change colors and erase parts of the sketch gives the user control and freedom over his/her drawing and allows the user to make mistakes. Also the navigation mode allows the users to zoom in and out of the sketch and switch viewpoints. This gives the users to freedom to switch between an exocentric view of the drawing and various egocentric viewpoints. Also, this allows users to work on details of the drawing, since they can zoom in close to it. In addition, the user is able to choose which viewpoints to save so that the users can efficiently and effectively traverse around their sketch, and work on different parts of the drawing.

Consistency and standards

SketchAR incorporates a consistent flow when switching up a task. First the three buttons on the left of the menu that list the different modes that when selected  display the associated buttons on the right side of the menu.

All of the blocks of color do the same thing; change the color of the wand to the color they display. All of the arrows in navigation mode do the same thing; they allow the user to move around their drawing in the direction they display. The magnifying glasses allow the user to move in or out of the drawing depending on whether the magnifying glass displays an ‘+’ or ‘-’ sign. This use of consistency allows the user to know what he/she is getting into when he/she selects a button.

Error prevention

SketchAR makes use of color and error messages to prevent errors and let the user know when errors have occurred.

In saving viewpoints mode, the buttons that are greyed out signify that no scene exists when you touch the button. In addition, if the user touches one of these buttons, an error message will pop up, telling the user that no scene exists there. The buttons that are active and hold a saved scene are a brightly colored blue, which intuitively tells the user that they are active.

In navigation mode, if the user zooms in too much, the drawing will take up the whole screen and block the image targets and palette target, preventing him/her from drawing anything else. Additionally, if the user zooms out too much, the drawing will disappear from view. To prevent this, maximum and minimum zoom limits are implemented. If the user tries to exceed these limits, an error message pops up, telling the user that they have reached these zoom limits.

Recognition rather than recall

The user’s palette is always available to the user and contains all the basic tools needed to run SketchAR. In sketching/painting mode, the most basic instruction needed to draw is located on the palette itself, and it tells users to press down on their Cardboard buttons to start drawing, so they don’t need to rely on their memory in order to use the application. In Sketch mode, the color on the cube at the end of the wand indicates the active color so the user does not have to keep track of which color has been chosen.

The saved viewpoints mode minimizes user memory load, allowing the users to not have to remember information from one part of the drawing to another. They know that the ‘1’ button signifies the first scene they saved, the ‘2’ button signifies the second scene they saved, and so on. To maximize recognition, SketchAR provides nine different scenes that the users can save, allowing them to switch between viewpoints.

In navigation mode, users can recognize that an arrow pointed toward the left, right, up or down means moving around the object in that direction. Additionally, the users will recognize that the ‘+’ and ‘-’ magnifying glasses will mean moving in and moving out of the drawing respectively.

Flexibility and efficiency of use

We followed the user’s mental model and organized the menu in modes. All tasks relevant to drawing is on one menu, those related to navigation is on another and those related to saving is in another. Hence, the user can change tasks efficiently and complete all grouped tasks without having to switch to different menus or run into discoverability issues. Having different modes also prevents the palette from being cluttered with too many buttons at a time, which can slow down the user as he/she will need to take more time to process the buttons.

Another feature that helps with the flexibility and efficiency of the software is the ability to save scenes and revert to different rotation of the drawing. This not only helps the users to speed up the process of manipulating different parts of the 3D drawing but also helps users to recognize areas that are not in of their field of view.

Aesthetic and minimalist design

The information that the user needs to process SketchAR is organized based on the user mental model. The image target on which the drawing is made is different to the image target that the palette is on, separating the tools from the creation. We ensured that the palette is never cluttered with too many buttons at a time, by separating functions out into four modes. This allows there to be a wide range of functionalities without over-complicating the interface. Our use of a handful of colors makes our application look appealing, without looking gaudy.

Additionally, our interface is minimalistic, as the palette can always be moved out of the way, so the user can admire his/her creations in peace.

Help users recognize, diagnose, and recover from errors

SketchAR includes a reset button that allows uses to revert back to the initial state. This allows the users to be able to have a clean start. A hard reset is a quick fix if a user recognises a significant error and would prefer to diagnose this by erasing the drawing. Additionally, this enables users to continue running the application and not have to exit and manually restart, which can break a user’s continued engagement with the application.

Help and documentation

When a user first starts using SketchAR, they will need to know how to start drawing. This instruction is included on the palette in paint mode, which is the default mode that the app loads with. The instruction tells the user to press the button on their Cardboard headset to start making brushstrokes. Since this instruction is small and unobtrusive, it does not interfere with or slow down more experienced users making creations.

SketchAR has a help mode that displays instructions for using the application on the palette. It is organized by modes/tasks. Since there is a whole separate mode containing all the instructions the user could ever need to use SketchAR, the user can always refer back to it, in the rare case that our carefully designed application fails to guide him/her.

Future Implications

We hope to further expand of this project to make the user interaction more seamless and aesthetically pleasing. For future implementations, we hope to see educational materials being implemented into the software such as a coloring book for preschoolers or teaching structures of atoms in chemistry class.