Understanding AR
This lesson is about augmented reality, its possibilities, what to think of when creating a digital piece and at what point you should bring in an expert. We will not go deeply into the technical aspects of how AR works, there are plenty of information about that on other websites. The focus in this lesson is to give a basic understanding, and some inspiration on how stage performers can use this technology and what to think of when doing so.
What is AR?
Augmented Reality (AR) is a technology that enhances a user’s perception of the real world by superimposing digital information, such as images, videos, and 3D models, onto the user’s view of the real world. This technology can be experienced through various devices such as smartphones, tablets, and smart glasses.
Example of AR
One example of AR in action is through mobile apps that allow users to point their device’s camera at a physical object or location, and then overlay digital information or interactive elements on top of the live camera feed. Another example is through smart glasses which allows users to see digital information and interact with it as if it were really there in front of them.
Understanding different types of AR
There is massive creative potential for producing artistic real-time interactions using AR technology. However, it can be a bit confusing when you are just starting to dip your toes in these waters. There are different types of AR. Which one to chose depends on what type of experience you want to create. Your budget is also a factor to consider.
There are two different types of AR
- Marker-based AR
- Markerless AR
With marker-based AR you need to scan a visual marker to activate the digital imagery. The most common markers are target images such as QR codes. A device’s camera, on a smart phone or a tablet, reads the code and translates it into an augmented experience. A marker can also be an object existing in the real environment. It is the marker that decides where the app will display the content in the camera feed or in the user’s field of view.
In markerless AR you don’t need to read an image pattern, or object to generate a digital effect. There are a four varieties of markerless AR:
Location-based/geo-based AR
There is no need for any physical markers in order for location-based augmented reality to work. Location-based technology uses pre-defined geographical points, and sensors of a smart device (camera, gps, accelerometer, compass) to place virtual objects at a set location. Using a smart device, viewers looks around at geographical locations to see or interact with different content. For example: Video, 3D data augmentations, audio or links. An example of this technology is Pokémon GO.
Projection-based AR
Projection-based AR overlays a digital projection – static och dynamic – on, for example; people, buildings, stages or other things in the real world. It can pretty much display a digital projection on any surface. To work with this type of AR you need a projector, mapping software, a computer, and of course image/video content to be mapped on your surface. Viewers don’t need a smart device to see this type of AR.
Overlay AR
An overlay AR enhances a real world object, image or room with additional 3D digital content on top of that item. It gives the viewer several perspectives visually. An example could be a real-world performance that offers the audience additional digital visuals on a scen, or about the costumes that the actors are wearing, or more about the theme. It’s virtual content that the audience can interact with via a smart device’s camera.
Contour-based AR
Contour-based AR is a technology that uses computer vision to track and understand the contours, or shape, of objects in the real world. It can then use this information to overlay digital information or images on top of the real-world objects, creating an augmented reality experience. You can find examples of its use in 3D map viewing to enhance the visualization of topographic information. It is also used to promote safe driving by providing drivers with additional information and visual cues about their surroundings, helping them to make better decisions and avoid accidents.
Artistic possibilities and pitfalls in AR
Augmented reality in stage art
Augmented Reality (AR) can be used in stage art to enhance the audience’s experience by adding digital elements to the performance. This can include visual effects such as projected images, animations, and special effects, as well as sound effects and other audio enhancements. AR can also be used to create interactive elements that allow the audience to participate in the performance in new ways. For example, a stage production could use AR to allow audience members to use their smartphones to interact with the performance in real-time, or to view additional information about the production. Additionally, AR can be used to create virtual sets and environments, which can be used to enhance the overall look and feel of the stage production.
5 tips for stage artist creating augmented reality
- Plan the overall AR experience: Before starting to work on the AR aspect of the performance, it’s important to have a clear idea of what you want to achieve. This should include the overall design, the type of AR effects you want to use, and how they will be integrated into the performance.
- Use a reliable AR technology: Choose a robust AR technology that can handle the demands of a live performance, such as real-time tracking, image recognition and high-quality graphics. Make sure to test the technology before the performance.
- Consider the audience’s perspective: Keep in mind that the audience will be viewing the AR effects through their own device and consider the optimal viewing distance, angle, and lighting conditions.
- Practice and rehearse: As with any stage performance, practice is key to ensure that the AR effects are seamlessly integrated into the performance and that any technical issues are ironed out before the performance.
- Have a backup plan: As with any live performance, things can go wrong. Make sure to have a backup plan in case of technical difficulties, such as a backup AR system, or an alternative way of delivering the performance.
Working with experts
Search for augmented reality works that inspires you
Before you get in contact with a professional AR developer it’s a good idea to find AR projects and ideas that you like. Check out Snapchat, TikTok, YouTube and other places to find out who made the ones you are attracted to. Use this information to find the right artist/developer for your project. Snapchat, for example, is a strong augmented reality platform that has directories of artists that are working with Snap-lens filters. A directory is also available on 8thwall, which is a big AR company doing web AR. Many companies and platforms have directories where you can find artists to contact.
AR developer/artist on or off site
If you are working with a stage performance you will need someone that can be on-site, if not all the time, then at least the majority of the time. You need that interaction as you develop your creation. In streaming however, due to the nature of the medium, you could stream as you develop. If you want to add augmented pieces to a stream it could be an interesting way of working with your AR artist/developer.
In pure stage art, where the audience is on site, and they are going to have the AR layer there at the live set, it’s best to go with a local AR artist. Or cover their travel expenses if your budget allows for it. For other types of work, you can find a lot of ways where you can work remotely together. It’s really easy to test stuff today. Look at what the person has done before, ask how they work in a team, compare the work to others and find out what type of experience and background the person has. A good idea is to do a small, quick test with the AR artist to get a feel for the person. Because it’s not just about how good they are at their job, it’s also about how well you guys work together. Having a similar passion, or experience will improve the chances of you getting an end product that you are happy with.
Two important tips for your AR planning
- Have a small pre-budget for a very scaled-down test version of your idea. This to test how a collaboration would work and to see what the AR person is capable of.
- If you are happy with the outcome, gear up and start your production.
Common professional titles in AR
- Creative technologist
- Digital artist
- AR developer
What kind of professional you should work with depends on how well you understand the medium and if you have a good idea of the result. Working directly with a developer works well if you have knowledge and can form a mental, as well as a verbal, picture of your piece.
AR developer or AR artist
A developer is more of a technical person and that can help you realize your vision from the technical side. If you want more of a dialogue with another creative, one who can mediate between tech and the arts, a creative technologist or a digital artist would be better.
Some AR projects are very technology heavy and need development – as in people writing code. Other projects are more visual and can use higher order tools, for example Snap-lens filters that are just boxes that you connect. Adobe has their own package that is very visual in the same sense. This means that you don’t always need a developer. If you are new and want someone to be by your side and develop together with, go with a more artistic AR person. They will be able to judge if you need to bring in a developer or not.
Please remember that these are general tips and there are professionals out there that fit well in all of the above-mentioned areas.
Download Checklist for augmented reality (pdf).
Download the Terminology sheet for AR (pdf)
Terminology in augmented reality
Do you find the terminology hard to understand, or did you stumbled across a word that made no sense to you? In the Terminology document (pdf) you will find word commonly used in augmented and virtual reality.
Tips and links to more info
Expert for this lesson
Expert for the augmented reality lesson is Adam Wittsell. He is the Creative Lead & Founder at Superficial Machines (superficialmachines.com). He creates concepts combining storytelling and new technology. Adam is one of the pioneers of 3D-animation in Sweden and he has always been interested in telling stories through new mediums and technology. Adam has led creative and technical teams for more than 15 years in everything from visual fx, commercials, games, advertising and stage shows.
linkedin.com/in/adam-wittsell
