Virtual reality (VR)

Extended reality (XR)

XR is an umbrella term and stands for extended reality. XR includes different technologies such as virtual reality (VR), augmented reality (AR) and mixed reality (MR). In this lesson the we focus on virtual reality.

Virtual Reality (VR) refers to computer generated three-dimensional simulated environments. Scenes and objects appear to be real and depending on the type of VR, the user can play, socialize, engage and create. In VR a user actively has to think that the world they have entered is not real. Because of the immersion they have a sense of stepping inside and interacting with the virtual world.

Why learn about virtual reality?

There are several reasons why you as an actor, dancer, circus or cross-disciplinary artist, may want to learn about virtual reality.

New Opportunities

Virtual reality technology is creating new opportunities for artists to create and perform in new, innovative ways. As the technology continues to evolve, virtual reality will likely become an increasingly important medium for artistic expression, and artists who are familiar with it will be better positioned to take advantage of these new opportunities.

Audience Engagement

Virtual reality can be used to create highly immersive and interactive experiences that can greatly enhance audience engagement. By learning about virtual reality, physical stage artists can develop new ways to connect with their audience and create more compelling performances.

Experimentation and Innovation

Virtual reality allows artists to experiment and innovate with their work in ways that were previously impossible. For example, virtual reality can be used to create interactive performances where the audience can move through and explore different parts of the performance space.

New revenue stream

Virtual reality technology can also open up new revenue streams for artists by allowing them to create and sell virtual reality experiences, such as virtual reality performances, that can be experienced by audiences around the world.

Future of Performances

Virtual reality technology is at the forefront of the entertainment industry, it’s likely that in future, virtual reality will be a major medium for performances and as an artist who understands how to use this technology can be a great asset.

Knowledge is power

Understanding the medium, its potential, as well as its limitations, gives you the knowledge to explain when, where and if a digital production should be created.

Types of VR

Understanding the different types of virtual reality is important because it allows you to make informed decisions about the type of VR experience you want to create or consume, and the type of VR equipment and technology you want to use.

For example, fully-immersive VR typically requires more advanced and expensive equipment, such as a VR headset with high resolution displays and motion tracking sensors.

Non-immersive VR

A user interacts with a virtual environment using, for example, a computer. In the experience they control an avatar (a character) or some activities within that environment. The virtual environment is not interacting directly with the user, it is the user’s avatar that interacts with the environment.

Semi-immersive VR

In this type of VR the experience is something in between non and fully immersive VR. Using a headset that immerses the viewer in an audiovisual experience they can look around, hear and feel like they are in the environment. However, they cannot interact with it. The productions can be a representation of our real world or an imagined one.

Fully immersive VR

In a fully immersive virtual reality it will feel like the person is physically present in the virtual world. The real world is replaced by an artificial one. Things that are happening in this world feel real – it is convincing enough to make the user suspend disbelief and fully engage with the virtual world.

Collaborative VR

A virtual environment in which users from different locations come together and are represented by some form of avatars.

360 videos, 3D rendering, Real-time 3D

There are three categories of 360: real life scene (video), 3D rendering and Real-time 3D gaming. Each can be used singularly or in a mix.

VR – 360 videos

You can film a 360 VR video in monoscopic or stereoscopic format. Monoscopic VR uses a single lens system. In this type of video, you will not get a real feeling of depth. Monoscopic VR is based on one image and can be viewed on any device. Stereoscopic VR has two images, one for each eye, which represents the human eyes. The lenses have, just like our eyes, slightly different angels. This gives a sense of 3D depth in 360 spherical videos. VR headsets have two lenses and are made for stereoscopic viewing. In 360 video the viewer can turn their heads in 3 ways; vertically, horizontally and up/down by turning their head. They can’t move outside the space, as they are locked to the location of the film. Viewers are bound to the 3 degrees of freedom.

Stereo gives depth to a scene. However, it is only effective up to 10 meters. Further than that and it is not so good. So, when shooting with a drone use mono.

• Monoscopic = single lens cameras set up in a circle formation
• Stereoscopic = two lenses in each direction

What you should choose really depends on your project requirements, it is a creative decision. Remember that stereoscopic productions can be quite expensive – camera equipment, production crew and post production.

The process of making a VR video can be split into three parts: planning/pre-production, shooting/production, and editing/post-production.

Pros of 360 video

• Authenticity
• Believability
• Relatively fast and accessible

Cons of 360 video

• Time for post-production
• 3 degrees of freedom
• Shooting limitations: stitch lines /lighting
• Parallax distorsions

3D rendering

Computer generated Imagery (CGI). Various inputs i.e., 3D modeling, sculpting, animations, photogrammetry, laser scanning, motion data. Pre-computed.

Pros of 3D rendering

• Great for reconstructions; architectural, historical …
• No limits for camera movement, no limits at all actually
• Perfectly optically correct result
• Superb visual outcome

Cons of 3D rendering

• Needs HUGE computing power and a matching team of 3D artists
• With few exceptions still 3 degrees of freedom (DoF)

Real-time 3D

It’s like a video game but in 360. Various inputs; modeling, sculpting, photogrammetry, laser scanning. Calculated in real-time using a game engine.

Pros of Real-time 3D

• The only option offering full 6 DoF
• Interactivity
• Deepest level of immersion

Cons of Real-time 3D

• Limited visual quality
• Required processing power
• Require an even more extended team

Best option?

What you should choose depends largely on your creative vision. All are extremely capable tools in the right hands. Some of them offer more options for compromises. They can also be mixed together.

Creating a VR experience

Before you start your VR project

Your first step before starting your VR project is to ask yourself WHY – why do you want to go digital. And specifically, why do you want to use this specific technology? If it is just because you think it is cool, that is not a good answer. You have a responsibility – both to your artistic practice and to the audience – to really think it through. Another question should be: does this idea lend itself better to a digital format than a live one? If it doesn’t, don’t do it.

If you are recreating things that are possible in real life, it is not a reason. Make what you want to do worth it. All mistakes you do will be multiplied by 360. Don’t waste your time creating something that doesn’t make any sense, difference or lacks a story.

Space and scale

Most spatial experiences come from perception – visual, auditory, touch, smell, taste. Think broadly.

The biggest promise of VR is that it allows impossible things. For example, visiting a dangerous place or situation that you couldn’t get to or want to take part of in real life. You could get close to wild animals or visit a distant planet. In VR the environment becomes an actor, it is not just a background.

There are no laws of gravity or architecture in VR. It allows you to sit on a chair that is balanced on a blade of grass, to float around in space or be in a completely abstract sphere. There is total creative freedom in what your VR space can be

You can create different effects by placing the camera at different heights. To give a viewer the sense of being one of a group of people, set the camera at chin level.

Scale; as grown-ups we have a sense of scale in relation to our body. In VR we can break that scale. Viewers can be anything from a microscopic bacterium to a giant body in space.

3D audio

Sound is very important in VR. It is key, for great immersiveness, that the virtual world is able to copy the natural positioning of sounds in relation to people and objects in the environment as well as to the position of the user’s head.

Things to keep in mind for VR projects

You shouldn’t listen too much to people saying what you can and cannot do. You have to see what works for you. Here are a few things to keep in mind.

• Give your project a real thought and think about what you want to achieve. Are the rules or guides a requirement for your vision. The industry is still young. Just because someone said something doesn’t make it true. Research and experiment to find a solution.
• Ease the audience into your experience and back to real life again after they have taken part of your virtual world.
• Leave the discovery to the viewer. Instead of telling them outright what to do, guide them and let them find clues on their own. Use the 360 to its full potential. Work with sound, movement, crowds or light to direct them. However, it is hard to predict what a user will do – so test, test, test.
• VR is highly potent and efficient in shaping various emotions. Don’t forget that although it can be fun to generate those feelings in your audience it is also easy to overdo it. And you have a responsibility not to traumatize them. You are responsible for the emotional output from your creation in VR.
• Make sure you collect the right team for your project. You need a wide field of experts when fashioning a VR piece. Be a team player and don’t think you can do everything. Concentrate on the area that you are an expert in and allow the team around you to work with the things they are experts in. Share the load.

Virtual reality and performance art

Not too many years ago, there were few physical artists that worked with VR. Today more are starting to explore this digital arena and imagine new forms of artistic works. As a performer it is valuable to ask what you as an artist will bring to the table. What kind of content you are bringing into this technology. Art is about transportation. People like to be transported and VR is a way to transport them to another space.

People coming from stage performances generally adapt to the VR medium faster than those coming from the movie industry. Possibly because performing artists are used to thinking about space in 3D, whereas movie directors think in terms of a flat screen. Physical performers often work through interactions and improvisations to bring about a storyboard. In cinema, the script and storyboard is usually created first and then interpreted by the actors.

Four steps to create a 360 video

When simplified you can say that there are four phases when creating a 360 video. Here you find a short description of each phase. Of course, there is a lot more to it, but it will give you a basic understanding of the different steps.

Phase 1 – brainstorming and writing

Start from an idea, inspiration or subject and draw, write, discuss and explore it to find and settle on a project to realize. Create a moodboard with a collection of images, objects and text to help frame the idea. The moodboard will help you develop your artistic idea. Next step is to devise a storyboard. This document is used to plan all of the shots, with thumbnails to represent each shot, that will make up the film. The storyboard includes the technical aspect; framing, camera movements, special effects and the artistic aspect; scenography – both virtual and built sets.

How to make a moodboard.

Phase 2: production and research

In the second phase it is time to test your ideas.

The thinking, planning as well as your storyboard is a great start. However, you will encounter surprises during the test phase. Try to keep an open mind and do your best to accept both the positive and negative surprises along the way. When you have created a storyboard that you are happy with, done several run-throughs, tested different scenarios and briefed everyone, you are ready for phase 3.

Phase 3: filming

To create a smooth film shoot, make sure that you are well prepared. Have your storyboard in place, make sure the tech team knows their responsibilities, the ensemble their cues and that your equipment works as it should.

Remember! Framing doesn’t exist in VR, all the elements that are visible on set will be in the film. You are filming in 360 degrees.

You need several 360-cameras to film a 360 video.

A bit on the camera placement – the height of the camera will define the height of the viewer. Consider the system for stabilizing the camera for movement shots. And, anticipate post-production if you use camera stabilization equipment. Please note that the movement must be very slow and precise to avoid problems with bad images and the stitching in post-production. There is also the issue of the viewer experiencing motion sickness if you are not thoughtful about the camera movement.

The lighting is crucial for filming VR, avoiding lights that go directly into the camera. If you move lights during your shoot you will create problems when stitching the pieces together.

Microphones – choose ambisonic 3D microphones as they allow for sound spatialisation. Use cavalier microphones for catching the actors’ voices.

Phase 4: post-production

For post-production, editing software such as Adobe Premiere Pro and Final Cut Pro is used. For special effects (VFX) and to add interactive elements, you need to work with software designed for gaming – video games and 3D production. For example Unity or Unreal (game engine). There is also other software out there.

Import the video files into the software to stitch the images together. Then synchronize the video with the sound. Calibrate the images so the scenes all match in color, exposure, contrast etc. Add effects. Export when you are done with the last bits – titles, credits and so on.

VR equipment

There are quite a lot of equipment and accessories for VR. Here are some common ones:

• Headsets, provides a 3D view of the virtual environment
• Headphones for spatial audio
• Gyroscopes, accelerometers, magnetometers – used for head and motion tracking to adjust the virtual environment according to the users position and direction of the persons head in the room.
• Controllers – held by the user and used to interact with the virtual environment.
• Haptic gloves – a wearable device that allows users to experience realistic touch.
• Treadmills – there are 360 degree movement treadmills and ones that the user can move forwards and back on. The user can control speed and direction.

Production tools

Cameras

The camera market today is a trap, if you are trying to look for your first camera you will read and hear a lot about resolution. The resolution and the pixels you get are absolutely not the same. You can get super high resolution from your phone nowadays but it is nothing compared to the visual quality of the cinema camera of even lesser resolution. It is much more about sensors, lens quality, lighting in the scene and such factors. A lot of manufacturers of cameras give you their own proprietary software to do the stitching, and it is awful. Never rely on the camera native software, except maybe for the GoPro. 

Motion capture suit

There are several types of motion capture suits, including optical motion capture systems, inertial motion capture systems, and acoustic motion capture systems. Optical systems use cameras to track markers on the suit, inertial systems use accelerometers and gyroscopes, and acoustic systems use microphones to track the movement of the suit.

Motion capture suits for physical performers

There are also specific motion capture suits designed for dancers and other physical performers. These suits are typically lightweight and flexible, and include markers that are placed on specific points on the body to track movement. Some examples of motion capture suits designed for dancers include the Xsens MVN, the Motion Analysis Phantom, and the Qualisys Dance Motion Capture System.

Post-production

It is normal to feel a bit lost, afraid or uncomfortable, when you are getting into a new world. There are new rules you have to learn. If you are coming from film there are new tools you have to learn. Once you learn what rules apply with the new tools it is not that different. Things will start to make sense and you can use all your knowledge in 360 post production. Don’t believe people that say it is much more difficult, impossible or undoable.

Real-time engines

Free real-time engines – Unity and Unreal. Which one that is the best depends on what you want to do. With Unity it is easy to start but not to finish a project. With Unreal it is the reverse. When you think of visual quality and appealing pictures use Unreal. If you need a quick prototype in a few hours go with Unity. Both are very capable tools. It helps having an understanding of how the tools work. However, be a team player. Know your limits and collaborate with the experts.

Preparing the final file

It is critical in post-production to have an expert who will prepare the final file for you. The medium from the technical point of things is quite fragile. You need to save every pixel. There is so much information in the 360 sphere and it is so easy to lose that information due to some tick in the wrong place in the mastering phase. It is important to do your best to make the post-production as lose-less as possible. Use uncompressed footage from the camera and work in a software that does not mess with the original footage so the file doesn’t look bad in the end. You need a person that understands the process, software and potential problems. It’s easy to mess everything up in post.

Important!

Whatever you do – don’t underestimate the time post production takes!

Challenges to consider

Health and VR

When people are immersed and moving around in a virtual world while their bodies are standing still, they can get disoriented and experience nausea, headaches or feel drowsiness – this is similar to the experience of being motion sick in a moving car. Treadmills can alleviate motion sickness by simulating walking or running.

Age restrictions for VR

There is an ongoing discussion regarding the effect on young children immersed in a virtual environment. Some recommendations say that a child should be at least 12 years old to step into the virtual world. This is, in part, due to the sense of reality in the simulated world, and this could potentially lead to a child thinking the experience is a real memory.

Budget planning

VR equipment and production (pre and post) can be very pricey. Add to that the cost of external experts, and you will need a hefty budget to be able to finalize your creative idea. Before starting your project make sure your budget is planned well. Set aside a contingency fund for unexpected costs.

Download Checklist for VR (pdf).
Download the Terminology sheet for VR (pdf).

Lecturers for this lesson

Corinne Linder Creative Director, Fheel Concepts. The company Fheel Concepts was founded in 2016 with the wish to bring different art forms together all through the fundamental premises of contemporary circus. Corinne Linder on LinkedIn.

Simas Chomentauskas CG Supervisor at Gluck Media. Gluk Media develops interactive media and augmented reality solutions that blurs the line between the virtual and the real world by integrating innovations into everyday life.
Simas Chomentauskas on LinkedIn.

Dimitri Sourzac VFX artist & producer at Novolab. After completing his engineering studies, he worked for 6 years in the field of digital arts, from audio software development to the design of visual effects in the field of virtual reality. He experiments with different means of expression including video, music, photography and digital painting. As a visual transfiguration perceived at every moment: places, people and actions are distorted, augmented and interpreted symbolically. Imaginary visuals is then considered a tool for simulation, analysis and understanding the real world. Check out Dimitris website.