Product Q&A: Virtual and augmented reality

What trends and requirements are driving developments in this area?

JA: Due to the new challenges forced upon companies by the pandemic, they have no appetite for technology that is unproven, they have a requirement for technology that is ready to use now, out-of-the-box.

The challenge for many companies is how fast they can equip their teams with assisted reality solutions to stay competitive, or to meet the demands of their customers. Much of this depends on whether the solutions can offer seamless integration with core enterprise applications and backend systems; for instance, Microsoft Teams or Zoom as collaboration platforms. Compatibility with these is essential.

Operations leaders also require technology that provides their frontline workforce an intuitive way to collaborate, and provide multi-party hands-free calling. It's also vital that the technology is comfortable to wear throughout the day, provides a ‘full-shift’ and hot-swappable battery, is easy to use and complies with health, safety and environmental (HSE) requirements.

ML: Device performance shares some of the responsibility for driving AR developments in industry. We’re at a point in time within the AR ecosystem where devices are much more capable than use-cases we’re deploying to them. This is due to improved optics, purpose-built chips for sensor fusion and processing and cloud computing over previous hardware generations. A lot of value can be derived from downloading a CAD model onto an AR device to check the validity of an assembly in-process; but the devices themselves are capable of so much more.

Devices of old (mostly consumer tablets and phones) presented challenges. Tracking, for example (a process whereby an AR device or application maintains an understanding of its physical position ‘pose’ within the real world) was limited in accuracy and consistency. Devices were overwhelmingly dependent on markers or distinct features to be visible to present AR graphics to the user. This restricted developers, and ultimately users, by decreasing the number of potential use-cases.

What we’re seeing now is an explosion of transformational use-cases due to the glut of device performance on offer. This recent emergence of new capabilities has begun to shine a spotlight on other areas of manufacturing which cannot satisfy the precursors to unlocking the potential of AR.

AK: We view Mixed Reality as the fourth wave in computing, following mainframes, PCs and smartphones. As a result of the pandemic, we have experienced a huge reduction in our in-person social connections. In areas like the workplace, we’re seeing a move towards virtual meetings, workshops, design sessions, social gatherings, and meet-ups. These social shifts have accelerated development of platforms such as Microsoft Mesh, delivering the feeling of presence and 3D collaboration to these hybrid and/or virtual gatherings.

How are your products helping customers meet the challenges they are facing?

JM: A defining characteristic of our software is its ability to ingest huge CAD files and import them into an immersive visualisation. This significantly reduces the work required to create a VR scene and expands what’s possible within VR.

Daily design reviews are being performed on large models, including detailed ships and process plants, both remotely and in dedicated facilities using VR. Robust CAD importers that don’t require manual pre-processing or simplification allow us to be a practical day-to-day review tool across copious industries.

The use of VR for training purposes has also made a significant impact. Instructors can easily adapt scenarios to create rewarding and dynamic situations, from training police on how to respond to crime scenes to surgery training for medical professionals.

Scenes can be animated, adapted and decorated without writing any code, so users aren't bound by a single scenario and do not need any software experience to adjust situations -  a distinct attribute of Virtalis software.

JA: Global professional services firm Marsh & McLennan, and its subsidiary Marsh, needed to identify ways in which its risk assessment team could keep operations functioning despite travel restrictions imposed by the pandemic. With the need to keep underwriters involved in the risk assessment process, it developed a solution to utilise RealWear devices so that the on-site engineer dials in for a video call, and underwriters — now based remotely — can dial in to see and hear what the engineer is seeing on-site, negating all of the cost, pollution, congestion and productivity problems that come with travelling.

Another example is UK customer Mace Group, a global consultancy and construction firm headquartered in London. Its deployment — among many other advantages — has banished the logistical complexities of trying to coordinate diaries of several site inspectors to be present at the same time on the same day.

AK: Working with Lockheed Martin on the production of Orion, their only spacecraft capable of taking humans to space, they have achieved a 90 per cent increase in efficiency by using HoloLens 2 supported by Azure Mixed Reality services. HoloLens 2 understands the physical environment and allows the Lockheed Martin team to place digital content within it. Therefore, when a technician puts on a HoloLens 2 headset, they can instantly see work instructions in their field of view, increasing their on-task efficiency. The team at Lockheed Martin discovered that through Mixed Reality, they were able to take an eight-hour activity and reduce it down to 45 minutes.

In the healthcare sector we have also been working with Imperial College NHS Trust during the pandemic. In an environment where hospital staff were used to working in close proximity with each other, we introduced HoloLens to reduce the number of staff on ward rounds from four to one.

What is the key product / technology for you currently and how is it being applied?

PH: PTC is seeing enormous traction for our market-leading Vuforia suite of AR technologies. There are two ways we are bringing it to market. Firstly, by using case-specific technologies that answer a particular problem: for instance, out-of-the-box offerings that are easy to use, easy to consume and deliver value. These typically use cloud infrastructure and our main products include Vuforia Chalk, VEC and Instruct (the latter specifically focusing on work instruction). 

Secondly, a platform approach where PTC provides a set of tools for clients to build things specific to their business. Vuforia Studio uses AR to guide the worker in the construction/assembly of products by ensuring information is at their fingertips, eliminating cognitive distance. Vuforia Engine delivers high-fidelity, high-quality graphics in an AR experience, which is a godsend for sales and marketing departments trying to visualise the technology/products they are selling.

JM: While defining how to best serve our market, we developed the concept of ‘effortless visualisations’. We concluded visualisation software required five key features for organisations to use it across departments and functions: performant, secure, context-sensitive, always available and hardware agnostic.

As a result, Virtalis Reach was created with these five principles firmly in mind. The XR platform can be hosted on customer sites with distribution controlled entirely by them. Virtalis Reach helps streamline layout planning, design reviews, training and education, sales and engagement, and maintenance. We are seeing increasing deployment of it as a tool to support these processes across organisations.

What technologies do you see changing your sector in the future?

ML: WebXR and AR-enabled web content has the potential to impact manufacturing and wider society. So much of the software that used to have a home as installed applications on our devices is becoming web-based. AR devices of the future will be able to consume web content just as easily as your mobile phone; the advantage that presents is a significant increase in the number of available applications, use cases and improved communication via remote support and file-sharing. The next challenge will be in the optimisation of user-interfaces; as a contextually and environmentally aware technology, we will engage with AR content in a way never experienced on a 2D screen and with that we will have to completely rewrite our existing user interface design mantras.

PH: Our technology roadmap is continually evolving, and this is being driven by our own innovation and by listening to our customers, ensuring any new solutions we are developing in AR deliver tangible benefits.

Interest in spatial technology is going to increase, as this gives us the ability to map an environment and understand what is happening within that area to help drive worker productivity and improve health and safety. These spatial technologies - integrating AR - can also provide new ways for a consumer to interact with a product. One example is creating a single interface that could control all the devices in your kitchen.

Wearables is another area we should see major strides in. Industry is calling for devices that are lightweight, low cost, powerful and offer a battery life that will enable it to complete an eight-hour shift without needing to recharge.

Finally, the ability to connect through 5G, which provides greater density of connectivity, speed and bandwidth, can only help drive the digital transformation.