From the operating theatre to the factory floor, XR technologies (which include augmented, virtual and mixed reality systems) are increasingly changing the way that specialists work and helping to open up new levels of capability. The Engineer put together a panel of experts to discuss the trends and technologies that are shaping this rapidly advancing field.
Meet the experts
Richard Vincent - CEO and co-founder, FundamentalVR
Paul Haimes - VP European Technical, PTC
Leila Martine - UK product marketing director of Mixed Reality, Microsoft
What are the key current trends in your area of XR?
LM: As businesses focus on planning for and adapting to the unprecedented challenges, we are seeing clear trends where MR (mixed reality) can immediately support, including direct and remote servicing of customers equipment. In recent months, the pandemic has brought to light some of the specific Mixed Reality solutions that can help solve major business issues that companies from several industries are facing. MR is also giving businesses the opportunity to rapidly restructure assembly lines and corresponding re-training of their workforce, while ensuring the health and safety of staff.
PH: Within the last 12 or 18 months, we have seen bigger companies move from a more general level of assessing technologies, such as proof of technology and proof of value, to scaling AR (augmented reality) to suit their requirements. This is also supported by the continued development on the hardware front, such as HoloLens and RealWare technologies – there is a sustained advancement in terms of the hardware and, although there are still some limitations, they’re nothing stopping companies deploying at scale now.
RV: In the healthcare space, we are seeing XR implemented in several ways from medical training all the way through to patient care. This includes using VR (virtual reality) and AR to train and educate medical students as well as help physicians and surgeons maintain and hone their skills. Our platform Fundamental Surgery is an example of this as it lets trainees and surgeons practice their skills through VR and haptics anytime and anywhere. The use of XR allows companies like ours to scale training efforts fast, provide detailed insight into skills development and deliver remote, mass multiuser experiences. We are also seeing XR being used in the OR for the development of surgical planning and for patient treatment to help reduce stress and execute treatment plans.
Can you give us an example of an innovative application of XR that you or a customer has made?
PH: We are seeing companies wanting to push the boundaries of what is possible. That’s particularly the case of augmented reality, whereby bespoke assembly AR experiences, based on a PLM managed configuration context, allows customers to build experiences that are tuned to the product variant that they are making – this is one of the most exciting areas that we are developing our technology towards. A good example of this is PTC’s customer Volvo, who, within its truck engine quality control and assurance process, are exploring the use of PTC’s Vuforia Augmented Reality Suite at the inspection end of the engine build line to track any quality issues. Building from this context, the prospect of customer specific experiences configured to consumer products become possible.
LM: A consortium of major businesses have come together and selected two existing ventilator designs from UK firms Penlon and Smiths Group to meet high-level specifications for a Rapidly Manufactured Ventilator System, developed by clinicians and the Medicines and Healthcare products Regulatory Agency to aid the NHS. Leveraging Microsoft HoloLens, Avanade and PTC are supporting the VentilatorChallengeUK to enable a number of solutions including PTC Vuforia Expert, Dynamics 365 Guides and Dynamics 365 Remote Assist to fast track training and knowledge transfer of hundreds of assembly workers across multiple manufacturers and multiple sites.
RV: Our platform, Fundamental Surgery, is a unique and market leading example of XR in use in the surgical training arena. The multimodal platform has two offerings.
First, our HapticVRTM product combines VR with the sense-of-touch to allow users to experience the same sights, sounds and feelings they would in an actual procedure. This allows trainees to build muscle memory needed to be a surgeon.
Our new @HomeVR offering compliments our HapticVR platform by bringing our accredited simulations to standalone VR headsets such as Oculus Quest. This combines Cutaneous (tactile vibration) and Kinesthetics (force feedback & position) haptic technologies in a single platform, which is optimized for various levels of the learning process.
Overall, our Fundamental Surgery platform lets users train and hone their skills virtually from anywhere and at any time with simulations accredited by the American Academy of Orthopaedic Surgeons and the Royal College of Surgeons of England.
Are we going to see more widespread adoption or will it remain a relatively niche technology?
LM: For broader adoption of MR technology to happen, businesses need to establish the ROI and suitable out-of-the-box use cases before deploying. There are many immediate solutions available, such as Dynamics 365 Remote Assist, which allows technicians to collaborate with colleagues from different locations. Businesses preparing for a reduced workforce due to pandemic-related safety measures can use Remote Assist to leverage employee expertise remotely. Additionally, medical clinicians or technicians maintaining critical equipment in the field can use Remote Assist while sharing an in context, real-time view of the work site with an expert working from a different location. As businesses realise the potential of MR, the question is not if we will see more widespread adoption, but when.
RV: I believe that XR training in the medical field will become mainstream. The Covid-19 pandemic has shown the need for new ways of delivering ongoing education and bringing remote individuals together quickly. XR is perfect for this. XR allows people the unique opportunity to step into virtual worlds from anywhere to practice, learn and teach.
PH: Last year forward-thinking companies made it a breakout year for AR, moving projects along rapidly from pilot to production. Now, as the Covid-19 outbreak forces them to figure out new ways to provide assistance to workers and customer in other regions, more and more companies will see the true value of AR. Newcomers are already seeing the gains in worker efficiency and quality, whilst reducing the cost of training by more effectively transferring knowledge to new workers via AR. Of course, the issues presented by Covid-19 go way beyond the everyday business concerns of productivity or cost. This is about the health and safety of workers and thus demands a far more urgent and targeted response. AR can provide this, because it is easy and fast to set up and requires no specialist equipment to run – we see many companies using tablet and smartphones to consume AR experiences.
What exciting developments can we expect in the coming years?
PH: I believe that a Geo Fencing driven AR experience will be a massive part of new developments, and evolve at an unprecedented rate, once the devices can enable it. Through this, operators will be able to move from one environment to another and have information streamed onto their device, that is specific to their surroundings and proximity to the machines.
RV: I believe we are at the exciting beginning of XR’s full potential, especially in the medical space. I expect we will see developments in the use of full haptics, which will allow for deeper immersion and skills development.
Additionally, the industry will hopefully advance to offer patient specific modelling through XR that will allow for surgeons to personalize their training on each patient. The merging of mixed and virtual reality hardware will also allow seamless movement between both.
Finally, developments in improving GPU capabilities on smaller devices is expected, which will enhance experiences.
LM: From construction sites to factory floors, to operating rooms and classrooms, mixed reality, AI, the intelligent edge and intelligent cloud are changing how we work, learn, communicate and get things done. MR, along with smart devices and IoT will no doubt transform several areas of our world, and as technologies continue to merge, the rate of innovation will increase exponentially. Over the coming years we will continue to see an increase in retail, education and healthcare businesses developing MR applications and utilising out-of-the-box applications to solve a range of business challenges.
MORE ON AR AND VR FROM THE ENGINEER
AR system helps surgeons tackle tumours
Promoted content: Does social media work for engineers – and how can you make it work for you?
So in addition to doing their own job, engineers are expected to do the marketing department´s work for them as well? Sorry, wait a minute, I know the...