FascinatE
Joanneum Research, Arri, Softeco Sismat, Interactive Institute, Salford University, Technicolor, BBC R&D, Fraunhofer HHI, Alcatel-Lucent, TNO
A pub on Tottenham Court Road in London is one of many in Britain that shows football matches from Sky Sports in 3D.
The sight of adults wearing stereoscopic 3D glasses and staring at a screen is quite disconcerting but lots of people do it. Importantly, they are prepared to pay the landlord of the Rising Sun for the privilege of watching their team play in 3D.
This viewing phenomenon is just one of a host of advances that have heightened the TV-watching experience. HD, On Demand services and the ability to pause TV shows demonstrate how consumers are increasingly being given control of how they use TV.
Consumers are set to become more involved in their television viewing thanks to the FascinatE (a tortuous acronym for Format-Agnostic SCript-based INterACTive Experience) project, a €9.35m (£9m) scheme funded by the EU’s Seventh Framework Programme.
With 11 partners across the EU, the FascinatE team is developing a system that allows consumers to view and navigate around an ultra-high-resolution video panorama showing a live event, with the accompanying audio automatically changing to match the selected view.
The output can be adapted to their particular kind of device, covering anything from a mobile handset to an immersive panoramic display.
At the production side, this has required the development of new audio- and video-capture systems, plus scripting systems to control the shot-framing options presented to the viewer.
Moreover, to make it work, intelligent networks with processing components are being developed to re-purpose the content to suit different types of device and framing selections, and user terminals supporting interaction methods will allow viewers to control and display the content.
The system is expected to deliver a unique experience that will transform how live events are enjoyed
Once complete, the system is expected to deliver a unique interactive experience for consumers that will transform how live events can be enjoyed on TV, on large-screen public displays and on mobile devices.
Project member Ben Shirley, a senior lecturer at Salford University, is involved in the audio aspect of the project.
He explained to The Engineer that FascinatE could already be implemented on current smartphones.
’The only problem is network speeds,’ he said. ’Some of the context of the full FascinatE system, namely fully immersive audio, large full-resolution panorama — being able to zoom in and out to the full resolution standard — needs network bandwidth that’s projected to be there in four to five years’ time.’
One interactive element of FascinatE lets the viewer watching a live event ’grab’ a participant using a remote control or gesture interface, then places that person at the top of the screen. This screen selection lets the viewer follow the person around, for example, a football pitch with the rest of the game dominating the screen.
’My job on the audio side is to make sure you experience appropriate sound for exactly where your camera is pointing,’ said Shirley. ’So you would hear everything the sportsperson is saying and doing by positioning all the audio objects taking place during the game.’
Shirley added that an essential element of the project requires the format-agnostic delivery of audio-visual content.
’When you’re broadcasting football or music you’re looking at what the end-user device is,’ explained Shirley. ’If you’re broadcasting for 3D it’ll be a completely different workflow with different engineers and production values to if you’re viewing in 2D.
’If you’re packaging it for IPTV or mobiles it’ll be broadcast differently too.
’The idea of the format-agnostic part of it is it’s a single end-to-end chain from the acquisition through to the packaging, transmission and distribution, which is not dependent on the end-user device.’
To test the system, the team took FascinatE to a Premier League football match between Chelsea and Wolverhampton Wanderers in October 2010.
The aim of the trial was to gather a complete set of audio-visual material to take the concept forward.
The team set up an omni-directional high-resolution camera system from HHI, the Alexa high-dynamic-range camera from ARRI, an Eigen-mike from Technicolor and two Soundfield mics.
Shirley said that the camera set-up was relatively straightforward but the audio presented its own set of challenges, adding that the trial led to a development that could see an aspect of the technology being spun off from the main project.
From the end-user perspective it is important to know where a selected player is relative to the ball. Fifa, football’s governing body, will not, however, let balls be fitted with tracking devices.
’In an ideal world we’d have liked to have had large arrays of microphones so that we could very accurately pinpoint where the sound was coming from,’ added Shirley. ’However, we were only allowed to deploy 12 shotgun microphones around the pitch and a couple of stereo pairs.
“You’ve got 47,000 people behind them shouting at the top of their voice, which isn’t ideal”
BEN SHIRLEY, SALFORD UNIVERSITY
’We put in a couple of additional ones: an Eigenmike microphone, which is a 32-capsule surround-sound microphone that is very accurate, and two Soundfield mics that are lower-order surround-sound mics.
’Restrictions meant we couldn’t put more in. In addition to that, when you’re trying to identify where on the pitch the sound is coming from, you have mics that can be triangulated and are reasonably accurate. However, despite being well spaced out around the pitch, you’ve got 47,000 people immediately behind them shouting at the top of their voice, which isn’t ideal.’
Despite this, the team developed methods for extracting audio objects off the pitch without the crowd noise, thus pinpointing where the players were during the game.
This in turn has led to aspects of the audio side finding new applications.
Following a conference, two broadcast companies expressed an interest in automating the mixing process for live sound.
’So we’re pursuing that as a separate spin-out from the project as it provides benefits to existing broadcast workflows,’ said Shirley.
The team has yet to decide where the next trial will take place.
Shirley said that between 20 and 25 events have been suggested and that the final decision will be based on TV rights issues and how challenging it will be to deliver video and audio capture.
’It has to be a fairly high-profile event so it’ll attract interest from industry and the general public in terms of dissemination,’ said Shirley.
The project concludes in 2013 and includes input from partners including the BBC, Technicolor, Alcatel Lucent, Fraunhofer HHI, Joanneum Research and TNO.
runners-up
The other shortlisted candidates in this category were:
High-dynamic-range enabled television
University of Warwick, Altera
High Dynamic Range (HDR) is a set of techniques that allow a greater range of luminances between light and dark areas of a scene than normal digital-imaging techniques.
This wider dynamic range allows HDR images to more accurately represent the range of light-intensity levels found in real scenes, ranging from direct sunlight to faint starlight.
The collaboration has developed a prototype embedded system that allows dynamic HDR content to be directly displayed on future HDR televisions and tone mapped and displayed in real time on existing LDR televisions, providing a significantly enhanced viewing experience.
A low-cost eye tracker for games
De Montfort University, Sleepy Dog
This project developed a prototype of the hardware, a software interface to enable eye-gaze data to be incorporated into games and three demonstrator games showing different applications of gaze in games.
In addition to being suitable for the games market, the low-cost device enables eye gaze to be used as an assistive technology by people with physical disabilities — not only for playing computer games but for other education- and work-related purposes. The device not only performs better than the available commercial systems that are for communication but it allows far more people to have access to the high-bandwidth input channel, which is needed for many games and online social activities.
Five ways to prepare for your first day
If I may add my own personal Tip No. 6 it goes something like this: From time to time a more senior member of staff will start explaining something...