Looks familiar

The 3D reconstruction of a face from a single 2D image could be made faster and more accurate with new software being developed by scientists at York University.

Dr William Smith, a lecturer in computer vision at York, aims to combine the advantages of two face-recognition techniques to challenge the most advanced method for recovering 3D shapes when there is only one image to work from, such as an image from a CCTV camera. One of the techniques, which is also the most sophisticated, uses a morphable statistical model of facial appearance, while the approach that Smith has been working on uses classical shape-from-shading techniques.

The statistical approach works by taking a model and adjusting parameters to try to fit the model to an image.

'This is something that has been really big in computer vision,' said Smith. 'The idea is that faces are all quite similar and you can learn the variations in face shape by looking at loads of examples.'

According to Smith, the advantages of this technique are that it works well on real images and can recognise a large number of different faces. A weakness, however, is that the shape it recovers is completely determined by what it has learned earlier, which means it is not able to recover atypical features that it has not come across before. This is where the latter face recognition technique is relevant.

'Shape-from-shading is a really old idea which goes back to the 1950s and 1960s — they used it to try and recover the topography of the moon before they landed there.

'Our method uses the information in the image, so we interpret the changes in brightness and darkness as changes in the direction of the surface of the face. The strength of our method is in recovering local shape, for example, wrinkles on the skin or some subtle difference in nose shape,' he said.

The details missed by the model approach vary depending on factors such as the position of the face.

'For example, if you view the face in profile, you can recover the shape of the nose quite well because you get to see the outline. If you view it from the front then the nose shape is not so good, because there is not as much information there,' said Smith.

'The issue of face expression is quite difficult with morphable model approaches — you would need hundreds of examples of every expression. So we are looking at using shading information to recover 3D shapes when they might be in a completely different expression, then we can feed that back into the model so the model will improve and learn to recognise these images.'

While the shape-from-shading technique is better at recovering local features, Smith said the global 3D shape recovered in this technique was of inferior quality to that recovered using a morphable model, hence the aim to combine the methods.

By combining techniques the researchers may also be able to speed up the time it takes to fit the model to an image. According to published results, the morphable model now takes four and a half minutes to fit to an image.

'When you fit a morphable model to an image, all you look at is an error value, which tells you how different your predicted appearance and your actual image is, and you are just trying to minimise that error,' said Smith. 'That is quite difficult and takes a lot of computational power, so it is a really slow process.

'The shading information allows us to take the 3D shape out of the image and use that as a rough fit of the morphable model directly. That gives us an approximate fit of the morphable model and we can iterate those two processes. This may allow us to fit the model much quicker.'

In addition to using the software to reconstruct shapes from CCTV images, Smith said the software could also be used in the gaming industry or to quantify the success of plastic surgery operations.

'In graphics animations, being able to take an image of someone and get a 3D model that you can stick in a computer game is useful.

'There's also things like reconstructive plastic surgery. If someone is involved in an accident and the tissue of their face is damaged, in that case all you have is images of them prior to the accident, so it would be quite useful to be able to recover a 3D shape from an old image, then use that and a scan of the damaged face to make suggestions about what sort of surgery should be done,' he said.