Painting fool

If a team from Imperial College London’s Department of Computing have their way, bespoke, individualised and beautiful pieces of artwork generated by Artificial Intelligence could become a reality.

Last month, Dr Simon Colton, PhD student Michel Valstar, and Dr Maja Pantic from the Department of Computing, won the prestigious award for the best Machine Intelligence demonstration at the British Computer Society’s Specialist Group on Artificial Intelligence (SGAI) conference, held at Cambridge University.

The SGAI award, voted by some of the world’s foremost authorities on Artificial Intelligence (AI), recognised the team’s innovative and creative automated software technology – The Painting Fool – which creates portraits by reading the facial expressions of a human subject and interpreting the emotion conveyed by expressions.

Dr Colton, who designed The Painting Fool, said he was excited by the challenge of creating an automated artist, which produces beautiful pieces of artwork. He also said his work raised some interesting questions about how people perceive creativity.

’I’m a computational creativity researcher, which is a sub-area of AI research, so the Painting Fool, can, and has been, used to test out some theories about how people perceive software as creative - or not. For instance, because people have to watch The Painting Fool produce its artwork with painstaking detail they seem to project more value onto it than they would if Photoshop produced something in milliseconds,’ he said.

In order to create a painting, The Painting Fool programme first takes an image of its subject, via a video camera, and segments it into a number of regions of colour. It then takes each segment and simulates how a human artist might paint that region. For instance, The Painting Fool may choose to imitate an artist’s brushstrokes by painting a circular region with ever decreasing circles, or straight lines with fine strokes.

In addition to simulating the artist’s style of work, it also replicates what art materials, like acrylics, pastels, chalk or watercolours would look like.

’In recent developments, I’ve enabled the choice of colour palette and materials to be controlled by other software. For the competition we used machine vision software to control the painting styles, so the portraits were appropriate to the emotion being displayed by the sitter,’ Dr Colton explained.

He said this technology allows the The Painting Fool to use a number of techniques to depict emotions - for instance, if a person displays anger it may paint the subject in bold angry reds and black and use harsh brushstrokes to represent rage.

This type of intelligent interpretation of human feelings was possible because of the work of fellow contributor Michel Valstar, who said he felt ’absolutely fantastic’ about winning the award.

Valstar, who is a PhD student of Dr Maja Pantic, contributed software which helped The Painting Fool interpret human emotions. As part of his PhD, Valstar developed the Facial Action Analysis system used by The Painting Fool to recognise complicated facial features.

’I developed a computer system that is capable of detecting 23 out of a total 27 facial muscle actions which are known as Action Units - or AUs. These AUs help to describe typical expressions of the six basic emotions which include anger, disgust, fear, happiness, sadness or surprise,’ explained Valstar.

Apart from detecting which AUs were present, Valstar says this technology was the first in the world to actually pinpoint when an AU starts, reaches its peak and when it starts to return to its normal expression.

’We use this knowledge to select from a video frame where the emotion shown was most intense. We call that the apex frame,’ he said.

The apex frame is sent to The Painting Fool programme, together with information about which emotion was detected and where the facial points were at that time.

Colton and Valstar conducted their demonstration in front of delegates at the conference. Michel was the first ‘guinea pig’ to have his portrait painted, followed by Paulo - a delegate. Unfortunately, the exhibition did not go without a hitch.

’About ten people in the audience decided to use flash photography to capture the moment. Unfortunately, this meant the machine vision software was put off. It actually thought Paulo was angry when he was smiling. So, instead of painting him in happy colours, The Painting Fool chose angry greens and reds,’ laughed Dr Colton.