Emotion Recognition

We’ve taught computers to recognise human
emotion, enagement and attention levels

Proprietary technology that’s built on the facial action coding system

The Facial Action Coding System (FACS) was
developed by Paul Ekman to categorise human 
facial movements.

It works as an automated computer system which 
can categorise human emotions according to
changes or movements of the face. Facial
expressions are broken down into the individual 
Action Units that make up a specific expression
over time. The output of your collected audience
emotions are aggregated and displayed within the
dashboard as six basic emotion metrics and three
proprietry metrics for measuring the emotional

Happiness is synonymous with a smile. It consists of the following action units: AU 6+12, which indicate the cheeks raising and the corners of the mouth pulling up respectively.
Synonymous with a 'shocked' expression, surprise consists of the following action units: AU 1+2+5B+26, which is a combination of raised eyebrows, eyes wide open (raised eyelids) and the jaw dropping to reveal and open mouth.
Confusion is synonymous with a lowering of the brow. It consists of the following action units: AU 4+5+7+23, which is a combination of lowering the brow, raising and narrowing of the eyelids, and a tightening of the lips.
Sadness or empathy is synonymous with the classic downturned mouth. It consists of action units: AU 1+4+15, which represent lowered brows but upturned at the inner ends, and the depression of the lips.
An expression of distaste, disgust consists of action units: AU 9+15+16, which represents nose wrinkling, the downturn of the lower lip and the corners of the mouth.
Synonymous with a fearful expression, Scared is represented by the action units: AU 1+2+4+5+20+26, which indicate the eyebrows raising or lowering, the corners of the lids raising, the lips stretching and the jaw dropping.

Going Beyond the Basic Emotions

Our 3 Proprietary Metrics

In addition to our emotion metrics, we also
track a variety of proprietary metrics. These
have been derived through our own research,
and can be used in conjunction with basic
emotions classifiers to gain deeper insights
into the content tested.

  1. Engagement When a participant has an expressive 
    reaction to a stimulus, they are said to be
    ‘emotionally engaged’. It represents the % of
    participants who showed any reaction –
    either during a particular second
    (second-by-second as seen on the site charts)
    or at any point during the test (top-line
    figures). This includes but is not limited to
    displaying any of the six basic emotions.
  2. Negativity A proprietary metric to demonstrate how
    positive or negative a reaction is. It is,
    essentially Positive emotions minus Negative
    emotions, and is calculated using specifically
    trained Positive and Negative classifiers in the
    Realeyes facial coding software. These
    additional classifiers help to elucidate the
    emotional “tenor” of the viewing experience.
  3. Valence The percentage of people showing an
    emotion classified as negative either during a
    particular second (second-by-second as seen
    on the dashboard charts) or at any point
    during the test (topline figures). This is not
    merely the sum of emotions such as Disgust,
    Sadness, and Scared, but uses specifically
    trained ‘Negative’ classifiers.

Triple Refinement

Both people and machines recognise emotions in the same way; separateing
background with the foreground, the ability to focus – and detect the face and shape of
it’s expression.

Face Detection

We’ve developed our own face detector which is more accurate and reliable than the Viola-Jones detector, generally considered the industry standard.


Feature Detection

Once the region of the face has been detected, we need to be able to identify the landmark points of the face: the nose, the eyes, the mouth, the eyebrows.


Expression Level

Using all the frames in respondent’s
video, we create a person-dependent
baseline, or mean face shape. By measur
ing their expressions as they deviate from
their ‘neutral’ face, accounts for people
who naturally look more positive or nega
tive and correcting any bias.


Face Detection

For each frame, the algorithm estimates the position of the face.

Keypoint Alignment

49 keypoints are tracked around facial features including the mouth, nose, eyes and eyebrows.

Expression Representation

From these keypoints, we extract the face image and feed it to our deep learning based facial interpreter. It will produce a signature encoding the expressions of the face.The model learned by looking at millions of faces.

Feature Vector

To encode the dynamics of facial emotions, a temporal module stores those signatures and processes them in a sequence to estimate whether or not the appropriate expression is present.

Emotion Curve

This frame by frame data is collected to give a trace of when the face is expressing the emotion.

How We Compare to Humans

To review the accuracy of our motions, we plot the aggregate output of
our metrics against ground truth data – the majority voting of 5 human
annotators who notate frame by frame the emotion they see.

Tech That’s at Home in the Wild

Bad lighting, heavy shadows, thick facial hair,
glasses, typically makes emotion detection more
challenging. However, we use ‘in the wild’ datasets
that employs machine learning that teach our
algorithms to cope with such obstacles.

Get more information on how we collect
respondent data from our tech white paper.

We took it up a notch


Complete Tracking

Face detection Facial landmarks 3D headpose Basic emotions Expressivity Valence Attention


Fully Optimized

Non-exaggerated natural reactions Complete in-the-wild environment Video watching in mobile and desktop web browsers Precision over speed


Value Added

Emotional events Emotions duration, intensity, volume Emotions by video segments Emotions by audience segments Interest score Attention volume & quality Memory & emotional hook

Reporting metrics that matter

Realeyes Score

Score combines viewers’ emotional engagement and attention levels to help maximise the ROI of media campaigns.


EmotionAll® Score

A snapshot of your performance per asset, enabling you to predict social media activity, and inform media distribution decisions.


Attention Score

See how well your content grabs and maintains audience attention to know whether they’re gripped or easily distracted.


Brand Logo Score

See the level of audience engagement during the precise moments when your brand logo is featured to ensure a brand connection is made.


Sentiment Score

Find out what people really think of about your brand, ad or campaign using AI to analyse written comments.


Play Video

Here’s the science bit

Our white paper covers the science behind our facial coding technology and how we’ve taught machines to recognise emotions, just like humans.