Discover how Heineken’s media partner Verizon, found which content resonated most with the target audience to help guide a more efficient media spend.
Menu
Case Studies Category: Creative Performance
Twix – ‘Conveyor Belt’
Mars Inc. wanted to know how the choice of soundtrack to an ad might affect its impact on the audience – so we ran a small study with an old Twix ad to find out.
Automotive Study
We collected the social data and measured the emotional responses of 30,000 viewers watching 130 car ads from most major automotive brands to see what drives campaign success.
Coca-Cola – ‘WOAH’
Vice’s creative agency Virtue, conducted robust large scale content testing for Coca-Cola’s “Where Only Awesome Happens” – discovering what assets to drop and which markets were most effective.
CokeTV – ‘BucketList’
CokeTV were ready to launch their #CokeTVBucketList YouTube channel – wanting to ensure that it grabbed and retained a young audience’s attention whilst outshining the competition.
Cleveland Golf – ‘Levels Up’
Cleveland Golf - Levels Up
Unlocks creative performance with Realeyes advertising attention tool
Situation
Global golf manufacturer Cleveland Golf has grown significantly in recent years and its advertising budget has followed suit. The executive team and board of directors wanted a universal way to predict creative impact in advance of campaign launches to ensure revenue growth and spend efficiency.
The company previously used traditional research companies on pre- and post-purchase interest. However, results were not actionable for campaign optimization. They arrived too late and failed to deliver granular feedback on what was working, and not, within any given video.
With a large group of executives and marketers involved in global creative decisions, Cleveland Golf needed an objective way to identify and decode high-performing creative.
Challenge
Global golf manufacturer Cleveland Golf has grown significantly in recent years and its advertising budget has followed suit. The executive team and board of directors wanted a universal way to predict creative impact in advance of campaign launches to ensure revenue growth and spend efficiency.
The company previously used traditional research companies on pre- and post-purchase interest. However, results were not actionable for campaign optimization. They arrived too late and failed to deliver granular feedback on what was working, and not, within any given video.
With a large group of executives and marketers involved in global creative decisions, Cleveland Golf needed an objective way to identify and decode high-performing creative.
a universal way to predict creative impact in advance of campaign launches to ensure revenue growth and spend efficiency
Impact
Cleveland Golf needed to establish a benchmark for all its past, current and future creative, while also making the right creative decisions for a crucial upcoming campaign with a high-production budget.
In addition, the creative team wanted to explore why some recent ads underperformed with their audiences.
The company needed new creative to reinforce its golf prestige and equipment performance.
establish a benchmark for all its past, current and future creative
Outlook
Cleveland Golf partnered with Realeyes, using its PreView solution to measure and benchmark existing creative attention, to adopt a decision model for creative performance.
- When PreView Quality Score verified high attention and ad efficiency thresholds for a creative asset, Cleveland Golf would run media and spend at planned levels or more.
- If results for a creative asset were average, the team would use second-by-second and key-moments data to edit and re-test the video until it achieved a high Quality Score.
- For much-lower-performing creative, Cleveland Golf would apply a smaller budget, reallocating spend to other creative with better PreView results or cut the creative altogether.
The PreView results showed that some ads intended to run were failing to capture attention. PreView surfaced areas for improvement, including in storylines, visuals, and celebrity endorsers. Cleveland Golf considered the PreView results from its previous creative and felt comfortable moving in a new direction for better performance.
“The PreView system checked all of our boxes by bringing objective data to the table and quickly,” said Scott Carlyle, SVP, Global Marketing at Cleveland Golf. “Even better was that we could get started immediately with literally no work or setup on our side other than sending video files and receiving turnkey report summaries that only take minutes to interpret.”
PreView surfaced areas for improvement, including in storylines, visuals, and celebrity endorsers
Impact
Three of the five ads for the new campaign, created using insights from utilizing PreView on previous creative, had a Quality Score of 7 out of a possible 10.
The attention performance indicative in the Quality Scores also results in significant cost efficiencies on media outcomes. Assuming a nominal CPM of $10, the cost of 1,000 quality exposures (qCPM) for the three top ads was $15. Compared to three legacy ads that had a qCPM of $35, that’s a 58% improvement!
Cleveland Golf focused the new product advertising on the three superior creatives. The team will edit the two middling ads and flight them later if their scores improve to 7 or better.
“Since using PreView, we have a more critical eye when evaluating how much information we can expect a viewer to hear, understand, and internalize in any given asset,” said James Fodera, Cleveland Golf Creative Manager. “We’ve also greatly simplified our editing style, with fewer cuts, less montage, and fewer featured faces. It’s been freeing and refreshing to confidently write, produce, and edit delightfully simple campaigns.”
58% qCPM Improvement!
“Since using PreView, we have a more critical eye when evaluating how much information we can expect a viewer to hear, understand, and internalize”
Outlook
In addition to tactical creative performance improvements, Realeyes and the PreView tool have prompted Cleveland Golf to rethink how it positions its products, replacing detail-laden messaging with simplicity. PreView has helped build a communication bridge between the creative and media teams, enabling them to work more effectively together toward attention and business outcomes.
Management and the Board of Directors of Cleveland Golf are excited that PreView will serve as both a creative qualifier and an engine of better advertising. The brand will continue to prioritize ads with the highest attention and qCPM. And they will use PreView’s diagnostics to edit underperforming creatives – making the most of their creative and media investments.
PreView has helped build a bridge between the creative and media teams
Results
| Brand | Date Tested | Quality Score | qCPM | Creative Efficiency | |
|---|---|---|---|---|---|
| XXIO | Jul-21 |
5 |
$30.13 | 33% | |
| XXIO | Jul-21 |
4 |
$30.03 | 33% | |
| XXIO | Jul-21 |
2 |
$43.79 | 23% | |
| Cleveland Golf | Sep-21 |
7 |
$14.70 | 68% | |
| Cleveland Golf | Sep-21 |
6 |
$18.34 | 55% | |
| Cleveland Golf | Sep-21 |
6 |
$22.27 | 45% | |
| Cleveland Golf | Sep-21 |
3 |
$42.80 | 23% | |
| Cleveland Golf | Sep-21 |
2 |
$44.64 | 22% | |
| Srixon | Oct-21 |
9 |
$11.55 | 87% | |
| Srixon | Oct-21 |
8 |
$11.98 | 84% | |
| Cleveland Golf | Oct-21 |
6 |
$20.06 | 60% | |
| Cleveland Golf | Oct-21 |
5 |
$28.43 | 35% | |
| Srixon | Oct-21 |
4 |
$32.15 | 31% | |
| Cleveland Golf | Nov-21 |
8 |
$14.68 | 68% | |
| Cleveland Golf | Nov-21 |
7 |
$15.94 | 63% | |
| Cleveland Golf | Nov-21 |
7 |
$14.25 | 70% | |
| Average |
5.6 |
$24.73 | 50% |
Methodology
Realeyes uses front-facing cameras, in a privacy-safe manner to measure naturally occurring human response as opt-in viewers watch the videos from their own devices.
The Quality Score featured throughout this study is part of Realeyes’ flagship PreView product, a content intelligence tool for predicting and managing attention in advertising. PreView enables advertisers to reduce media waste by predicting and eliminating low-performing video creative and boosting strong attentive creative.
Quality Score is reported on a scale of 0-10 and measures a video’s ability to perform in market. It includes three critical elements:
- Capture: Ability to capture audience attention in the first seconds.
- Retain: Ability to retain the audience throughout the ad.
- Encode: Ability to encode the brand message into the brain through emotional engagement.
In addition, we assessed the creative for a qCPM, which is the average cost per 1,000 quality exposures (qCPM) if there were a nominal CPM of $10, and Creative Efficiency, which is the extent that a video creative 1,000 attentive exposure at a cost on par with the cost of 1,000 paid media.
Realeyes tested 16 Cleveland Golf ads from July – November 2021 where each video was viewed an average of 155 times.
We analyzed Realeyes’ performance metrics against several dimensions:
- Duration
- Brand
- Device
- Presence of Brand
- Presence of Product
- Presence of Human
- Human Interaction
- % Time Presence of Product Human Interaction
Coca-Cola - ‘WOAH Content’
Vice’s creative agency Virtue, conducted robust large scale content testing for Coca-Cola’s “Where Only Awesome Happens” - discovering what assets to drop and which markets were most effective.
Heineken - 'Dropped'
Discover how Heineken’s media partner Verizon, found which content resonated most with the target audience to help guide a more efficient media spend.
Get in touch
Optimizing for Attention to Increase Creative Efficiency
Challenger Food Brand Optimizes Creative Attention to Increase Efficiency 60%
A global challenger food brand sought to improve campaign performance and drive business insights through continuous pre-launch and in-market creative attention testing.
Measuring Attention and Engagement of Interactive Mobile Ads
Measuring Attention and Engagement of Interactive Mobile Ads
Challenge
Adludio is an AI-driven technology company that delivers premium mobile advertising.
Mobile ads have the ability to deliver richer experiences to audiences, but the majority of mobile advertising remains static.
Adludio’s Sensory Ads fuse audio, video, animation, and physical interactivity, prompting users to take action to engage in learning more.
Each ad has at least two parts: the set-up and the pay-off, which is a different experience from conventional linear digital ads.
This active rather than passive viewing of ads is geared towards driving greater attention, memorability, and brand uplift.
As a market leader in mobile engagement for nearly a decade, Adludio involved Realeyes to evaluate how user engagement affects audience attention. In particular, Adludio wanted to measure the performance of these Sensory Ads and their ability to Capture attention to encourage user engagement and Retain attention once they’ve interacted with the ad.
Key Takeaways
- Adludio ads outperformed their product category benchmarks
- Ads that performed strongly in Capture (ability to grab attention within first few seconds) were most likely to perform well in Retain (keeping attention)
- Use of audio in the ads increased Capture by 22% and Retain by 38%
- Static video garners distraction
- Optimizing attention via techniques such as editing and adding subtitles, increased creative efficiency by 60%.
“Realeyes have proven to be essential in validating viewer attention and engagement levels of our Sensory Ads”
Paul Coggins, Adludio CEO
Solution
Adludio partnered with Realeyes to validate their hypothesis that viewer interactive engagement is the ultimate attention metric in digital advertising. Multiple ads were passed through Realeyes’ PreView ad-testing. Each mobile ad was exposed to an average of 155 viewers.
Given the native interactivity of Adludio ads, we focused specifically on the Capture (ability to capture audience attention within the first few seconds) and Retain (ability to retain the audience throughout the ad) scores.
Results
The WhatsApp ad, takes advantage of mimicking the mobile app, yielding a 10/10 Capture score – one of the highest Capture scores in the database, along with a 9/10 Retain score.
After the initial animation encouraging the viewer to ‘swipe up’, a video plays a short story that continues to punch above the category benchmark throughout.
Lego notably featured an additional interactive stage, which accumulated attention and viewer engagement as the experience continued through to the end.
Despite being a sequence of simple animations, the combination of instruction and interactivity encourages greater focus on the ad.
Pinterest had a similar set-up to WhatsApp, achieving benchmark against all ads within its category. Once the user interacts, attention rises exceeding benchmark through to the end.
The video also had a soundtrack, those ads that featured audio had a higher Capture (22% increase) and Retain (38% increase) than those that didn’t.
Orbit scored a 9/10 Capture score and the intrugue of the transition animation drove attention above benchmark. However, the ad suffered a poor Retain score due to the final frame being static for too long.
Outlook
Adludio is using such PreView insights to win business from other major brands. In the longer term, Adludio has a vision to optimize creative in near real-time based upon Realeyes PreView data.
Methodology
Adludio used Realeyes PreView to measure the facial cues and eye movements of participants as they watch video from their mobile device. Each Sensory Ad received over 155 views among participants.
Capture: Ability to capture audience attention in the first few seconds.
Retain: Ability to retain the audience throughout the ad.
Download Report
Cleveland Golf - ‘Level Up’
Global golf manufacturer Cleveland Golf wanted a universal way to predict creative impact in advance of campaign launches to ensure revenue growth and spend efficiency. Using PreView it was able to get actionable data that influenced a change in creative strategy.
Coca-Cola - ‘WOAH Content’
Vice’s creative agency Virtue, conducted robust large scale content testing for Coca-Cola’s “Where Only Awesome Happens” - discovering what assets to drop and which markets were most effective.
Get in touch
Flowers Foods’ Early Iterative Testing Increased Creative Efficiency by 25%
Flowers Foods’ Early Iterative Testing Increased Creative Performance by 25%
Challenge
Flowers Foods strongly believes in iterative creative testing to provide insight from concept to production – informing decisions across the company and its agencies.
Historically, Flowers found testing to be a long and expensive process that hindered frequent iterative testing, risking investing in less successful concepts.
Using Realeyes, Flowers could get both quantitative performance scores for selecting the strongest creative based on audience attention and emotional response, but equally unearth the qualitative insights behind them too at speed.
Key Takeaways
- Selecting the best animatic was made simple through testing and avoided the cost of producing weaker concepts
- Testing ad production early surfaced how creative elements affected audience response – decreasing negativity by almost 12%
- Iterative testing garnered positive results each time, informing decisions to ultimately drive overall performance by 25%
Iterative testing garnered positive results each time, informing decisions to ultimately drive overall performance by 25%
Solution
Flowers Foods used Realeyes audience attention and emotional response measurement to test their Nature’s Own’s fairy tale- themed campaign. Creative testing began with rough cut animatics based on five different iconic fairy tales, using a voice over to explain the thrust of the creative. Early testing immediately surfaced the strongest performing creative, avoiding the costly path of producing underperforming concepts.
The report included key emotional KPIs, attention, survey, and sentiment analysis data. The second-by-second analysis revealed which scenes garnered attention or even elicited an emotional response such as happiness, fear, sadness, or contempt. Whereas the qualitative survey data provided additional feedback including the remembered experience: Brand recall, Likeability and Shareability. This approach provided clarity on concept selection, enabling them to go into full production with confidence.
Animatics
Out of the five animatics, The Red Riding Hood creative performed the best for Attention levels, along with high levels of happiness and engagement. The peaks of Attention revealed that the wolf scenes gripped the audience and that the plot twist (where the wolf turns out to be friendly) piques audience Engagement. The second-by-second analysis also surfaced opportunities to edit scenes where Attention and Engagement fell flat.
second-by-second analysis surfaced opportunities to edit scenes where Attention and Engagement fell flat
| Concept | Score | Attention | Happiness | Engagement | |
|---|---|---|---|---|---|
 |
Red Riding Hood | 73 | 70.8% | 17.5% | 27.1% |
 |
Cinderella | 67 | 70.7% | 11.7% | 20.8% |
 |
Hansel & Gretal | 63 | 64.4% | 15.8% | 22.8% |
 |
Jack and Beanstalk | 61 | 65.7% | 12.6% | 20.1% |
 |
Snow White | 58 | 69.6% | 11.2% | 18.4% |
Mid-Production Test Surfaced Opportunities to Drive Greater Positive Engagement
Using these insights, Flowers and their creative agency began producing a 30-second spot for further testing. As expected, the wolf scenes elicited fear and peak negative responses, but this also retained attention and high levels of engagement. It was other creative elements such as the voiceover, music and lingering dark scenes that provided an opportunity to optimize to drive more positive engagement.
Final Edit Test Revealed a 25% Increase in Positive Engagement
The second iteration had tighter scene edits, a new soundtrack, a different voice-over, and additional sound effects to make the ad more light-hearted. As a result, happiness increased 30% for the improved concept and negative reactions decreased 12%. The survey results also found that positive reactions increased 25% and negative reactions decreased 49%.
| Concept | Score | Attention | Happiness | Engagement | |
|---|---|---|---|---|---|
 |
Mid-Production | 60 | 78% | 3.7% | 6.5% |
 |
Final Edit | 75 | 81% | 4.8% | 5.7% |
| Change | +25% | +3% | +30% | -12% |
Outlook
Flowers work on iterative creative testing has led to 12% increase in online video media effectiveness and 68% in on TV media effectiveness, as determined by Nielsen. It will continue to test its animatics and creative using Realeyes platform.
The success of the Nature’s Own has also led Flowers to conduct an audit of creative for brands Nature’s Own, Wonder, Dave’s Killer Bread, and Tastykake).
We found that shorter ads (0-10s) tend to maintain a higher percentage of the audience through to their completion. While longer ads (10-20s and 20-40s) retain a smaller percentage of the audience, they can elicit a stronger emotional reaction and increase recollection.
Since 15s spots have lower executional costs with minimal impact of effectiveness, Flowers prefers shorter ads for brand and product creative, while using 30s for innovation launches and complex benefit communication. All valuable insights that will be used to inform upcoming campaigns.
Flowers work on iterative creative testing has led to 12% increase in online video media effectiveness and 68% in on TV media effectiveness
Methodology
Realeyes uses front-facing cameras and computer vision to measure opted-in and privacy-safe audience attentiveness and engagement as they passively view Flowers Foods video content on their own device.
Participants were given pre-viewing and post-viewing survey questions for qualitative responses relating to Brand Favorability, Purchase Intent and Brand Recall.
5 animatics were tested in August 2021, with an average of 240+ views per video by male and female viewers aged 18-69 on desktop and mobile.The first iteration of the Red Riding Hood ad was viewed in December 2021, and the final version in January 2022.
In addition to the fairy tale campaign, Realeyes ran a brand audit, including 28 Flowers Foods videos from 2020 and 2021, across 4 brands (Nature’s Own, Wonder, Dave’s Killer Bread, and Tastykake). Each video had an average audience of 240 viewers. This portfolio intelligence provided brand specific benchmarks alongside other industry norms.
Coca-Cola - ‘WOAH Content’
Vice’s creative agency Virtue, conducted robust large scale content testing for Coca-Cola’s “Where Only Awesome Happens” - discovering what assets to drop and which markets were most effective.
Heineken - 'Dropped'
Discover how Heineken’s media partner Verizon, found which content resonated most with the target audience to help guide a more efficient media spend.
