Creative Modelling: Using Data Science to Understand Creative Effectiveness

Measuring campaigns against objectives and effectiveness is an important part of marketing. The goal is to understand what worked, what could be improved, and what learnings you can take into your next campaign. Did you know this process can extend to your campaign creative too? Using data science models, we can help answer this big question: how can marketers evaluate the effectiveness of creative messaging on a macro level? We dive into this process below and offer some tips to consider when you’re developing creative for your next campaign.

Why it’s Valuable

First: the why. Brands spend a lot of time and effort developing creative and messaging. It’s a critical part of a campaign, and we want to understand how creative and messaging impacts ad performance on different channels. Creative modelling can be used to find out which ad characteristics drive the most success in terms of awareness metrics like click through rate (CTR), cost per click (CPC), and conversion metrics. A goal of this process is not to declare one ad or set of creative a ‘winner’ over the other but rather to identify commonalities in top-performing creative. This knowledge can then be applied to future creative development.

How it works

Creative modelling leverages Google Cloud’s Vision API model to analyse a catalogue of assets and their performance metrics to detect features within an image.

These features can include (from Vision API’s Feature List):

• Label & Object Detection
• Text Detection
• Image Properties
• Face Detection
• Logo Detection
• Explicit Content Detection (SafeSearch)

For the purposes of this process, the catalogue uses static images and video-thumbnails for creative run on platforms that are not using responsive assets like Google Demand Gen or Performance Max. The nature of responsive campaigns means we can’t see what specific image drove performance within an ad. Channels like Meta are therefore a better option for this process. 

Assets can be further categorized based on their ad format and grouped according to their shared campaign properties, then assigned a custom performance score. This custom score is based on a variety of metrics depending on the KPIs of the campaign. Insights can then be gleaned based on these scores and the annotations assigned to the assets from the Vision API model.

Evaluating Performance & Insights

Overall, we have garnered really interesting and valuable insights on creative performance, that allow for an additional perspective when evaluating creative performance. 

For one post-secondary client, we learned that images with simple, bold type, legible program name and a notation on program time-line (one-year certificate, two-year diploma) performed better than hero assets containing images of people/students interacting with one another.

For a quick-service restaurant it was discovered that imagery with ‘healthy’ foods like salads or sushi resulted in a high click through rate and engagement, though that did not necessarily translate to an actual purchase of a salad or other healthy option, suggesting a potential level of virtue signaling or at the very least interest or consideration.

Tips for Creative Development

After running this analysis on multiple clients, there are some key learnings that I feel can apply to all creative asset developments

• Consider the device the asset will be viewed on – if your user is mobile-forward, consider that the size on an average phone screen is 6.3 to 6.5 inches, which means overcrowded, text-heavy images will likely be scrolled past
• Ensure the asset is visually impactful  in both regular and dark mode
• The image/graphic should be clearly connected to the advertising message, and should not be overly cluttered with too many small graphic details

However, it’s important to remember that Google comes with its own set of biases. While Google resolved the race & gender biases flagged in 2020, the model utilizes pre-trained data sets that rely heavily on human inputs – and we know humans are flawed. In our own running of the model on various clients, we found SafeSearch was particularly egregious, with sometimes flagging red-toned or dark-toned imagery as ‘violent’ or when a clothing item matches a skin tone, as ‘adult’. Therefore, like all tools that use machine learning, it’s important to take learnings with a grain of salt and pay attention to the details.

Final Thoughts

While the machine acts as a great proxy for a user, at the end of the day it is not a human being and lacks contextual awareness and nuance. As a result, you will always need some level of human involvement when evaluating the model’s effectiveness and extracting results when understanding creative performance. 
If you’re interested in learning more about how to use creative modelling, reach out to us.