In an e-commerce company, it's common for the success of creative material used in marketing campaigns to be measured in terms of numbers, e.g. click-through rates and conversion rates. However, just numbers may not give a full picture of what are good creative material.

Lately, we are trying to focus more on qualitative research, where we bring in actual users and show them creative material for feedback.

A hypothesis would be for example: clean and modern looking materials bring back the customers and make them spend more in the long run, opposed to cluttered material with too many elements and information. Besides the obvious AB-test that would be to run, which would be resource-heavy and time-consuming, could we use a qualitative research to test this and get some insights?

Another idea of a test would show customers the website or app with competitors' images swapped in and get their feedback.

I know it's more common for testing out UI and new features, but has anyone done this sort of research for campaign materials before? If so what would be things to test to look for and how would you carry them out?

  • 1
    You've hit on the reason that creative is difficult to quantify. It is also why computer algorithms create such bad music, poor prose, and horrible colour combinations. What are the metrics? How will you choose your "actual users?" There is no task to perform. What is a win? What is a fail. With UX, there are time limits and performance levels. Creative lacks these.
    – Stan
    Commented Jul 10, 2018 at 16:13
  • 3
    For the sake of your designers, please keep in mind that users are generally not great judges of knowing what is best for them, especially in the details, until they've had a lot of experience working with something specifically. Commented Jul 13, 2018 at 2:26
  • 1
    Hi Ameen Akbar, I've edited your question a bit. I added some punctuation and fixed some minor spelling. I have also changed 'creatives' to 'creative material' or just 'material' as I am not sure it is used as a noun in the way you seem to mean it. If you disagree with these changes, you can always edit your question. You can always find me in chat if you want to discuss.
    – PieBie
    Commented Jul 13, 2018 at 10:23
  • 1
    @PieBie Creative was used correctly - mbaskool.com/business-concepts/marketing-and-strategy-terms/… -- it is jargon though so if making it more precise helps I don't see it as a bad thing.
    – Ryan
    Commented Jul 13, 2018 at 10:37
  • 1
    Hmm, I have never seen or heard it used in that sense. Thanks for sharing @Ryan, you learn something new every day. But even in the definition it seems very... undefined. Let's keep it at 'marketing materials' for now.
    – PieBie
    Commented Jul 16, 2018 at 8:34

3 Answers 3


There are market research firms that specialize in this. They started in the days of traditional advertising post WWII and still exist today.

Working with a firm of this type you would hire them to test your hypothesis. They have pools of focus groups that they generally pay with some sort of rewards for their time which they cover plus some by what they'll charge you.

One of the most basic forms of this from the golden era of publishing is showing a bunch of magazines and ads to a focus group under a time constraint then see what they remember.

Here's an article from AdWeek on the subject that's pretty informative: https://www.adweek.com/brand-marketing/why-traditional-focus-group-dying-82346/

The same type of research can be done on nearly anything including websites and UX.


My ideal research tool would be some kind of electroencephalography AI driven apparatus to measure the brain response to different inputs and feelings... But the technology is still on diapers and for visuals only https://www.youtube.com/watch?v=6FsH7RK1S2E

But we can probably work with less. There are some ways to measure the movement of the eye and see where the user is focusing and for how long. The problem is that with only this information, some time on a zone could either mean "Oh, I like this" or mean "what the hell is this", that is why we need different additional inputs. https://www.youtube.com/watch?v=XsCkaUIUzRQ

Probably a more sophisticated eye analysis would also measure heart beat rate, blood circulation on the face, etc, to make sense of the emotional response. https://www.youtube.com/watch?v=e9ASH8IBJ2U

But for most mortals, a card and a question asking which one do you like more are the tools we can afford... probably an automatized webpage is at our range too.


You must find a performance-based standard creative problem to solve that allows a comparison of the result to be evaluated by a standard user.

It doesn't exist, yet.

We've been working on creative evaluation since hand paintings were painted on the walls of caves found in Altamira, Spain and Lascaux, France 20,000 years ago.

There are some who claim to be able to test for creativity and creative solutions but most of it is bogus.

Sales is the most reliable metric for advertising; but, as the saying goes, "Half of every advertising dollar is wasted; but which half is that?"

  • Hi Stan, I think there's been some confusion. I think the OP was not asking to measure creativity as such, but rather the impact of creative materials (eg as is done through focus groups).
    – PieBie
    Commented Jul 13, 2018 at 10:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.