Kenshoo is a SaaS Marketing platform that helps clients efficiently and effectively manage their search engine and social marketing campaigns across the major online publishers (Google, Facebook, Instagram, Yahoo!).
Overview
On a recent Product Survey (read case study), clients rated themselves as “dissatisfied” with the Kenshoo’s reporting capabilities.
I lead a research study that involved both generative observational field studies and evaluative task analysis studies to uncover what was causing clients’ headaches with the existing capabilities, market research to understand the competition, as well as potential opportunities to inform the product roadmap. I also provided detailed wireframes, user flows, and designs to help facilitate and increase time to value.
resulted in a
+18% YOY increase
in Client Satisfaction
Objectives
Understand the what, why, and how of client reporting so that we can develop a product strategy and design that centers on differentiation and innovation.
Deliver actionable insights and recommendations to shorten time to value for our clients.
To improve the usability and overall client satisfaction with Kenshoo Social’s reporting capabilities.
MY Part
Qualitative Research
Contextual Inquiry
Task Analysis
Field Studies
Remote Studies
Competitive Analysis
UX Design
User Flows
Mockup Designs
TEAM
User Researcher (me)
UX Designer (me)
Product Owner
Product Manager
UX Team Lead
Visual Designer
Lead Engineer
Timeline
4 weeks (research)
2 weeks (design + testing)
BUT FIRST, A NOTE
To comply with my non-disclosure agreement, I have omitted and obfuscated confidential information and in particular the report findings in this case study. All information in this case study is my own and does not necessarily reflect the views of Kenshoo.
The Problem
On a recent Product Survey, clients rated themselves as “dissatisfied” with Kenshoo’s reporting capabilities.
After two successive rounds the product survey showed a less than ideal score on reporting—2.8 (dissatisfied) out of 5 (extremely satisfied). Eeek! Together with the Product Leadership and design teams, we decided that further information would be needed—more specifically we needed to know more from a usability and task perspective.
“Pretty poor compared to the Kenshoo Search Platform....Overall the most disappointing feature of Kenshoo Social.”
The Challenge
We needed to improve the overall client satisfaction with Kenshoo Social’s reporting capabilities.
To do so, we needed to clearly understand the what, why, and how of client reporting so that we can develop a product strategy and design that centers on differentiation and innovation.
With the project team, we penned some high-level objectives for the research:
Understand our clients’ reporting challenges, wins, and opportunities
Uncover challenges & opportunities facing Kenshoo
Understand our clients’ end-to-end social reporting workflow and how those might differ by persona
Uncover opportunities for innovation & differentiation
My Role & Team
I wore both the User Researcher and UX Designer hats for the project.
As User Researcher, I architected the study, recruitment, planning and orchestration, study moderation, and synthesized the data gathered from the study into actionable insights for the product and design teams. In addition, I supplemented the research by doing a competitive analysis on key competitors.
As UX Designer, I provided product and design teams with a new UX design recommendation to address critical use cases uncovered in the research.
I was joined on the project team by the Product Owner for Kenshoo Social and Product Manager for Kenshoo Social Reporting. I also collaborated after the research with the UX Design Lead, a Visual Designer, and Engineer to bring the design to life.
Consisting of 4 weeks of research and 2 weeks of design & further testing, the project spanned 6 weeks from project kickoff to presentation in the spring of 2017.
Scope & Constraints
I executed a Mixed method research study To understand the user needs along with identify usability challenges & opportunities.
I conducted 12 observational field studies comprised two distinct portions:
A 60 minute contextual interview to define the what and why of our client’s reporting needs to better help product define a long-term strategy and roadmap for reporting going forward.
A 30 minute task-based usability study to uncover what was causing clients’ headaches within the existing solution. This allowed me to offer a short-term UX solution that would deliver value quickly.
In order to provide product a more complete picture, I also did market research to complete a competitive analysis focusing on key competitors that surfaced in the research.
The Gotchas
As with any project, we had a couple of challenges that we needed to mitigate:
Disparate experience across multiple products.
Due to previous engineering & product decisions, reporting features and user flows spanned not only Kenshoo social, but also Kenshoo Search platforms.
Frenemies with Facebook.
As a Facebook Marketing Partner, Facebook was a friend to Kenshoo as it allowed us to gather and get our clients data, but also a direct competitor through it’s free Power Editor and Ads Manager tools.
Remote product team that spanned oceans.
My project team PM & Designers were located in the Tel Aviv office and were not able to join me physically onsite for the meetings.
Who
The project focused on two of Kenshoo Social’s personas.
From previous research and design work, we knew that Kenshoo Social’s clients were mainly bucked into two groups—each with distinct use cases, requirements, and defining attributes:
Agency Clients - Account or Campaign managers that are part of an agency.
Advertiser Clients - Typically social campaign managers that are “in-house” and embedded in a marketing organization at a company.
What
For this process, I followed the typical human-centered design process:
Research
“It all starts with the user.”
To build an empathy with our clients, we knew we needed to start with research.
Over the course of 4 weeks, I planned the study, recruited participants, conducted the research sessions, and synthesized the data gathered from the study into actionable insights for the product and design teams.
Below are some of the highlights of the research phase.
The research PLAN got everyone on the same page
Together with the project team, we unpacked the why, what, and who of the research and larger project.
I distilled the team discussion into a Research Plan that acted as the foundation for our discovery. The brief included information like the project background, big questions, underlying assumptions, and the goals & objectives we had for the research and the project as a whole.
The research plan was integral in getting everyone on the project team on the same page.
internal Interviews helped perfect the client sessions and build morale.
While the primary purpose of starting with CS is to help me work out the kinks and further refine the approach of the research, performing these interviews also has some really significant side effects:
Provide context - CS Manager will understand what I’m asking of their clients and are better equipped at not only providing recommendations of clients to include in the research, but also introducing me to potential clients with confidence.
Evangelize User Research - With their involvement in UXR, CS teammates often become advocates and become my cheerleader with their clients and internally.
Jump start client introductions - After the sessions, the CS manager has a direct reference and often starts rattling off different clients who they’d think are a good fit for the study.
Make them feel heard & appreciated
“This is so cool. Thanks for letting me be a part of this and ask my opinion….this is awesome that Kenshoo is doing something like this. ”
I gained a deep understanding of our clients reporting needs and use cases through contextual interviews and observation.
I spoke with and observed 12 clients about themselves & observed their experiences building out their own weekly reports.
6 studies where conducted onsite with clients while 6 were conducted remotely using Zoom. When permitted, the studies were recorded for post session analysis, sharing with the wider remote project team, and follow-up with CS teammates.
Market Research shed light on our own weaknesses but more importantly the opportunity to do something different.
I did market research on competitor products that surfaced during our interviews to inform the project team and wider product marketing teams on our reporting position.
For each product mentioned, I completed a full SWOT analysis to include in the final report and presentation. The competitive analysis was crucial for collaborating with Product on the strategic roadmap for Kenshoo’s reporting.
This analysis also enabled me, as the UX designer, to push my designs toward differentiation and away from assimilation.
define
Note: for confidentiality, I’ve omitted and obfuscated some of the data conclusions and findings from the information and visuals below.
Affinity diagrams identified challenges and opportunities.
Reviewing the detailed session notes and recordings, I used the trusty post-it notes and markers to write down common themes, data, and quotes to synthesize the data into high level challenges, changes in mental model, and market insights.
Slides at right are taken from the final presentation that was circulated among the wider product, design, and engineering teams.
RESEARCH REVEALED a disconnect between our clients and our product
Working with the team, we collaboratively reviewed and distilled a list of interesting findings and a-has from the data:
The definition of “Reporting” is evolving. There is a shift in the mental model concerning reporting. While Gen X still widely uses excel and uses offline reporting, Millennial and younger generations expect all of that to be available directly in the UI.
Pivots, pivots, & more pivots. They need to be able to pivot the data in the UI by multiple aspects, but most notably time.
Trust is eroding. Trust is missing from most because what you see in the UI doesn’t necessarily translate to what’s available in offline reporting nor even what’s in other engines.
Creative reporting is a serious pain in the @$$. There’s no great reporting solution which means an opportunity for Kenshoo to innovate.
“...have all the data in the platform and be able to manipulate it, and visualize it without having to export it and do it offline. I should be able to do [all of that] in the platform.”
The task analysis helped to visualize the steps and where to focus.
I mapped out on a whiteboard the average customer journey of our persona to pull a typical weekly report.
For sharing purposes with engineering and other teams, I created a more polished task analysis document (below) that the Product and Design teams could refer to that included insights like: the individual steps, avg. time spent on each step, the tools used, opportunities, and market & competitive insights.
IDEATE
Reframing the problem for better brainstorming and less solutioning.
Using the How Might We framework, I worked with the team to diverge and go broad thinking about the problem at a high-level.
This allowed us to create a solid foundation for a long-term product strategy for reporting and avoid the common trap of diving into solutions too early.
UX Principles guided the design and strategy
Collaborating with the Project team, we reviewed the data and penned some design principles for the overall experience.
These design principles helped the team to envision the ideal client experience working with Kenshoo’s reporting solution—helping to guide the product strategy and roadmap.
All in on Step 3
Using the Task Analysis as a guide, we decided as a team to focus our short-term efforts on Step 3: Run Reports.
Step 3 was not only the most frustrating for the user, but also we could potentially save the user significant time.
How might we...
allow Social Marketer Sally to analyse her social marketing data in flexible and meaningful way both inside and outside of Kenshoo to save her time and energy?
STORYBOARDING created a visual narrative to guide the reporting story.
Given the timezone constraints, I took the lead on the UX design by first storyboarding out the user flow for the persona’s main scenario: weekly daily reporting.
Using the design principles as guides for the user experience, I created a user flow that would provide users with the flexibility to perform the task in the UI, but also export the details for offline analysis and integration to their existing offline reports.
prototype
Rapid Prototyping allowed me to quickly validate the new user flow.
I identified and prototyped the two new user flows for the critical use case addressing the major concerns identified in the research.
The flows mainly differed by where the user started the task & ease of implementation:
Campaign Grid - more frequently utilized, but would pose an engineering challenge.
Analysis Pivot Grid - less utilized screen but easier implementation
TEST
Testing showed The new flow to be intuitive and easy to use.
I quickly tested the flows on internal users as well as external clients to see which flow was more intuitive, easier to use, and addressed the main goal.
Both revised flows addressed the main concern and received rave reviews during testing; however, the campaigns grid approach was voted as slightly preferred since that was where clients indicated they spend the majority of their time.
“Oh my God, this would be amazing! This would literally save me so much time. Seriously? When is this happening? ”
implement
Phased approached lead to quicker time to market.
I collaborated with Engineering and Product to create a phased approach.
After reviewing with Engineering and Product, we opted for the Analysis grid flow given ease of implementation and slated the campaigns grid solution for future enhancements. We broke the final solution into three separate phases introducing more complex features and functionality in each phase.
Results
The research and new user flows resulted in a 18% increase in client satisfaction on the next Product survey.
Additionally, the project was lauded as a success internally within Product and Design. It became a shining example of what user research, product, and design can do if given the right direction and right focus.
“You’ve really outdone yourself. This is the best research we’ve seen to date at Kenshoo.”
Reflections
Hindsight is 20/20. Here are some of the interesting things I learned through this project.
😊 Challenge your personas! The shift in our personas mental models was helpful in working with product on redefining our product strategy.
😊 Market helps both strategy and design. It was a great exercise to do the competitive analysis and really helped not only build out a solid strategy with Product, but also a UX design that’s rooted in differentiation.
😃 Shorten recruiting by leveraging existing surveys! The product scorecard survey was a great way to identify participants because they were already engaged in giving feedback and elated that we followed through.
😞 Narrowing scope means narrowing impact. While the study was primarily focused on Kenshoo Social, it would have been nice to include some additional Kenshoo Search only clients to make it more relevant to the Kenshoo Search team.