Getting regular feedback from your organization can help the data team to prioritize the work that will have the biggest impact on your stakeholders. However, it’s hard to turn ad-hoc feedback into action. A recurring survey about the work of the data team can provide insight into quiet successes, pain points, or areas of confusion, and over time enables you to measure improvements against a baseline trend. Whether your team is in the early stages of building tools and infrastructure or is well-established with a track record of success, the feedback from regular user surveys can help you prioritize maintenance and infrastructure work based on real user impact.
Data team surveys can help you understand your users
Data teams support a wide range of stakeholders, with sometimes very different levels of engagement. Your team may have some stakeholders that you work closely with on a weekly or even daily basis, and some stakeholders who rarely interact with the data team, but who rely on existing reporting or other data products to achieve their goals. It can be hard to get systematic feedback on how well the data team is supporting those stakeholders solely from day to day interactions. One-on-one conversations are great for getting deep feedback, but are very time consuming and better suited to interactions with very close stakeholders. Sending a regular survey provides all users of your products an opportunity to provide feedback, and provides the data team a chance to understand their impact and the limits of their work.
Of course, a survey will only capture part of the story – the data team should integrate this user feedback into their broader understanding of the highest impact projects the team could work on. As with any form of stakeholder input, just because a few vocal users are asking for something doesn’t mean you should prioritize it. But broad feedback from a survey can be particularly helpful to understand what maintenance and enablement projects can have a wide impact on your users’ experience. Are users getting the value they should from existing resources? Do they struggle with slowness or performance issues? Do they need more training, more organized data content, or other resources to enable them to get more value from your products? These all tend to affect a large number of users, and be particularly visible in survey trends.
Start with evergreen questions on data usage and comfort
Begin with a base of consistent questions that you ask every time you send your data survey. These questions should be simple, focused on the team’s core areas of focus, and applicable to most users who will receive the survey. Choose a few questions, and ask them each time you send the survey so you can begin to understand trends over time.
The exact questions will depend on your team’s areas of focus, but here are a few examples (designed to use a Likert scale):
Data usage and reliability:
- I rely on data to make decisions in my job.
- I can easily and independently answer my own questions with data.
- I am confident that I understand the meaning of the metrics and data fields I use.
- I trust the accuracy of our reporting.
Tool-specific knowledge and training:
- I am satisfied with my ability to use [our self-service reporting tools].
- [Our reporting tool] has reporting on everything I want.
- It’s easy to find reports on specific topics I need to understand in [our reporting tool].
- I am able to quickly access the data I need in [our reporting tool].
Data team support:
- The data team is a reliable source of information about our data.
- The data team helps me understand our data and how to access it.
- The data team is responsive to my requests.
- The data team provides valuable thought partnership to me.
- The data team understands the business context of my questions.
- I am satisfied with the speed at which my data questions are answered.
- I am satisfied with the performance of the data team overall.
Add questions related to recent or upcoming data team changes
If you have recently made major changes to the tools, training, or level of support offered by the data team, or if you are planning to, the regular survey can be a great way to get more feedback from users. Are you thinking about launching a data training program? Ask how satisfied users are with their ability to use the data tools, or whether they would find value in more training on various topics. Are you prioritizing a cleanup of old reporting? Ask if users can find the reports they need, or if they trust that reporting is current and relevant. These can help prioritize these projects (“Wow, the Sales team really wants more training on how to create dashboards, let’s start the new program with that!”) or can help you better understand the impact of your work (“Our cleanup project increased satisfaction with reporting relevance from 3.2 to 4.5!”).
Leave room for open ended input on issues and stumbling blocks
Give respondents space to provide open-ended feedback. As is common with surveys, many respondents will skip open ended questions. However, the responses you do get will often provide helpful context around low scores, and sometimes will be the opportunity an introvert was looking for to provide very thorough feedback.
Here are a few open ended questions that may be helpful, depending on your areas of focus:
- What are the biggest impediments to your ability to use data in your day to day decision-making?
- What are manual tasks you complete on a regular basis that could be improved with automation?
- What questions or requests have you encountered that our current products are not able to answer?
- What is your favorite dashboard?
- What else do you want the data team to know about how you use data, or how you *wish* you could use data?
Collect enough user information to segment your survey results
You’ll probably have a relatively small sample size, so don’t think that you’ll be able to uncover meaningful differences between nuanced user groups – for example, specific cohorts within a single team. However, you will hopefully have enough responses to see some very high level differences, like Sales users are generally more satisfied with data team support than Marketing users. That kind of information can sometimes be very helpful in prioritization, or in understanding the root cause of changes in your baseline responses. Focus on the differences you believe are most relevant in a person’s experience with the data team.
Here are a few segmentation questions that may be helpful, depending on the user types your data team supports:
- What team are you on?
- How long have you worked at the company?
- How often do you use [a core tool]?
- How comfortable are you [writing SQL or performing another relevant technical task]?
Pulling it all together – our sample data team survey
Of course, we want you to get your survey going as quickly as possible. Here is an example of a complete data team survey that you are welcome to use as a jumping off point. Make a copy, modify for your team’s needs, and get ready to send it!
Pick a cadence that makes sense for your goals
With surveys, there is a tradeoff between frequency and depth. You may be able to get a meaningful portion of your users to respond to a long survey once, but probably not quarterly. The more often you send the survey, the shorter it should be. I have found 2-3 times a year to be the most frequent cadence that is useful for my team, given the pace of company growth and hiring as well as data team changes. That cadence enables us to understand improvements over time without overwhelming our respondents. If you are rolling out major infrastructure changes or new tooling, you might want to time your survey to give you more useful data about that endeavor or target a specific team that you’ve been partnering with more intensely.
Be prepared for humbling results from your data survey
Hopefully, you have a pretty good understanding of the biggest issues facing your organization’s ability to access and leverage data. You may not be surprised at all by the results. However, the anonymity of a survey can sometimes unveil surprisingly strong feelings, especially in companies that otherwise have a culture of niceness. This is good! It’s much more helpful to have an unvarnished perspective on the issues so you can get started. If you see more negative feedback than you expected, don’t take it personally, take it as an opportunity to fix the real underlying issues and keep surveying so you can see the improvements over time.
Share your results – and act on them via the data roadmap
Sending a survey just to know how you’re doing isn’t particularly helpful or necessary for most small teams. If you aren’t going to let the survey results impact your team’s roadmap, save yourself the effort of sending it!
Once you have the results, do some quick analysis. You don’t need to go too crazy, but go deep enough to understand the primary pain points, any specific segments who tend to have those pain points, the areas where you’re doing very well, and the company’s overall satisfaction with your team.
There are some limitations to surveys, and you can watch out for major sources of bias (like one team being highly over-represented, or getting far more responses from newer colleagues than from tenured colleagues). If the results are wildly different from what you expected, do a little bit of digging to understand if you’ve uncovered a real issue, or if the results are due to the survey itself instead. But this is a situation where good enough is good enough, and listening to your stakeholders and taking action to improve is better than aiming for perfection.
Share the analysis with the data team and determine whether this changes any areas of focus, or if work that is already planned will address the issues. Summarize the actions the team will take based on the survey results, and share them back out to the folks you asked to take the survey so they understand the impact of their feedback. Envoy shared a great success story at a dbt meet-up of how their recurring user survey drove major changes in how they work.
Hold yourself accountable to reporting back on what work has been done before you send the next data team survey. As the company starts to see a positive feedback loop from their responses, you’ll see more and better responses to the survey.
Special thanks to Matt Mazur and Anna Pritchard for contributing to this post via conversations in Locally Optimistic’s Slack. Do you regularly seek feedback from your users? Have you ever gotten any surprises? I’d love to chat more about what you’ve learned!
