Share
What You’ll Learn
John Marling has been conducting market research for almost four decades. In this episode, he shares insights on what marketers and communicators need to think about in order to get meaningful, actionable data from their research. Learn more at pioneer.coop/surveys.
Guest Speaker
John MarlingShow Notes
Transcripts have been lightly edited for clarity and readability.
Intro: A production of Pioneer Utility Resources. StoryConnect, helping communicators discover ideas to shape their stories and connect with their customers.
Andy Johns: What are some of the questions you need to ask yourself before you start doing any market research? That’s what we’ll be talking about on this episode of StoryConnect: The Podcast. My name is Andy Johns, your host with Pioneer, and I’m joined on this episode by John Marling, who is the founder and president of Pulse Research. John, thank you so much for joining me.
John Marling: I appreciate the opportunity to get together this afternoon, Andy. Thank you.
Andy Johns: Absolutely. Now, John, like I said, is founder of Pulse Research. We’ll talk a little bit more about that in a moment. They’re Pioneers market research partner, and we’re excited about everything happening there. But John, let’s start right where that question kind of opened up, because you’ve been doing this long enough to have seen the good and bad of it. What are some of the things folks need to consider before they jump in and start doing market research?
John Marling: Great question, Andy. And we always, and we pride ourselves, in getting actionable information that doesn’t sit on the shelf. Now, the way we accomplish that is the very first step before we start anything with the research is getting together with you guys and really putting your feet to the fire. What do you want to find out? What you want to learn? What action steps? What decisions do you need to make, you know from the research? You know, so that there’s a very clear objective and mission so that then when the research is completed, you will have information that, as I said a moment ago, is actionable and usable and meaningful for you. And so, we make sure that in that first step, we ask you the question, what do you want to learn? What’s most important to you? Do you have any decisions? You know that your board is going to be making? And are there some standard elements relating to, you know, client satisfaction? But in quick summary, we make sure that we clearly identify together what your priorities are, what your needs are, what action steps you want to get out of the research.
Andy Johns: You touched on a lot there, and particularly asking what the board might want to know, I think is a tip worth coming back to. But I’ll preface this interview, and John and I were talking beforehand, in my undergrad and graduate program, I think I wound up with three C’s overall, and two of them were in the two statistics classes that I had to take. So, John, I’m relying on you for quite a bit of this, but that’s good because when it comes to experience doing surveys, you’ve been doing it for a while.
John Marling: Yeah, 39 years, I’m proud to say, doing thousands and thousands of surveys. And speaking of sample size, you know, in this type of research where you want to get some insights to make decisions, to get some clarity in terms of where you stand, over all the years, you know what we’ve advised, what we found out is important, is that either ends of the spectrum. In other words, decision making research is what’s important. You know, what’s the takeaway? And what do I mean by that? What that means is – and I’ll just give you a for instance. If nine out of ten respondents to the survey say you’re doing a heck of a job, you’d agree with that. That’s meaningful, right? You know, if one out of 10 or 10% say you’re doing a good job, meaning 90% say you’re doing a bad job, that is also quite meaningful, isn’t it? And so –.
Andy Johns: The truth hurts.
John Marling: Exactly. But either end of the spectrum is where you make the decisions. Okay? So if your board is making a decision on a new facility or something – and we’ve had those questions come up in the research we’ve done – you know, nine out of ten people say stay with and just remodel the current facility. That is obviously insightful, you know, in terms of direction. You know, and but coming back to sample size, you don’t need for this type of research, you know, a sample of 1,000, because, once again, you’re making a decision based on either end of the spectrum. And so many of the cooperatives have 1,000 members. Well, the gold standard in research is 400 completed interviews. Well, you’re not going to get 400 out of 1,000, a 40% response rate. You know, so we suggest 10%, you know, 10% of your member base, you know, which a thousand, you know, members would be 100 sample. But if 90 out of those 100 say you’re doing a great job, once again, you’re getting the results you want. You know, the insights you want, not the results, but the insights you want from the research.
Andy Johns: And I think that’s a really important piece because like you said, some of the co-ops out there, especially, you get some, I think in Iowa, there’s some with 700 members, there are a lot of them in that, you know, 1,000 or 10,000. And there are some that are huge. There are some with over 100,000. But the research, the techniques, all of that, it can be applied whether you’re talking about a real small organization or a much bigger one.
John Marling: Absolutely. Speaking of the research, one of the key elements that you know, we advise on and assist with is making sure that each of your members have an equal opportunity of participating in the survey. And what that means is you don’t just, you know, put out one invitation in your newsletter. You use multiple channels available to you. So you’re getting a diversity of outreach, and therefore getting a diversity of your members, both young, old. You know, members who have been a member for a year. Members have been, you know, for 40 years. So using all the channels available to you to get a valid representative sample response from your membership.
Andy Johns: Perfect. And I think that’s important to bring up. I want to kind of shift gears a little bit. So if we’re talking about, you know, the ways to promote surveys, the audience, the sample size. Once somebody determines, okay, there’s some things that we need to know. And this is something that we have struggled with because every other year we do a Pioneer member survey that we ask our members and our clients to tell us about. Just like anything else, one department may have these 3 or 4 questions to ask, and then another department has 3 or 4 more. And then, you know, the CEO wants a few more. The board wants a few more. All of a sudden you’ve got a 50 question survey. How have you tried to help folks zero in on the key questions? Is there a magic link to, you know, “no, you can’t go above X number of questions.” And then how do you fight that survey creep? Because sometimes when you’re asking good questions and other folks may have more, like you’re saying actionable things they want to do, how do you help kind of keep folks focused on what they want to know for the survey?
John Marling: Once again, a very good question. But in that process of the initial meeting, we establish priorities. You know, what is most important? Because obviously, like you just stated, there’s a lot of stakeholders, you know, who want to participate in the survey, obviously, with questions that are germane to them. So one of the things we ask is, have you done a survey before? You know, if they haven’t, then we want to touch all the bases. What do we mean by that? You know, you’ve got questions relating to satisfaction, questions relating to perception, questions relating to needs, demographic questions, etc. Communication, which is an extremely important set of questions relating to the takeaways, the action, you know, from the research. So in answering your question, Andy, if it’s a first time survey, we’d want to make sure and our advice would be, you know, to prioritize and to touch each one of the primary elements, you know, of a survey. If they’ve done a survey before, what questions do you ask before, so we can do some benchmarking, you know, to your responses from before. You know, the good news is though, Andy, this is not a phone survey. You know, in a phone survey, you’re severely limited by time, number of questions. Okay. Particularly in today’s world, you know, you don’t do so in phone surveys because nobody wants to answer them.
John Marling: And Pulse was one of the pioneers of doing online research many years ago. And so the good news is when you doing an online survey. And here’s the other positive, in this case, you’re doing a survey of those who have a germane interest in the survey. They’re members, you know, of this utility, or they’re members of this broadband provider. And so they’ve got a vested interest in doing the survey. So you’re going to get a much higher response rate. So we at Pulse are less worried about the number of questions because the people are interested. And they will take the time because it’s their survey, their utility, etc. And also, you know, Pulse is not going to limit per se the number of questions. We’re going to do the questions that are needed to get the results that you’ve prioritized when we have that initial meeting. You know, so it could be 40 questions. It could be 60 questions. It could be 50 questions. You know, the importance here is getting, you know, asking the questions to get the information that are a priority for you and knowing where you stand and what decisions you’re going to be making, etc.
Andy Johns: Got it. That opens up a few other questions that we haven’t talked about before. But is there a frequency? Can you over survey? And I know this may be a loaded question for a guy who makes surveys for a living. But, you know, are they better off if there are those kind of disparate requests from different departments, would they be better off doing a number of short surveys throughout the year instead of one longer one? Or do you hit a point where if you’re surveying too frequently then the response rate starts to drop? Have you seen anything to point you to a direction there?
John Marling: Well, once I get a good question, it goes back to what have you done before? What are your objectives? So, you know, in a pretty first phase survey, you do it maybe a year to two years afterwards. Maybe a year to see how you’ve improved in areas that maybe you were soft. But let’s say, you know, a particular client, you know, came to Pulse, this utility had a specific decision or a specific area. Maybe they were thinking about offering a new service. You know, we found out that in that initial survey, you get some pearls of insight that beg a follow up survey to get more detail. And so it really depends on the client and the focus objective of the research in terms of the follow up and, you know, kind of the frequency thereof.
Andy Johns: Got it. You talked about it earlier that a lot of the folks taking these surveys, particularly with the co-op, with the PUD, with, you know, a telephone cooperative where they are members, they have a stake in it. Maybe that’s incentive enough. But with some of the other surveys you do, and you see things like contests to, you know, win certain things. Do you guys normally recommend doing incentives or does that help at all?
John Marling: Even with all the differing types of research Pulse does, this is the one that is, how should we say, easier. Because they have a stake hold. They are members. And I want to put it out. I’ve been a member of a utility co-op for about 49 years.
Andy Johns: That’s right. You know it firsthand.
John Marling: I would love it if they would do a survey. I would immediately fill it out. But you know, either the good, bad, and indifferent. You know, I’m a member, and bottom line is it’s my utility. Okay. But to answer your question, we normally suggest you do a thank you, an appreciation. You know, a drawing for, you know, maybe a Visa card, $100 Visa card or something that’s universal. Maybe $100 in groceries or something, just to say thank you. We appreciate you being a member. We appreciate you being a client customer and for taking the time. It would be less important, in this case, to enhance response. But I think it’s important to say thank you. We appreciate you being a member, a customer you know, of us.
Andy Johns: Perfect. So as we’re kind of doing an overview of the surveying process here, are there things that you’ve seen and maybe it was early in your career, and you’ve learned those lessons now. Are there some common things that kind of lead to those follow-up surveys, or the people do the survey, and then they wind up thinking, oh, I wish we would have asked about this? Or wish we would have asked about that? What are some of the common things that come up that you’re able to help folks realize, now, before we launch this, don’t you want to consider A, B, and C?
John Marling: I think in response to them. If I’m understanding your question –
Andy Johns: It was not my best question, to be clear. It’s a little intimidating asking questions to who a person who is a professional at asking questions.
John Marling: In every survey we’ve done, in thousands of surveys we’ve done, there’s always every single time an, oh my gosh. And I call those pearls; things that you don’t expect.
Andy Johns: Okay.
John Marling: And also in the surveys we do, there’s a lot of reinforcements. Well, we thought that, and now it’s nice to have that reinforced. Okay? So now in response to your question, in those aha’s that come up, those pearls, that’s where they normally, clients want to follow-up and get more detail, get more insight, get more depth, you know, to take an applicable response. Because in the first time you do a survey, it’s the broad brush. You’re getting feedback on satisfaction, perception, needs. You know, communication. You know, type relating questions, effectiveness of communication. And then when you go in maybe a phase two, you dive deeper because of what you learned from the first level. You know, so that’s, and so yeah, every single time we find out the aha’s we point what we recommend you do about it. And it ends up being a very, very positive because it helps them overcome a weakness and turn it into an opportunity.
Andy Johns: You mentioned another example was about a facility being built. And again, we don’t want to identify anybody, but are there any other examples for the kinds of things that the utility organizations have wanted to know about?
John Marling: Well, you know, at a board level. You know, I’ve made presentations of the results of the utility survey to the board. And I think that has been very positive for all the stakeholders involved, because, number one, it reinforced, in this case that the satisfaction of the services was overwhelmingly positive. But number two, it pointed out some areas, of softness that the board was very, very positive to find out. You know, they could respond. And then of course, in all the surveys we do, we ask, what decisions do you need to make? What decisions do you have on the horizon in terms of services you’re thinking about or offering, or facilities that we’ve asked about here? You know, there’s a range of – how should we say – as the utility industry looks forward in time, a balance of renewable versus legacy power sourcing. You know, so questions relating, querying their membership as you look into the future, what are they willing to pay for, not pay for? You know what are they willing to commit to, etc. So at a board level, those are phenomenally beneficial insights. And what I’m thinking about it before I forget, one of the things that we do in the survey is ask a couple of open-ended questions.
Andy Johns: Oh, okay.
John Marling: And oh my gosh. Oh my gosh, Andy. Do they open up. And that’s one of the benefits of doing an online survey. When you’re doing a phone survey, you’re limited time. But an online survey, you ask an open question about, you know, what do you think about x, y, z and let them just respond, just go on their little keyboards. It is awesomely insightful to really digest and really analyze, you know, the gist of the open-ended responses.
Andy Johns: Well, and that’s interesting. And I’m glad that you took us there, because having done a few very basic surveys, not anything to the level that you guys do, but. You know, you can see a shape, on the the curve, on the responses to be positive or, you know, whatever it’s showing. It’s easy with the quantitative numbers are on there. But it can be tough to me, when you, the open-ended responses, you know, it can show that, like you said, nine out of ten are really happy with this. But one comment down there, in the open-ended part that says, you know, is there anything else you’d like to add? That’s negative, sometimes that rings louder in our ears than what the statistics would show up above. And maybe that’s just the questions not being crafted properly when I’ve done it. But how do you weigh that? How do you weigh those open-ended requests versus the bigger picture statistical data coming in from the survey?
John Marling: It’s through experience, recognizing that people are people and some people are overly positive, some people are overly negative. And so you’ve got to be able to sort out, you know, as you’re analyzing the open-ended responses to really identify the true meaning. And then – and this is important also – then you qualify that or corresponded correlate it to, you know, the specific data set relating to satisfaction scoring, like one out of five. And so there’s no clear cut, Andy. You know how do you – it’s based on the experience and knowing that human beings will be there. There’s a group who are over positive and a group who are over negative. And you’ve got to sort through that and then compare it to, you know, the statistical, you know, results and come up with a recommendation. But I’ll tell you, the clients love the open-ended responses.
Andy Johns: I’m sure that kind of puts flesh on it, for sure. Well, John, just a couple of other final questions for you here. What are some of the trends? Obviously, you’ve been doing this almost 40 years. What are some of the things that are different about surveying and doing member research, market research now? You know, is everybody’s attention span really short as we think? Or what are some of the trends and changes and maybe where even do you see it going, just as you’ve seen the changes in the way that we do market research or people’s attitudes towards it?
John Marling: I think in response to that, Andy, is there’s a lot of – and I’ll be quite blunt – a lot of entities who think they know it. They know their audience, and they know their members, etc. And the reality is, in today’s world, it’s a very complex world. We unfortunately don’t. And so I mentioned a moment ago and in terms of, in response to your question, the important reason to doing research today is you’re going to get reinforcement, you know, to those areas you think you know. Well, I thought we knew that, and it’s been reinforced. So you know then that management team or the board is much more confident to move forward. But then there’s always the ahas, the unknowns. You know that you learn from that research, you know moving forward. And then taking it to the next level, and I’ve thought a lot about this because our commitment is to actionable information to be a real catalyst, you know partner not with just Pioneer, but with the individual utilities to be able to provide information, to be able to assist in the communication to those members. You know, being a member of a co-op, it’s important, you know, for me, you know, and I was doing a presentation a while ago to a board of one of our utilities. We had just completed the research. And I’ve been a member of this co-op for, you know, almost 50 years. And what I was shocked to find was in the presentation to the board, that there is a high probability of electrical outages. That there isn’t a reliability of power sourcing, and there could potentially be rolling brownouts. And I go, oh my gosh, I mean, what does that mean to me in my, you know, co-op? Point being in response to your question, in today’s world, you can’t take things for granted, and you’ve got to have an eye to the future because the world is changing and, you know, power sourcing, the balance of green versus legacy, you know, power source is very important. But what the real element here is what the real takeaway is, Andy, how do you communicate that? How does that public utility, the co-op, you know, communicate to its members? You know, not only the power reliability, but, you know, a fee increases, price increases. You know, and that’s why I think it’s really important in answer to your question is you find out so you can be able to – and this is important – tell your story. That’s what you know Pioneer is all about. You know, tell your story.
Andy Johns: I’ve heard that a time or two before. Yeah.
John Marling: You know, the research is discover your story. Discover your story so you can communicate that story. I mean, that’s critically important because members of these, it’s their utility. Okay? And they feel that the utility board, that utility leadership, you know, has a responsibility to communicate to them. And so in response to your question, I think it’s more important than ever for, you know, those reasons to reach out and not just assume you know, but to find out from your members where you stand. And also dip your toe into the future, so you’re able to effectively communicate.
Andy Johns: Perfect. I like that a lot. Last question I have for you. Obviously I would love folks to go to pioneer.coop/surveys. You’ll get a link to case studies and the other information to get in touch with with us and with John and everybody, pioneer.coop/surveys. We would love for them to do the market research, with us, with you, through this partnership. But whether they do or whether they don’t, what’s the advice that you would have for somebody who’s looking to do market research? Who’s looking to know a little bit more about their story, like you were saying. What are some of the things that they need to keep in mind for that?
John Marling: Like we, and I think it’s worth restating. What do you want to learn? Know what the actionable areas of information before you start. Because the worst thing that happens is you just, you know, do the research, the time, effort, expense, and it just sits on the shelf. You know, in this world today, it’s too important. You have to know. And so if and when you do it, make sure you have clear cut objectives that your whole team is on board and what you want to get out of the information so that you do, in fact take action from it. And that’s the mission of Pulse is, is to provide information that gives you that insight, that gives you, you know, that reinforcement, if you will, in that areas of aha, that you didn’t know those pearls to take action.
Andy Johns: Perfect. That’s good advice from a long career of doing this sort of thing. So it would be good for us to take heed. John, thank you for joining me.
John Marling: Thank you, Andy. I appreciate the partnership.
Andy Johns: He is John Marling. He is the founder of Pulse Research, founder and president. I’m your host, Andy Johns with Pioneer. And until we talk again, keep telling your story.
Outro: StoryConnect is produced by Pioneer Utility Resources, a communications cooperative that is built to share your story. StoryConnect is engineered by Lucas Smith of Lucky Sound Studio.