Video:

Slides:

Risk Rating & Logo Churn Analysis (Excel Template)

Full Transcript

Justin Edwards
Joining us just want to understand about health scores and how do we use these? How do we use them effectively.

Ed Powers
Sounds good. Thanks, Justin for that intro, Hello, everybody, privilege to join you today, I’m gonna go ahead and share my screen here.

And that was a great intro couldn’t have done that better. So I’m just gonna skip this slide. I’ve been working with customer health scores for quite some time. And I’m sure you all know what these what health scores are, you probably read about them or using them now. Typically, there you know, the red yellow green assessments, typically used to ascertain the level of risk to an account or to a product, if you sell multiple products, a lot of times people will use health scoring, well, we have five products, which ones are are under that contract might be cancelled, I have seen some cases where people have used red, yellow, green for revenue growth instead of churn. But they’re in the minority. Those are folks that have very, very small turn, and they’re just their goals are really wrapped around expansion, contraction, and so forth. So most of the time this has to do with logo churn or product churn. And I’m sure you’ve seen these again. So what is it? What is it used for? I’m sure you know this as well, it’s to help your CSM focus. If they have a large number of accounts, where do I spend my time, this will help guide them on doing that.

Automation, a lot of times if we’re doing things at scale, you can use these kinds of health scores to figure out a treatment for a particular account given what the circumstances are. So this can drive your automation forecasting lot of times, you know, you gotta you gotta provide a forecast. It’s like salespeople do, what’s the rules look like this quarter this month, you can use and people have used it for forecasting as well. And then the last one is for improvement, you can show improvement on an individual account or across an account base and so forth. So a lot of different possible uses. Anybody have other uses? That that’s not on the screen? Anybody using it for something else?

Justin Edwards
What about add on the sales side? Is there information that can be given back in terms of like, informing the ICP? Have you seen anything like that?

Ed Powers
Yes, I haven’t specifically seen but it’s, it would be a good application for it to have a look at you know, if you imagine ICP being like a target, if you will, right? You have a bull’s eye, this is exactly the customer we’re looking for. And then you have expanding rings, right? till you’re off target totally off target. You could potentially say, well, for this group of customers, the bullseye, here’s, here’s what our profile looks like. So you could definitely use it for that.

Okay, so when do teams typically start using these? Well, you know, a lot of times they start saying, Well, what does your gut say? And this is actually what we did it until a secure when we didn’t have any health scores. It’s like, well, what is your gut say about this account? And people say I don’t, I think I call them yellow. And then you put together a little grid like this. And here’s my own set of customers. And I color code those and I have some ACV. And, you know, that is a way of doing it. And a lot of teams start exactly right here is just completely subjective. If you had to put a collar on it, what does it look like at Intel a secure this actually worked pretty good for us, we didn’t have to do anything more sophisticated than that. Because each of our CSM is managed about seven or eight accounts and they were talking to him every single week, they had really good coverage, they were talking to all the decision makers in this was reasonably good. So there are times when this this is sufficient. But what can happen is, as you have more and more accounts on your list, if you’re dealing with

maybe less lumpy accounts and it’s less less of a day to day attention, you have to manage more and more accounts and that’s harder and harder to do and now you start to worry about predictive accuracy and why is that important? Well as you obviously if you have bad accuracy in your health scores, you’re going to make some bad decisions and this leads to something that is commonly referred to in the industry is the watermelon problem right the green on the outside but red on the inside. So the you get some surprises when that when that renewal comes up. Geez, I thought they were in such great shape. You know, I didn’t hear a word from them or anything I heard was positive and yet they churn surprisingly right. So predictive accuracy and your customer health scores does make a difference because it’s going to you know, help you do the right kinds of things that we do.

start paying attention to it. So when do we start getting a little bit more sophisticated? Then what does your gut say? Well, like I mentioned before, if you have a lot of accounts getting on your list, you have, you know, 3040 50 100 accounts. Where do you spend your time? Right? How do you know where to spend your time? Well, with a greater and greater load, you need to start doing that. So in sales, lead companies, that becomes an imperative pretty quickly. And then a lot of companies or product lead where there isn’t a whole lot of touch with those, right? They start building these regression models and very sophisticated way early on, because there’s less to go on, right. And they typically see higher churn and you can do a lot more electronically. So it depends upon the the mix and nature of your company. But eventually everybody’s going to be doing something like this, right?

So how does adopting a more analytical approach help you go from reactive to proactive preventative? So let me talk about the first piece of this is, is customer success strategies and whether they’re articulated or not, whether they’re implicit or explicit? Anytime I work with, they’re usually doing some combination of these three things.

The first is to be reactive, right? You get a you get an email from your customer saying, Yeah, we’re not going to redo. So what do you do you get on the phone, you try to talk them out of it, right, that’s being reactive. So whatever comes your way, you have to jump on it and try to do the best you can. Reactive is typically not very effective, turning people around when they’ve already made that decision, you know, maybe five to 10%, you know, success rates, and that I was talking to someone the other day said, Yeah, we’re about seven. Okay, so that’s, you can do that. But that’s not terribly effective as a strategy to keep customers. The next is, of course, to be proactive. How can we work upstream? Right? How can we do things that will influence their decisions downstream? How do we have, you know, really good handoff between sales and customer success, maybe do some customer success planning, let’s do a really good job of onboarding. Let’s do some value realization, let’s do some customer health scores. Let’s build that relationship. So then when the time comes for them to make a decision about renewing, we’ve just raised the odds of that happening. Now I’ve done I’ve stood up brand new teams before, and I typically, you know, I’ve seen and other people have said, you probably see about a 30 40% kind of improvement on your churn just by doing this well, just by having a process. And by doing a reasonably good job, you’re executing it well, 30 40% reduction in churn right away. But the real place you want to be as over here preventative, this is where you start asking a different question. You say, Well, how do we create the environment where customers wouldn’t dream of going anywhere else? What would that look like? Right? So how do we build our products? How do we target our marketing and sales? How do we have a customer friendly policies? How do we have wonderful support? How do we create this kind of environment have this experience where we just go crush everybody? At the preventative level, what you’re doing is that’s enterprise wide. That’s not just a department, everybody’s pulling in that direction. Everybody understands customer success. Everybody’s working towards that. Whether you’re in development, product, finance and accounting, whether you’re in customer support, customer success, everybody has a metric, everybody’s working towards that, right. So that’s, that’s the highest level and over here, your 9596 97% effective in preventing customers from going anywhere, right? You just those customers just never leave.

And what enables all this? Well, it comes back to data. When you start using data more effectively over here, you may have completely subjective ways. Over here, you’re in order to be preventative, you need to be able to predict what’s going to happen. Any questions on this before I go any further, just make sense?

Justin Edwards
feels about right. Okay, add just one one comment on your green and red on the inside.

You know, like from a standpoint of how that actually feels for the organization is we’ve got mostly enterprise companies here today.

In fact, they’re just dealing with larger accounts. I think there’s one that’s a little bit more transactional, but

very large enterprise sometimes is really kind of hard to read and some one of the signals you see when it’s green on the outside, they’re very, kinda like easy to talk to, you always set up a meeting, but then the red on the inside from a standpoint of value extraction, and you don’t get that expansion that you’re looking for out of that account.

With that lens as you talk about the data, maybe kind of applying the data in a way where

are on this call, we’ve mostly got these enterprise accounts.

They’re all really love us, they want to talk to us. So give us meetings, but then we’re not necessarily feeling the effects of revenue on. And that would indicate a positive score. Right?

Ed Powers
Right. If you if your health scores are based upon retention and not expansion, yes, I mean, you would come back and say, yeah, they’re happy, but I’m not expanding on that right, then then you may want to do look at this differently and have a

red, yellow green that talks about expansion and contraction. In other words, green means we’re growing this account, red means it’s they’re contracting, yellow means they’re staying the same, right? So you could do this a little bit differently if retention is very high. And then look at the factors that drive that growth, right? How do we what causes that to happen? Are we having the right conversations with the right people at the right times? Do we have enough offerings? Do we have? What’s our strategy to expand it within this account? Right? So that has a different set of of connotations, you’re assuming in this case, we have delivered that value, they get it? You know, they’re they’re happy, they’re not going anywhere? But then what do we do next? Yeah. And it’s all about context, you know, every company is a little bit different. Some have, some have very low logo churn. But you know, like you’re saying want to expand, it’s all about an expansion play, then you code this differently, you have a different approach. We have I see Ashwin joined us. And then so you know, this is a perfect example of a very large account that could be signed, potentially. And then you so you’re contracted but then you have to deliver the value to collect on the revenue over a three year period of time. This becomes super important.

Exactly.

Well, let’s I think this next one, yeah. So we this is actually a little bit of discussion here. Want to throw this back to you? For those who are doing health scores? What variables? Are you looking at? What’s driving that? What kind of data are you taking?

Justin Edwards
Here’s some examples uses data adoption, support, sentiment, customer goals.

What are you guys doing right now? What are you using?

Unknown Speaker
Reuse, we use uses data, adaption data, we read. We don’t know how we innovate we use it the sentiment data is like just like, would you just say at the beginning, you know, the CSM says that we have a great relationship. That’s that’s the sentiment. I don’t know whether we use any specific metrics. And then we use support tickets always as part of payment history to as part of it. And then I think we try to align one last thing, which is Justin was just mentioning on? Like, what are our goals for the account for three years? And how are we tracking towards that?

Ed Powers
Gotcha. Yeah, that’s a good one. So progress on to plan would be that one, too? Yeah, that’s a good, Tara, you came off mute, we’re gonna add some,

Unknown Speaker
yeah, I was just gonna say we’re using a number of these, we have five different products here quality. And so we use a mix of these for each of our, each of our different products. You know, frequency of touch support tickets, customer goals, and fit attributes. One of the things that I always struggle with, and I’m hoping we talked about it, Ed is some of our products are, you know, enterprise financial software or enterprise Research Administration. So if you want to spend money at a university, or if you want to get a grant, you have to use our software. And so I always struggle with usage and adoption, for those for those products, because that’s the only way to do the job. And so I’m looking for more insights into you know, how do we dig deeper and don’t and not rely just on sentiment or things like that to really gauge the

Ed Powers
health of these customers? Yeah, I’m glad you brought that up, Terry. And what’s really important is that everybody has their own Mac’s right. I mean, every company is a little bit different. Every segments a little bit different. What works in one area doesn’t work in others, and part of the hard work here is figuring out what’s right for us. You know, what is the right set of metrics? Josh, you came off mute. Oh, I’m sorry.

Unknown Speaker
I got I got one. Church. I’ve analyzed this for several of our portfolio companies and I don’t know that there’s like a one size fit all answer, but I think one of the things that shocked me most when I when I really dove into this was how often we could see that there was a customer With a very, very high NPS score, and they still churn and you know, the things that we’ve started anchoring on or at least advising companies on anchoring on more was, you know, this usage data and adoption data, at least, like, find a benchmark for what adoption should be what usage should be, that goes to like how often they’re using it, but also how many people in the organization are using it, because if you only have one champion, and nobody else is using it, you have a higher likelihood of churn. And then the customer goals and fit. Like one thing, that vision, looking at you, John Landon, you’re constantly measuring is how quickly a customer gets to a positive ROI from the software. And then you know, how, you know, they attribute that ROI over time. And so when a company has a high ROI on the usage, and, you know, a probably a pretty high alternative, you know, might be like, heavy capex or something to go put new hardware back in, if they’re not using the software, measuring those types of things, where it fits the customer’s goals and what they why they originally came over. And then how consistently you’re hitting those goals for them. On top of the data usage, data and adoption data seems to be really telling.

Unknown Speaker
Thanks for bringing it up, Josh, I appreciate the compliment there. I’d say. You know, John, and I too, as we’re looking at the customer success, journey here with invasion, the whole reason why I wanted to hop on this call is to kind of get a better idea of what people are doing in general, when it comes to health scores. Because this is a place I think we’re we have room to improve is to better capture the true health of our customers, though, we feel like yeah, we’ve got a pretty sticky product. This data can help us understand why that product is as sticky as it is, and being able to measure that better. So I appreciate you calling out and the other comments have been made as well.

Unknown Speaker
Yeah, that watermelon analogy is spot on. I mean, so many times we’ve been caught with that thinking that you’re green when you just peel it back. It’s that’s fine that that actually hit home pretty hard with me.

Ed Powers
Yeah, we can surprises. Andrew, anything from you any comments?

Unknown Speaker
You want the unvarnished truth, I’m sitting here, I wish we could do any of this. You know, I because it’s all good. We’ve got usage data that I think would be helpful. We are probably in sentiment land. And I’m probably eager to see your sentiment now, which is a scary prospect. So I’m still interested, I’m interested in the algebra behind this like, Okay, so you’ve got all these elements and areas? How do you compile it all? How do you give weight to the scoring? You know, I, you’d mentioned sort of the rep universe, or how many customers are on your roster? I’m sure it varies, obviously, by company. But, you know, how many accounts should each customer success person have? And obviously, there’s no objective answer to that. But you know, how do you know if you’re, if you’ve got enough, or you’ve got too many? So I have, I have too many questions to ask. So I’m just listening quietly.

Ed Powers
Well, hopefully we’ll unpack some of that Andrew K. How about you?

Unknown Speaker
Andrew, and I work together. And I’ve started Customer Success teams from the ground up for other SaaS companies that have now grown into big boy companies, and was hoping to do the same here with live furnish. And we’ve got some ways to go. But the answer is 100%. Right. We’ve got the data. It’s all in a lot of different places, and some we have access to and some we do not. But I think Preet and the team are are building that type of a user dashboard that is going to be able to bring in some of this and he like I am really interested or I like he is very interested in you know, if we’re going to put a some type of health score, and we’ve talked about it and talked about it for our customers, how do we go in and weigh those? What’s the math behind it? What what data can we do, obviously, the company I was with before there was a ton of stuff that we pulled in, we did it was payments and donations and online type of transactions. And so it’s a little bit different than the product here. But some of the same principles probably can still be applied. So he and I are both really kind of interested I’m in the GGR the gang grow retain group and I go to a lot of their meetings and stuff. So you know, everything here this is all great. You’ll see me shaking my head doing Yes, yes.

Unknown Speaker
Yeah, full disclosure. This is old hat to Kay because she’s been doing customer success. I’m a different kind of dinosaur coming out of the advertising world. So I’m accustomed to client service and account manage judgment, and I’m looking at customer success just as a thing. And I’m not sure what to make of it, it feels very familiar in some respects. But it also seems like everything boils down to a playbook and a scorecard. And there’s no art left. It’s all science. And of course, that just makes me think I’m about to be put out to pasture because I’m thinking, you know, who’s in charge of the relationship? Here? There’s some, there’s some mystery still and some voodoo involved. So I’m still trying to wrap my head around customer success. How much of it am I already doing? And how much of it is a true new discipline? That is getting, you know, sort of support structures built around it? Yeah. Facing at the end of the day, so yeah. And

Ed Powers
it’s it’s a little bit science a little bit, you know, relationship, you know, it kind of blend together for sure. One, one can certainly inform the other.

Well, let me walk through how a lot of companies do actually build their health scores. I’ve said it on several calls with customer success platform vendors, and they go through their onboarding configuration finally goes like this.

The vendor says, Well, what factors are using now? customer says, I don’t know, what do you? What do you suggest? vendor says, well, our other customers use NPS usage, number of support tickets, blah, blah, blah, blah, blah, they give you a laundry list. Okay, that sounds good, says the customer, you know, putting together their Gainsight or tango? Well, let’s start with NPS. So what’s a good number for you? I don’t know maybe more than seven. Okay, well, so how should we wait that? I don’t know, maybe 20%. Right. So they go through this process, with, with their customers, when they’re building these health scores within these platforms. And what they come up with is something that’s pseudo mathematical is how you would describe this, right? So they take some subjective factor, multiply it by some subjective weighting, and so forth, they have some list of stuff. And then because of how this is all configured, it’s a very simple linear equation, and then you just color code and boom, there’s your health scores. This is how it’s mostly being done. But just kind of walking through this, see, see, well, how good is this? This is all this is a little bit better, maybe than sticking your finger in the air, but not much, because it’s all very subjective. So what about a different way to do this using regression actually using the data itself, to create the model? So now let’s take the subjective out of it. And let’s just see what the evidence says. Right? So how would we actually do this? And this, I think, is true, no matter what your business is, this is a good approach to go do something like this. And it doesn’t start with the data. Actually, it starts with customers. Surprisingly, what you want to do, the very first step is go figure out why they leave. And why other stand by more. If you’re not sure what this is, then it’s really hard to build a customer health score. So at all, all good things flow from customers, if we know the reasons that they give us for why they stay and why they leave, then we have something to work with. This is most often overlooked. People start with, well, we have this data, we should make sense of it, we should do something with it, right? That’s not where you start, you actually look at customers, and understand we have two different groups, we have a set here that’s very successful. We have a set over here that’s been unsuccessful. What’s the difference between these two groups? What can we learn from that? Right? And when we when we understand those contrasts, and we quantify those, now we can do things like well, customers are telling us these are the top reasons we can Pareto that out. And we can look both in terms of frequency and dollars. So when customers make this decision to quit, here’s how much money that represents in which category. So for example, you know, we love doing business with you, but your product sucks, or, you know, it’s not reliable, or, you know, your customer support is terrible. We love your product, right? So you get some distribution, some frequency, and then you can tie that back to your accounts, how much ARR did we lose here? So when you can quantify those reasons, and then start to say, well, what’s the difference between these two groups? There’s a lot of things you can actually go and measure. And from that, you start to form hypotheses. I keep hearing this over and over again, they keep mentioning this one thing. Well, I think I can I think I can predict that I think there’s something upstream in my process that points to that, you know, we found this just wasn’t a fit for our needs. Okay, where do we measure fit? We’re not measuring fit, okay, well, maybe we need to go measure that. Because they’re telling me that’s, that’s a reason why they either renewed or they didn’t renew. So we formed some hypotheses, some beliefs, then we go get that data, right. run some experiments. This This part can be hard because, you know, I’ve read a study a little while ago, like, they looked at 500 Different companies, and only 11% of them said that they had any trust in their data at all. You

know that that data quality is generally really bad. It’s not the hygiene is bad, it’s not complete, it’s not accurate. We’re using two different definitions to mean the same thing, you know. So there’s, there’s questions on that. And getting the data is a challenge. And by itself, in fact, in data science, I think data science is more about data than it is about science, you know, there’s so much work you got to do on the data. But once you get that now you can analyze factors what I’m hearing this, I have some data does this actually predict that that’s something called factor analysis, we can run some statistical tests. And we can say, Yeah, that does connect with that there is a relationship between these two, there’s something called dependency here. If some something depends on something else, then it’s a factor, we can actually test that mathematically. Then we develop and test this model, and then we can deploy it. So let me walk through an example here, what this might look like. And let me tell you about a client I worked with recently, they’re in the the appointment setting business. So they do appointments, right? And in their business where they have a churn problem. And so they wanted their dependent variable there, why that they wanted to that that outcome was whether or not their customers would cancel or renew. And they believed that while we think it has something to do with the lifetime bookings, in other words, how many bookings are we getting to get into them, the more bookings we get, the more likely they’re going to stay with us versus those that we have a small number of bookings for weeks on being How long have we been working with them? They believe that the longer they work with them, the more likely they will when they would be to stick with them. No shows when they set up an appointment. How often does that prospect show up? Or just blown off? Right? Because they get some complaints about that. So they felt that that was a factor. And then q ratings are quality ratings. At the end of that appointment? They would always they had an automated system that would say, Well, how good was that league? Give me a score on that. Right? So they believe this. These were the predictive factors. And it seemed to make sense logically in that business. Great. So we collected some data on that, we were able to say, well, these are the accounts and we have all these x 1x 2x. Four, and then we have some outcome, right? We have a complete data record. Great. Well, where do we start? Well, we need to have a model. They are looking at low butcher. This model here is one that you see a lot of it’s used all over physics and biology and an economics, it’s called the logistic model. And what it is, this is this model is a really good one to use in our business for looking at ligature. And when I do a whole lot of math here, but this is an interesting one to think about. So this, this chart here, is comes from this equation, we have one over one plus e which is the natural log, it’s an irrational number you probably remember from high school, it’s like pi. Well, it’s 2.718 and continues on, and then that’s raised to some number theta minus theta. So if we just look at this curve, this will trace this out. So let’s take a large negative number over here. We have one over one plus e to a negative negative number, well, that becomes positive. So I have one over one plus some number raised to some positive number. This number explodes on me, right? So I got one over one.

Variable arrives that does here. On the other hand over here, let’s put some positive values and y one over one plus some number raised to a negative number.

This goes to zero. So I’ve got one over one plus zero, which is one, so that goes to one. And in between while let’s just set this at zero, I have one over one plus Any number raised to zero is always one, right? So I have one half. So there’s my one half. Well, this is pretty handy. Because now I can code this to mean churn. Right? So over here, let’s just pull this to say these are the customers that renew

these over here are the customers that churn and these are the ones I’m not so sure about. They’re kind of on the bubble, right? Really handy. There’s only, you know, two fundamental outcomes. We’re looking for one and zero, I renew our turn, and then there’s I don’t know, right? Well, this is a pretty handy little model. And as it turns out, this is not just a number, this is an expression.

So in here, we have an intercept, which is beta zero, and we have these factors x 1x, two however many factors we want. And each one of them has its own coefficient or scaling factor, right or weighting, you can think about it as a weighting. And then there’s some randomness over here that we can’t do anything about. It’s always going to be there. Because this is not completely deterministic. There’s going to be a lot of noise, right? So, so this is our equation. So what do we need to do? We need to solve for theta.

Because we want to be up here so we’re looking for a value of theta. That’s

going to be large, large and positive. That’s what we’re looking for. So solve this for something that’s large and positive, using these factors and those coefficients. So what do we do? We go into our statistical software and we say, Okay, this grab our data, we say, Great, these are the factors we want to study, booking bookings, weeks on no shows, and cue ratings. I talked about that a couple minutes ago. And I’m going to see how many of these people are turning in how many are renewing. Here, when I push a button, the software does it all. For me, it gives me a table of outcomes. This first column here, this is beta, these are my coefficients. My weightings, if you will, this is standard error. This is something called a wall statistic, a wall TAS wall test. This is a p value, which I’ll talk about here in a minute. This is just the E natural log raised to this beta, what’s that number, and then I have some upper and lower limits, confidence intervals. I’m not going to worry about most of this, I want to talk about this column right here, this P value turns out, what we look for here is some number that’s less than point 05. Because we want to be at least 95%. Sure, when we pick a factor, it’s a real factor. We want to be pretty sure that 95% confidence or point 05 error is a good benchmark. That’s a good rule of thumb. So now we look down this list and we say how many of these are less than point 05? Well, here’s one there, our intercept is 1.3 times 10 to the minus 12. So that’s zero point 11 zeros and a one three, right? So that’s clearly we’re going to use that one. Here’s another one, two times 10 to the minus 39. That’s way out there. So that’s tiny, we’ll use that one. What about this 1.94, that’s much larger than point oh five. So we ignore this one. This is one we keep this is when we throw away. So out of these four factors, there’s only two that matter. We can throw out the other two, they’re not predictive. Two of them are predictive. So what do we do we write our theta. Here’s our intercept. Here’s our R, we said our lifetime bookings here is a factor. Here’s the beta coefficient. So that’s my scaling factor.

We said no shows that’s another good factor here. Point Oh, nine times a no shows this week nor so now we’ve got that plus some error. We can’t do anything about. Here’s our equation. So we drop this in here. And we say, okay, let’s just test this. This is my model, how does a model stand up to reality and go back to my data and say, Great, let’s take a look at that. This is what the model predicts. And this is what I’m seeing in my dataset. How good is my model? So if I predicted success, meaning that customer would be retained? That’s this line. If I predicted they would churn that’s this line? And what do I actually see in reality, it’s these columns success in, in retention or fail, meaning that customer churn, so I build a little table here. And I can now figure out how accurate my model is. What is this? Well, yeah, I’ve seen I predicted that 279 out of 346. Would would, would be renewing, in reality was 279. On a 366. How did I do that? 76%. So 279 on a 366 76%. Of of that is correct. Likewise, how was I predicting churn a little bit better at 84? Overall, 80% to 79, plus 352 divided by 785 is 80%. So on two factors, I’m 80% predictive of my outcome, those two factors be my lifetime. Bookings are no shows. I don’t care about quality, and I don’t care about weeks on, they are not predictive. We’re gathering that data, but it’s meaningless. It’s not telling me anything about that behavior.

What do I do with this? Well, I just put it in my Excel spreadsheet. Here’s my accounts, two factors, lifetime bookings, and no shows. I can color code that these are ones we want to pay attention to. These are ones that are in great shape. These folks here and I might want to give them some love and attention to the rest of these folks. Looks like they’re pretty good.

Two factors 80% predictive, that’s all I need. And in fact, you know, a lot of times these you can get a lot of predictive accuracy with a handful of factors. Usually five to seven is where you wind up. You don’t have to have 350 factors.

You’re looking for a parsimonious model one that is reasonably accurate. Gets you pretty, pretty close, and I’ve seen it be up to 95 96% accuracy pretty repeatedly on a small number of factors just by using this

So that’s an example. And this is all derived from the data. It’s not what is your guts say this is given the the actual behavior of our customer set? That’s how you would do something like this. Any questions on that?

Unknown Speaker
How did you calculate the coefficients?

Ed Powers
Well, the software did it for me, which is really cool.

Literally, with the the algorithms that are out there. And this is just one example, logistic regression is one type of classifier. There are lots of different classifiers. But it’s commonly used and it’s, it’s easily accessible and easy to get. Literally, you select your data, if it’s accurate and thorough and complete, you push a button and there’s your results.

And then all you got to do is drop it into your equation.

Unknown Speaker
Well, it sounds weird if I said, this is absolutely beautiful. Everybody’s smiling, but nobody’s saying weird is not.

Justin Edwards
Yeah, demented, more like it is beautiful.

Unknown Speaker
Come on, now people stand up. But it’s lovely. This is This is phenomenal. Because actually, I have not seeing logistic regression put in this kind of way. So thank you for this.

Ed Powers
You bet. Yeah, it works really well.

And this is something that a client is actually applying right now. So and again, it’s just in a spreadsheet, so we can do much better than just guessing. And again, we can stay away from these. Add these watermelons question for you. All right. I love this. Um, so what can you be concise as it relates to the inputs in the model? So and I’m thinking of it in terms of a portfolio company, that would be actively like a CSM? Because obviously, we’re not employing mathematicians here.

Justin Edwards
Can you be very Layman and concise as it relates to the inputs that are needed from the customer? Or the dataset?

Ed Powers
inputs from the customer? Or the data set? On the model? Oh, to build a model? Well, I mean, you do need to gather back to that, that that process, you know, what do you what are customers telling you? Are the reasons why they’re sticking around or leaving? There’s a lot of clues in there, right? And if you hear the same thing over and over again, then you start asking the question, Where can we measure that? And what is that relationship? Right? So if you don’t have that data, you need to go get that data, if you do have the data, you need to use the data, right? So you may have some of it may, some of us you may not have any data, and you need to go gather it right.

Any detractors as it relates to qualitative data in forming quantitative data. And so like, like in a in a customer churn analysis that we perform, which we do in the portfolio, you’ll get things like too expensive, or not getting enough value or product too difficult to implement, or use, or we had new people,

like in the customer account that turned out and they didn’t know about the new product, or there’s an alternative solution. So you take these kind of like nebulous things, you create some key drivers. Do you get any? Do you get any pushback on on the model at all, as it relates to this kind of converting that to quantitative analysis? And then some kind of output? Well, and that’s a that’s a great question. A lot of times when you ask people, you will get a host of questions, right? And qualitatively, if you go out and talk to 10 different customers, and you say, here’s, here’s five in one of them comes up at least five times, that is a factor that you cannot deny that, but they’re gonna give you a lot of stuff, right? So qualitatively, it’s very hard to get your arms around, well, what’s meaningful, what isn’t typically, then you follow up with something that’s more quantitative. So for example, with this client with some others, when you’re doing you know, if you’re on the longtail, it’s very high volume, you’re getting lots of data, you ask them on the on the exit, why are you quitting? And pick from one of these categories? You know, sorry to see you go, give us some feedback here. And that’s all we’re gonna ask. And then we’ll process your your cancellation. And if you can quantify that it’s about frequency. And now if you got frequency, you know, and the Pareto Principle, there is a shortlist of things that, that explain the vast majority of the issues. There’s going to be a long tail of issues that are interesting, but they’re trivial. They’re not that impactful. So what you’re trying to do is narrow that list and say, well, 80% of the time, it’s these three to five factors that we keep hearing a lot about. So you need to that’s where talking to customers, helps you narrow in on what’s going to be important and what’s not. When you hear that

same thing over and over again, then you know, there’s something there, then you go say, Where am I getting some data? That kind of points to that? And can I use this to predict that? That’s the factor analysis. So that’s why I say go talk to customers, because what what your set of customers will tell you may be different from what someone else’s customers tell them. Right? So all starts with that. And if we can, we can start to break that out and narrow it down. Now we have something that will give us directionally where do we need to start measuring? Yeah.

Unknown Speaker
Related to that? How do you go about deciding when you’re going to start using maybe different scores for maybe different customer segments, and where my mind is on this, as I had a portfolio company that just naturally over the course of let’s just say, you know, three years or whatnot, their clients in like the financial services sector, might only churn, you know, 7% a year, for their customers that were, you know, software companies were churning, I don’t know, let’s say like, 15% a year pretty bad. Obviously, as we dug in, and we learned about the use case, there were different things that impacted their buying behavior, their renewal behavior, etc. And, you know, my my inclination here is that they probably would have had significantly different factors, utilizing some sort of scoring model. And, you know, maybe I don’t know, like, what’s your view on? Should there be a global scoring model? Should you have different scoring models for different segments? When should you do that? You know, my mind is all over the place on that.

Ed Powers
Yeah, no, I agree with you is that it’s, it’s, you do need to segment you do need to understand your customer groups. In fact, another client I was working with, they had a lot of churn, and they were, they had a lot of data like this. But as we were kind of unpacking and try to understand it, we didn’t have enough there to really tell what’s the root cause here, what’s really driving it. And I finally convinced them, hey, let me just go interview some of your customers. Let me just sit down with them. Because he had a lot of different ideas. But then, you know, after talk, I think I’ve talked to 15 different customers. And it was abundantly clear. Look, this is not one group of customers, you have two distinct segments here. And they are behaving very, very differently. You have, you have one where it’s a great fit, they love you to death, you have this other segment over here, it is a total mismatch. Right? So and their current about equal numbers. So guess what, that explains half your churn, right there, you have two different use cases, two different segments, and none of their data looked at that. They I mean, they were not treating them differently. They were treating them the same. So absolutely. Right, you need to be able to understand what are these clusters? What are these use cases and segments? Because they can behave very differently? And how they make those decisions? And how you score them will be very different than other segments? Absolutely. Yeah.

Ah, okay. We’re just a couple of things. And I know we’re, we’ve got about 10 minutes left. So what what tools do we use? Honestly, Excel does a lot of this, you don’t have to get too fancy. You know, Excel has a free plugin. And there’s some actual free software out there that uses Excel, this is 80% of the time, all you need. There are some statistical analysis packages, lots of very, very good ones. You know, Alteryx, does some cool stuff. Minitab has been around for years, they all have these models in there. And then a lot of these customer success platforms are getting more and more sophisticated. Some of these new ones over here, come with AI built into it, right. So you kind of set up and you know, get your best guess and then kind of learns over time. So there are some some interesting, kind of AI driven stuff that’s out there as well.

Yeah,

Unknown Speaker
it’s real quick. So we’re headed down a path, we’re evaluating stuff on the right side there and looking to possibly implement that beginning of the year, after we make a decision, which is in between a couple of those ones on the right.

Ed Powers
Your framework would work within this or in addition to just a little bit of insight there. I would actually go through the process manually in a spreadsheet, so that you can learn because they’re not going to know what what your model is going to be. It’s going to be that same conversation I talked about before. Yeah. And that’s right. I saw where you’re going with that. And it’s like, unless we do that homework up front, it seems like whatever was gets spit out through some automated work that we do with those people, they don’t know our business, right? So they don’t know and they’re just you’re one of many clients and they’re just going to set it up and move on and that’s on you. Right. So you need to do your homework for sure. Yeah. I in fact, what we’ll talk about some ways to do that to disease first briefly,

Justin Edwards
as one of the questions that pops up quite a bit, as we’re maturing and you We invest on the on the low end, it can be a three to $5 million company. But then we see the companies to maturity 400 million plus

Gainsight, churn zero, those are coming to mind and most of the portfolio is using one of those two, do you have any any recommendations or preferences as it relates to the customer score in their ability to make it less generic and more quantitative and database?

Ed Powers
Well, can I can I give you the consulting answer and say it depends. Because keep counting answer to,

it really, really does. I mean, there’s more, these platforms have more in common than they have different, but there I have run into times where, you know, one is clearly better than the other. And you really have to drill down and look at requirements, they have, you know, you can either do it in the product, or with some plugins, you can get very sophisticated up here. And then some of these other groups are out of the box, easy to do. But you know, if, if what they have out of the box doesn’t match what you have, that’s not helping you either, right? So there’s kind of, you kind of have to kind of go through your requirements, figure that out. And there are some some good resources out there that will help you narrow the list down. To some extent I can, we can probably look at some of those, but and there’s some that are circulating around in these different groups. So there may be something to help narrow the list, there’s probably 20 different vendors out there right now. So so if there’s no strong preference, in my mind, it’s the critical mass and then also replication of, of, you know, kind of systems. And so it between the two gain site in terms of where you don’t have any issues? Well, Gainsight is it’s both a blessing and a curse, you can do anything with Gainsight, because it’s works on its on the Salesforce platform. And you know, Salesforce is either really well done, or it’s atrocious, or it’s a disaster, right? How it’s been deployed. So it really depends upon your Salesforce instance. And whether or not that because if it’s junk, throwing Gainsight on, there’s not going to help you, right, same thing with to tango, churn, zero can pull from different ones, you know, it just really depends upon what you’ve got. Gainsight is very expensive. And you’re going to have to have developers on that use, that is a big step up, but you can do anything with it. Just be prepared, you’re going to have to invest a lot of customization and to get it to work the way you want. Right? Yeah, these other ones are off the shelf, but they have simple use cases, if you’re very much more complex, you’re gonna run out of gas with those guys. Right. So

back to

appreciate that. Yeah, I’ve seen him work and not work both ways. So I we’re starting to run low on time. And I want to give you one more thing here. So just to wrap up, who’s responsible for doing a lot of times you’ll have a CSS ops group, or you’ll have someone that kind of gravitates towards it. Or if you have a data scientists or consultants, you know, you have those options as well to help you set these things up. When once you have it, how often should you look at it? Well, any big change, you know, your, your model is based upon historical data. So if something disrupted that we have a new product or COVID happened or whatever, your models go sideways, so you have to start over again, right? So anytime there’s a very big change, you got to go back and adjust your model. And the whole thing is that even if you have a model, things do change incrementally to so you have to kind of maintain that and measure that. and continuously improve it, keep it on monitor, perspective.

Most important pieces, we’re wrapping up here, starve outside in with the customer, I cannot stress this enough, over and over and over again. This is the biggest area of opportunities that we think we know our customers, and we really, really don’t, and we don’t take the time to go figure that out. They will give you all the clues you need to get started. So that’s where you start connected to money. You know, you know, narrow that list down what is impactful? What makes it impactful? How does it deal with how does it impact your money? How does it impact your turn and your net recurring revenue? If you can’t get that association and narrow it, you’re wasting your time on things that don’t really matter.

And then apply the scientific method. This is this is observation hypothesis testing at getting evidence this is all scientific method. Been around for 2000 years really works well.

Okay, one thing I do want to leave you they leave with you I know we’re going to be sending an email with some follow up.

I do want to give you a little homework assignment. For those of you who have something now or are working with it. Some of you are building it right now. But here’s a handy dandy little spreadsheet when you

To improve anything, the the very key step is figure out where you are first, right? Before you do any changes. Where are we now? That gives you a baseline. And then when you make those changes, you can go back and compare it to where you started, did we make a difference? Didn’t we make a difference? Right? So go get a baseline, no matter how you’re doing it even even if it’s just subjective, what does your gut say? Right? Even starting with that up to an including we have a full AI model on the whole thing, you can quantify the predictive accuracy of that system, regardless of how you’re doing it. So how do we do that? Well, what what you do on the spreadsheet will guide you through this, go get some data. So here’s your set of customers. And then what did they do? This example is about low voter. But what did they do? What did they decide? Did they renew or churn those customers? And what was your prediction? What did you have in your health score? Green, yellow, or red? associate those two, what was the last score right before they made their decision, not one from the very beginning. But the most recent one, if you get this data, you can summarize it in a pivot table. Right? Here’s my red, yellow greens. Here’s my I just quantified all this. Here’s the turn renewed. Here’s, you know, just crosstab the whole thing. Great. Now I need to do is take that and make it even simpler. Give me four numbers, four numbers is all you need. Just give me four numbers. What were the reds? And what were the yellows, and the greens, just some of these two up? Out of the reds? How many of them canceled? How many of them renewed? Yellow, or greens? How many of them canceled? How many of them are due, just type in four numbers here? This will give you answers to four very important questions automatically, if you just type in those numbers, the very first question, is there a relationship even between these two? I have seen cases where people have their health scores, and there’s no connection between what they’re scoring and what customers actually did is pretty good to know whether or not there is a connection. This gives you a yes or no. How strong is it? Is it strong, weak, whatever? What is the predictive accuracy? It’ll tell you what it is? And what’s the margin of error? And this is something that people ask a lot of, well, given that I’m seeing a customer that’s marked red, what’s the chance that they’re going to actually turn? In this case? 19% less than you think? Right? So this spreadsheet will give you a host of really valuable information and a baseline to start from, then it’s a question of, well, how do you know 66%? Not great, that’s a little better than a coin flip. How do I get this to be at 90 95%? That’s continuous improvement, but at least you know where you’re starting. So you’ll get this this spreadsheet, again, gather a little data, put in four numbers, just those four numbers, and it tells you a wealth of information.

Justin Edwards
And I think we’re at time. That’s awesome. Edie? Thank you so much. This has been super insightful. I can’t can’t thank you enough. If anybody has questions specifically for Ed, and I’m sure that you’re open to continuing the dialogue. Is that accurate? Absolutely. Yeah. So thank you, again, appreciate that. If you guys would all take a moment to just fill out the survey. Um, that would be helpful for us. We’re doing these digital salons once a month. And it’s a continuation of what we’ve been doing for the last four years. And and so now that we’ve got them digital, we can meet people in North Carolina and all over the US instead of just in person. So we’re excited about the platform. If you have a dialogue, just comment on the platform as well. And anyways, thank you all for joining and we will keep you in mind next time when we send an invite. Please do?