Video

Slides

Expert Contact

If you’d like to connect further with Rajesh:

• rajesh@prodify.group
• https://www.prodify.group/

Transcript

Rajesh Nerlikar 0:00

Thank you, Justin. So excited to see everyone. Thank you all for coming out and excited to be here. All good on the slides. Can everyone see it? Okay, cool. Awesome. Well, yeah. So you know, I wanted to do a little bit of an intro mostly to contextualize how my experience has helped me who has led me to to create this talk, talk a little bit about why I think it’s so important and helpful to switch to outcome based roadmapping. Talk a little bit about how to do it. And obviously, the meat of the conversation, I do have a case study just to illustrate the concepts. And we’ll we’ll pause for, we’ll have some time at the end for q&a, although I will also keep an eye on time and try to pause between each of these major sections. So with that, I’m going to jump right in and tell you a little bit about my sort of journey. I live in Austin. Now I did go to college down here did electrical engineering and didn’t want to design chips got a minor. went to go work at Accenture for about five years started as a software engineer and architect transitioned into a business analyst role as I found myself just trying to draw on to understand the why our clients were asking us to build a custom software that we were building. And we worked in the government space, it was kind of interesting to see the citizen facing experience as well. Wasn’t sure that consulting was the right long term path for me as I went to business school up at Michigan Ross and I use that to transition into the world of startups. So my first year I worked for student run travel startup called hipsters, we got acquired after graduation. And then my second year I started a company called Terra products and design this horrible logo. We focus on sustainability. Our first product was a Facebook app that lets you compare your energy usage to friends and family. So I spent about a year and a half trying to sell that to utility companies and home energy professionals. And then I got engaged and realized that a salary would be nice. And so I moved out to DC and joined a growth stage company called low power that was doing something very similar. If you’ve ever gotten a report from your utility that shows how your energy usage compares to your neighbors. That was the core power product. I worked on it oh power. And I joined as a product manager probably 1112 years ago. Now I didn’t even know what product management was until I started doing the job search. And that was pre IPO. So power went public about eight and a half years ago. So it was kind of there for the final like last sprint of growth before we went public. Went on some internal tools found myself drawn to the more of the consumer facing side of things. And so I left her power and joined a Fintech startup in DC called Hello wallet. We made these web and mobile apps that help people reach their financial goals. So build savings, pay down debt, you know, save for retirement, they were actually distributed as an employee benefit. So now this was the second time I had done sort of a b2b to see product and I enjoyed the Balancing Act. We got acquired about a year after I joined Hello wallet. And so I ran the product team. After the acquisition, my manager went to go work for the company that bought us. And then I ended up doing the same thing. So I moved out to Chicago and joined Morningstar, Morningstar had some for robo advisors that worked inside of the year 401k. So it was another employee benefit product. So I manage this kind of workplace benefit portfolio, probably about $40 million team of about 20, product management, product operations, technical project management, really enjoyed my time and seeing how products work that a slightly larger organization, but I missed the world of startups. And so I started advising on the side at 1871. And I really enjoyed it. And so we decided to move back to Austin about five and a half years ago. And I reached back out to this guy, Ben Foster, who was the head of product design at Opower. He had basically retired at the day of the IPO and started doing some ad hoc advisory coaching work with startups up and down the East Coast. And so I reached out to see if he needed help. And it ended up being great timing. And so he ended up joining one of his clients as Chief Product Officer out in DC. And I took over this practice that we now call prettify about five years ago. And during that time, I worked I’ve worked with about 40 companies and product teams myself, collectively at prettify, we’re probably coming up on 110, maybe even 120. Now that we’ve worked with all SAS businesses, maybe one or two have some hardware components to them. But as you might imagine, we started hearing some very common themes. And when we started talking to this many companies and spending this many hours 1000s of hours with them. Two of them, as you might imagine, might have been red mapping, as well as sort of like outcomes. And so a couple years ago, we put our best practices together, as well as a set of stories from our own experiences and our clients into a framework that we call vision led product management. And then As Justin mentioned, a couple of years ago, we wrote a book called build what matters where we kind of explained how to apply the framework. And so in the book, The to two of the core tenants in the in the sort of framework are one, getting alignment on what outcomes the product is intended to drive and for who whether it’s the users the customer, the buyer, the the business, as well as connecting the dots between the roadmap and these outcomes. And so, this leads us into the sort of like core of this but before I just start talking about the value of outcome based roadmaps, I wanted to first like level set on what our roadmap means to us because we had a very special definition and I just wanted to like share it in the context of this So we’re gonna do an exercise. This is an interactive activity. So if you want to come off a video, that would be ideal, I’m gonna ask you to raise your hand and put it down. Or if you want to use the zoom reactions, I can see all of y’all on my second screen here. So we’re gonna start the activity then Alright, so imagine for a second that everyone is down here in Austin, Texas with me. We’ve got great barbecue, we’ve got a football team that’ll break your heart every Saturday. And the weather’s generally pretty good down here. It’s like 80 degrees here today, so I can’t complain. Okay, so imagine that we’re all standing in downtown and a bus pulls up. How many of you would get onto that bus if you had no idea where it was going? Okay, there’s one who’s who is this brave soul here? Right, Shawn? Okay, as LaShawn can sign up. Alright, cool. Slightly different question. What if I told you that we were in Austin, and that the bus is going to San Francisco? How many of you would want to get onto that bus? All right. All right. So started since John still wants to go on right. John’s gonna get on any books, I’m guessing. All right. Okay, what if we’re in Austin, and I told you I was going to Washington DC. How many of you would want to get onto that bus? Alright, so different if you were hands actually. Alright, how many DC fans okay, that’s okay. Let me come. Let me complicate the question and ask it again in a different way. Austin and San Francisco What if I told you it was an outdoor adventure and every night we were going to stop at a different national park and camping go hiking during the day? How many of you would want to get onto that bus? Okay, wait, way more people. lots lots more activity going on with the XOOM reactions. Okay, cool. Last question. And then I promise I’ll bring it back home to product watching or Austin to DC. What if I told you is the city slicker tour we will stop at some really nice cocktail bars, nice restaurants check out like nice museums, landmarks, how many of you who want to get onto that bus?

Okay, different people, way more people than the last time when I asked about DC on its own. Okay, cool. So let’s bring it back home here. Why am I talking to you about a road trip or a bus trip from Austin to different cities. Imagine for a second that the bus is your product, the passengers are your customers and your internal stakeholders. That final destination San Francisco or Washington DC. That’s your product vision where you’re trying to get to. And the route you noticed that you can’t get from Austin to DC or San Francisco. And one day, we had to stop in a few cities think of that as your product strategy, right? It’s a multi day journey. And we got to stop at some places and milestones along the way. And the daily itinerary being your roadmap. So imagine I had said, hey, at atm, we’re going to leave from Austin, downtown, we’re going to stop at four noon at this restaurant in Houston. And then at 4pm, we’re going to pull into this really cool coffee shop in New Orleans and we’re going to go to dinner, you know around the corner at six o’clock, much more clear on the details between how we’re going to get from Austin to New Orleans. But it’s contextualize in this sort of vein of longer term trip, right multi day journey. And so the other reason that we talk a lot about vision and strategy was this quote, right, which is if you don’t know where you’re going that any road will get you there. And I think the way this manifests itself to many product teams is that you hear ideas, and many ideas sound like they would be good directions to take the product. It’s not if you don’t have the vision of where you’re trying to end up to you don’t know whether you’re taking a detour that’s taking you further away, or whether you’re actually doing something that’s going to get you closer where you eventually want to go. And I think that knowing that I think a good portion of you might be b2b folks here on the call, I think the way this really comes to life for b2b products is they’re signing multi year contracts, right, two, three years. So they’re not just buying today’s product, they’re also buying the future of the product. And if you don’t clearly articulate where the product is headed, and why they should be excited about it. It leaves a vacuum. And this is where the slew of feature requests start coming in. And then then I see product teams playing to the roadmap Tetris game, trying to handle as many as they can and as fast as possible, right. So this is the power of division. If I were going to sort of like visualize it, you know, as always, the roadmap is intended to help you understand how you’re going to, you know where you’re headed, and how you’re going to end up hopefully at a final destination. This is sort of a visual we use to summarize the framework. In the road trip analogy, it’s really easy to know how far we’ve come, right? Because we can look at the odometer on the bus and say, Hey, we had a 1500 mile journey, we’ve come like 500 out of the 1500 miles. Alright, cool. We’re like 1/3 of the way there with a product like how do you know if you’re making progress? Right? Obviously, revenue is one. But what we often use is this sort of like metric of progress is the customer outcome, which is like what’s the value you’ve created on behalf of the customer, because if you lose sight of this, then you quickly start optimizing for like what the business needs, obviously, it was revenue, and sometimes the customer it becomes second, second place in that in that and so knowing that business value is generated as a, you know, a derivative or some portion of the customer value that’s being created, we often talk to teams about identifying what is the outcome of the ROI or the value prop to customers, first and foremost, crafting that vision. We like doing that in the form of a customer journey, which is what do we want our customer journey to look like at some point in the future? Usually it’s three to five years, and then working backwards from that from where we are today. to identify a strategic plan, what are all the things we don’t have today, the really big things that we would need to build in order to deliver on this vision. And its strategic, because you might need to sequence that work, to acknowledge fundraising timelines, the competitive landscape or competitive moats that need to be built data or technical dependencies, right. It’s not just like a random set of big features that you just sequence. There’s some thinking and strategy around why you’d sequence some over the other. And then again, the roadmap is really contextualize the details for how you’re getting it from milestone one to Milestone Two, along the way. So this is a little bit about how we think about roadmaps. I wanted to actually kind of pull on another analogy to explain this concept of outcome driven roadmaps, and all us building a house, has anyone built a house or recently and moved into it? Forever? Okay, cool. So the for those of you who know who’ve done it, and those of you who can imagine what it might be like, looking at the end of the day, it’s it’s like, the output is the only thing that matters, right? Every day, you just wonder, you know, when’s the house gonna be done? Did they put up the you know, the skeleton is the drywall gone up, does the electrical done is a plumbing done, and we pass inspections, right? Everything is about the output. And it’s highly visual, right, you could drive by every day and see progress on sort of like what the output of the work is, that’s being done with the end goal of saying I want this house to be ready for us to move in. Right. And I think that historically, knowing that product is relatively new function, a lot of product management functions and teams have originated out of a sort of, like, output oriented mindset, right. And I think that the transition that needs to be made is like, I don’t pay the builder for once, once they’re done, right, I pay them and I get the house, and then I start paying a mortgage. But I don’t pay the builder for something else that’s valuable to me, like the amount of memories that I make in my house, or like how comfortable the house is, right? Their job is to build the house and like move on with their lives. That’s very different from SAS businesses, right? They’re paying your customers are paying you every single month. And the reason they’re paying you is that they’re getting continued value. And I think this is one of the reasons that that there’s a need to move from a project oriented mindset to a product oriented mindset, right? Once the product is launched, I use sometimes I talk about it like a kid, it exists in the world, you have to think about it, you have to nurture it, you have to think about how it’s going to grow. But it’s continually providing value to customers, hopefully, right. So it’s not this sort of like one off project that’s just done. And then you kind of move on with your life, right. And so I think this is the reason that I often talk about the this idea of outcome based roadmaps, and I’ll be clear here, I would argue this as cutting edge stuff, maybe one to 5% of product teams have started doing these things. So like you’re hearing a little bit of like some some like cutting edge materials. And I’ll talk to you about why I think this is the future of Product Management. And in order to do that, I’m going to sort of like highlight some of the issues I’ve seen with output driven product development. And for that, I’m gonna use this sort of like standard triangle for product project management, right? You’ve got cost, often I think of this as the number of people that you’re paying to build your software design bill, right product engineering design, you’ve got time, hey, I need this shipped by certain date and you got scope and sort of in the middle is like, you know, if you cut any of those three things, you’re probably going to be cutting quality. So what are the issues that come with sort of like this output driven mindset or the project driven like initiative mindset? What is that the deadlines are very restricted to the team. The more you put on the roadmap, the less flexibility and agility they have to shift directions or adapt to changing things. So like, you know, I think of these often, as you know, maybe there maybe there’s a, you know, regulatory deadlines. Often it’s, Hey, we sold this to a customer and told them it’ll be ready by this date. So like, you guys got to fix it, you got to ship something, right. Another issue, you know, you’re doing a project plan. In order to project plan, you need to break down the work into tasks. Well guess what the estimates are rarely I saw you call Sorry. Sorry. And so you know that the estimates are rarely. And I think that these inaccuracies result in delays, and that those delays result in distrust of the product teams. And I can’t think of a faster way where I myself have seen like the trusted road between me and sales and marketing and customer success are like other stakeholders in the organization because we grossly underestimated what we thought the scope of some body of work was right. Also, when you have this sort of like fixed output mindset, you’re usually cutting scope or quality along the way, or especially when the deadlines exist, right? It’s like, oh, shoot, you just give us another deadline. We were already like, you know, sprinting just to hit the first one. And now we’re gonna have to cut the scope or the quality and it sort of usually reduces the reduces or results in a subpar product experience. The other thing that I’ve seen without driven Orient is sort of like development is that there’s an assumption that the solution that was brainstormed and estimated was the solution to the problem that the customer or the business or whatever brought, it doesn’t allow for time for experimentation or learning or trying to evaluate Hey, we actually have three different ideas for how we could do this. Let’s go test some of them out, maybe roll them out to a small set of users. And so this is the big change. And I think one of the first questions you have to ask if you’re interested in this is like, are you ready for that change? Let me pause here. I know that I’ve been talking fast for like probably 10 or 15 minutes. Now I probably I did build a little bit of buffer into this or like talk any major questions on what I’ve talked about so far?

Justin Edwards 15:29

Or just, I would just ask, you know, what, what do you think the holdup is? Why are people delaying on this idea of outcome based?

Unknown Speaker 15:39

Managing of the product? Yeah.

Rajesh Nerlikar 15:43

So I would, you know, I’ll probably cover some of the answers here. But I think off the top of my head, it’s that it’s kind of a standard answer. Right? Change is hard. And I think that part of this also has to do with the maturity of product development, and SAS businesses in the grand scheme of businesses, right, SAS has been around for whatever, maybe 20 3040 years or something like that now, so it’s relatively new as a business model. And I think that outside of probably Silicon Valley, in the rest of the US, at least, what we see in what I’ve seen is that the orientation is always that the role of product is to ship features on time. And so I think that this is at the core of this, which is when the product team was started at an organization, what was the original sort of like mandate or charter, and usually, I find that it leans a little bit more into project management versus like, this was like a strategic team, we need them thinking about the market and pricing and sales and go to market and like all those things, right. And then usually, the go to market teams end up being the ones who lead a lot of the major strategy choices. In fact, perfect segue, just into sort of like one of the things I thought I’d offer here, which is like, I thought it might be helpful, because I’ve gotten this question a lot. I was like, How do I know if I’m ready? Or if my company is ready for this? And so I thought I’d kind of go through a checklist of like, are you ready to make this switch? Right? I think that there’s two big categories of teams and things that need to change inside of an organization for this to be a successful transition. And I do think of it as a transition, probably a multi year journey, to be honest to start making this but I think the question is, should we even started? One is, can we get away from what I would call sales driven development, right, the random set of features that are going to win us deals? And I think the question asked, there is like, are we getting good ROI there, right? Hey, did we did we build one thing, and it really was some idiosyncratic feature that like no other customer has ever asked for right? Now, take this with a grain of salt, right? Some things you’re gonna do that are one off, right single sign on data integrations, those are not reusable. But hopefully the rest of the features you’re building are being built in a way that can be used by many customers, even if one customer asked for it in a very specific way. And secondly, are the customer requests actually improving our KPIs? And so what I mean here is once they are our customer, what I often see is there’s a slew of feedback and Customer Success puts a lot of pressure on the product development teams to address the feedback to come across as a responsive organization, right? Well, guess what very few customers actually end up paying us more money, maybe at renewal time, maybe there’s an upsell. But otherwise, you’re just like optimizing for revenue that you’ve already captured when you could be spending product development cycle is trying to figure out how to win the next deal or grow the business and the pipeline. Right. So I think there’s a question around whether the body of requests that we are accepting from our existing customers are moving the needle and helping us grow the business now, this doesn’t mean, ignore your existing customers, and their satisfaction doesn’t matter. Of course, it matters into SAS business. But I think there’s a question of what the balancing act is between how much energy you’re spending on customer requests versus like, growth initiatives is probably how I think about it. And then the last question is, can we get away with fewer delivery deadlines? Look, I’m not I think there are certain industries like you know, when they’re highly regulated, and there’s regulations and like deadlines, and GDPR was a good example that I think a lot of product teams felt in the last few years. Sometimes you just have deadlines. Sometimes I think about, you know what it’d be like to work on TurboTax. You guess what tax laws change every year, you probably have a finite amount of time to actually implement some of the changes in those tax laws before like, you know, January 1, or February 1, or whatever rolls around. So like, you might have a business where the timelines are actually critical, and you can’t get away from them. But I think you have to if you want to move to an outcome based model, because it gives the team a lot more freedom to test out ideas that are going to move the needle on some outcomes. And we’ll talk more about that in a second. The other major category of areas is sort of the product and then sort of like product operations, as I would call it. So first off, is the product even instrumented to measure metrics are the outcomes that people are looking for, like, do we are we can we accurately count the logins or engagement with features or like, you know, mobile versus web API calls, whatever the case may be? Do we have the instrumentation to actually track the results of a release and then separately? Are we willing to make time in our product development process to measure Through the outcomes are brainstorm ways that we can deliver those outcomes better, right? And so after a release, are we looking back to see, you know, I think back to the sort of like, lean startup manifesto, and it was like build, measure, learn. However, what I generally see in product development organizations is build, build, build, build, build, build, build, right. And so the loop never like, actually, is continued. And I think that part of this is that the deadlines exist, and there’s a desire to keep shipping and churning out features. But I think at some point, you have to pause and say, did we move the needle on the metric that we thought that that feature release was going to move the needle on? Right? And if not, which may be the case? What are we gonna do about it? Right. And so I think the process needs to change, you can’t just have like a, you know, Gantt Chart Style roadmap that says, we’re going to ship the move on to the next thing, we don’t have time to worry about that we did the thing we said, we’re gonna do high fives and move on, right? And I think another key thing is, or do you have the ability to experiment and release. And I think the general idea behind outcome based roadmaps is you’ve got the metric you’re trying to move the needle on, and you’ve got the set of ideas, hopefully, there’s a prioritization framework where you’re gonna, like, say, this, this is the idea that we feel like is most likely to help us achieve this outcome. But you might be wrong sometimes. And so I think the ability to roll a new concept or change out to a small group of users to observe their behavior or get the qualitative feedback, sort of critical to this idea of being able to test hypotheses or experiment in an outcome driven model. So I’ll talk a little bit now it’s sort of like how do you actually do this? So there’s sort of four steps, one, identify the outcomes? What’s the value you trying to create for the business? What are the metrics that are that matter? And how much are you willing to invest? Right? So like, I think, obviously, there’s business metrics, everyone’s got $1 sign in front of something that they’re working on, right? I think sometimes it’s helpful to go more deep, which is like, Hey, is the outcome for next year to grow the logo count? You know, for example, are we trying to sell add new customers? Or are we trying to retain our existing customers, or we’re trying to upsell our existing customers, you can imagine how the product roadmap might be drastically different depends on which of those three outcomes you decide is actually most important to the business at this point in time. So this step, it looks like it’s easy. But I found that many times this is the source of all the prioritization debates and like roadmap, like churn that exists inside of an organization is that people are keeping score in different ways, right? Sales only cares about like top line growth, customer success is all about retention or NPS. And the product is all about engagement, right? Well, it’s hard to optimize your behavior and a roadmap across many different sort of like metrics. And so I think the fewer metrics you can identify, and align on that the better for most organizations. Next up prioritizing ideas, okay, we’re trying to move these three metrics for each metric. What’s the set of ideas we’ve got? Right? What are the things we could do inside of the product or inside of our organization that can really move the needle. And I think, to me, the prioritization framework here, in an outcome driven model is more around which of these ideas is most likely to move the needle, or most likely to move the needle fastest on this metric, right. And I won’t go too far into this concept. But in the framework, we also talked about how there are iterative changes to the product that would deliver incremental results and maybe five 10% lifts on some of these metrics versus stepwise changes where you might be able to like drive 20, or 30% growth on some metrics. However, those often require much bigger level of investment. And I think, you know, to the question that you kind of see at the bottom of number one there, how much are you willing to invest? This is an important decision to make often teams have many goals that they’re trying to hit during the course of the year, don’t just assume that we’re going to split our staff and our capacity, like evenly across four goals, like maybe there’s a good reason that 50% of the staff and ours during the year should go towards one goal, because it’s actually the most important are the artists or things like that. So I think being able to, like, allocate capacity, as well as thinking about the prioritization frameworks is critical here. And next up, obviously, you’re gonna go ship some stuff, right, put stuff into the hands of customers, and evaluate whether there are whether you’re seeing the metric move in a way that’s meaningful. And I think that this might happen through customer discovery. So like maybe just a qualitative interview showing a mock up to someone, maybe you build an alpha version and release it to 10% of your user base, or one or two customers, maybe you run an A B test, if you have more of a consumer facing product, and like 20% of the traffic is just seeing a new version of the product. And you’re measuring the results, right? That’s the whole point, four step, measure, stop, take a minute, go look at the release, what happened afterwards. And I think there’s only like two, three decisions you need to make or one decision, which is, are you going to pivot which is, hey, we saw some early signs of the metric moving, but we’re not not enough. Where were we we felt like we can give each other high fives and celebrate a victory here. You might iterate and make some modest tweaks. Or in some cases, you might be like, You know what, this is not the idea that’s really going to deliver the value that we’re looking for, let’s just sunset this whole concept and move on to the next thing in our sort of backlog. So

these are the core concepts and I will share a little bit more detail about each one of them. Just to clarify. Again, if you have questions, just throw them in the chat. I think we should have some time at the end of the section as well to kind of cover about how to actually make this switch. So, remember, the first step is identifying the outcomes. First and foremost, I think you have to clarify who are you delivering the outcomes for. And here we talked about personas. Most businesses, there’s a handful, especially if you’re b2b, right, you’ve got the buyer, you’ve got the user, obviously, you’ve got the exec team or the board, and you got to deliver some outcomes from a financial perspective to the business. So I think first off, you need to clarify who are the you know, the personas that we’re building for that, then we use a concept that we call key outcomes, which is, what is that group of people trying to achieve? There’s a really easy answer in many cases. And we’ll look at an example in a second for the case study, that’s more of a consumer facing product. But in my mind, in the b2b world, the key outcome for any SAS business, that sort of product purchase is ROI, do the benefits that I’m getting outweigh the costs. And at the back of this deck, by the way, I will share the deck out. So if you’re not taking, you don’t need to take screenshots or anything, but we’ll walk through the the pyramid but you know, what are the benefits you can deliver from a SaaS product, you’re either going to grow top line revenue for your customers, or you’re reducing costs, or maybe in some situations, you’re helping them, you know, be compliant against some regulations, right? And then on the cost side, you know, that’s direct cost, how much does your product cost to launch and operate indirect costs, which is like, hey, how much time do I have to take to teach my employees how to use your product, right? And so I think the key outcome is really important. What is that person trying to achieve? Why does it matter to them in their life? And then just driving alignment? Like I said, this is the place where I’ve seen most of my clients as they start rolling these concepts out, it takes a few weeks, because the metric might change. And they realized that it was the wrong metric in the first place. Or they’re trying to realize, like, Hey, who’s the persona? And actually, we might have different metrics, depending on who the persona is that we’re trying to optimize our behavior and roadmap for right. Prioritizing ideas. So the first thing I would say is brainstorm using sort of like a design thinking question, which is generally phrased as how might we, and this is where having the metrics is extremely powerful, because now you’re brainstorming, like, how are we either going to move this needle? You know, how are we going to improve or reduce this metric? In some cases, you might want to metric to go down. cost or time or energy or something like that, right. And then I’d say you prioritize based on your evidence, and I think of three types of evidence that helps build confidence that an idea has has legs. One is quantitative evidence. So like, I think of behavioral data, can I go out and observe the behavior of my users, my customers, competitors in the market, or like a buyers in the market? Qualitative, which is I’ve talked to a handful of people and I believe that they they are circling around a similar concept, or this is a problem that they believe something they wish someone resolved with solving. And then in competitive Intel, I think, use this one cautiously, I’m not saying go out and just build all the things that your competitors are building, but I think it’s sometimes it’s helpful to understand, you know, as an example, in the wind loss analysis for sales team, what are some of the features or thing concepts that are winning for some of these, why we’re losing deals? Or why are we winning deals, right? Like that’s such as valuable to understand. So try to try to prioritize using that evidence. There’s a framework that a lot of product teams use, it’s called rice, it was made famous by intercom, it stands for reach times impact times confidence, divided by effort, the only thing I will say about that framework, and there’s a link to it here is just be cautious, it is only going to bubble up the quick wins, because you’re dividing by the effort, which means that anything that’s going to take a lot of time is never going to be at the top of this list, because of the way that the math works for this formula. So I think this is good for quick wins, maybe not good for strategic investments that you need to make for the product. Okay, evaluating ideas. You know, like I said, I think this is about getting things into the hands of customers. And you might want to start with some time box test, hey, let’s take one week to do some mock ups and prototypes, three weeks to do an A B test, you know, all those things. Again, I think the more you can roll changes out slowly, the better, you can minimize the risk of building the wrong thing or shipping the wrong thing into the hands of the wrong user. So I often talk to clients about alpha beta general availability releases, the general idea being that more and more users have access to a feature or product, so that you can minimize the surface area of like, reputational risk, or like all sorts of risks really. And then the general idea is that, you know, you build confidence. And you might even think of this phase rollout as having a stage gate in between, which is like, hey, we showed it to one, two customers, and they’re using it like, do we believe that, you know, customers three through five should get access to this thing? And if so, great. Let’s go figure how to make an app. And if not, like, let’s talk about what happened. And let’s talk about what to do with these one or two customers. And we’re alpha testing this thing, like, maybe we should sunset it so that it doesn’t create confusion and like feature bloat inside of our product, right? Measuring and learning. So again, I think the biggest thing that I’m seeing is that it takes a lot of time to stop the product development process, look back at how the releases are going and think about the people involved. Maybe you want your product managers to do that. But then they say Oh, but I gotta get Ready for the next sprint? Because otherwise engineering is gonna be twiddling their thumbs. And so, you know, one of the things I often offer there is like this idea of that kind of base camp uses, they call them cooldown, Sprint’s, right, maybe engineering just spends a week or two like paying down some tech debt or playing around with some new technologies that they’ve been wanting to get their hands on. So that product or design can go back and look at what what happened in the recent release and decide what to do next. Be honest, just because you ship something doesn’t mean you got to follow through again, you moved away from sort of that output driven project, Gantt Chart Style roadmap, you don’t have to keep building something, if you feel like it’s actually not delivering the metric that matters to the business, right. And then the last part is make sure that it’s not just like some tiny group of people that’s analyzing these results, think about how to share the results and the insights that came out of it often. So if it’s a sprint demo, at the end of every sprint, great if it’s at an all hands meeting so that everyone in the organization can understand what are we learning as we’re sort of like testing out new ideas? Are the metrics moving? And if so what’s driving those, those changes and all those things? So those are the sort of like core concepts that was like the heavy meat of like how to do this. Let me pause again, and just see if anyone has questions on the tactical side of like, how to actually roll these changes out or do this?

Unknown Speaker 31:19

Hi, my name is Nina. I guess I have a question around so so the product I work on is more direct to consumer. And we release daily, almost. So when you when you talk about the make time to analyze. We’re also very, you know, a pretty small startup. How do you kind of like, do you think that translates? I mean, we we I do look at everything that we’re releasing, but because we’re releasing almost every day, it’s a little bit different from a analyzing time perspective.

Unknown Speaker 31:48

Got it? Yeah,

Rajesh Nerlikar 31:50

I mean, I think in that situation, so it may depend on the sort of like situation that the product is in, right. If it’s highly competitive, or you’re trying to get metrics in a certain place for fundraising, then like, you know, maybe you do need to keep keep a really close eye on whether every single release is moving the needle. In those situations, you might also maybe have a weekly check in and like, but I think that the only way that you can switch to weekly is that you got to whittle the number of sort of metrics or the dashboard has to be sort of like digestible, right? If you only have one hour a week, instead of one hour a day, hopefully that one hour, it can be used to like talk about just maybe one or two or three metrics or something and be like, hey, cumulatively, all the releases last week resulted in like this lift on this metric, right, probably, you know, engagement or something, I’m guessing it would be one of them for most consumer facing products, right. And I think if you’re running experiments, great, if not, no worries, but I think being able to identify the difference between like tests and control might also be powerful during that session.

Justin Edwards 32:46

Yeah, Cooper as your hand up. Yeah. Hi.

Unknown Speaker 32:49

So under evaluate ideas, just curious if you had, what are some of your favorite methods for building up your customer base for testing and like, listening ideas and that sort of stuff? Yeah. Cooper, b2b or b2c?

Unknown Speaker 33:14

Or your we’re b2b, but um, it’s a bottoms up b2b strategy.

Rajesh Nerlikar 33:23

So it’s a lot of individual users versus top down. So I’m, I’m a fan of like, what I guess I would call the advisory boards. And I think the general idea there is that you invite a handful of users or customers or if there’s one of the same grade, and just set the expectations properly, which is like, hey, you know, once a quarter, we’re gonna call on you. And we’re gonna ask you to spend maybe 30 minutes or an hour with us some situations, you might pay them for their time and others you may not. But it in my mind, they should be? Well, a couple of things. One is you kind of feel like an honor that they were invited. And they should know that there’s some expectations on their end of like being a part of this, right, maybe the value prop of why they do it is they want early access to features because they like love geeking out on, you know, some some power users or super users would want that, right. I think the other thing to be conscious of who you invite into this is, who is the feedback coming from? Right? So if you’re trying to figure out why, like the average user is not logging into your product on a regular cadence, I would probably be cautious of just talking to a bunch of super users who do log in every single day or three times a day, right? Their feedback is probably gonna be drastically different than the others. And so I think I’m a fan of the advisory boards, I think for some products, 1020 30 people probably enough to get started. And I’ve seen some tools out there that are pretty good at helping you set this up and think about how often do I call on this person and who should I invite to the next round of interviews, knowing that maybe it’s been two months, three months or whatever, so.

Rajesh Nerlikar 34:57

I see one more hand up. Let’s see. Okay.

Unknown Speaker 35:00

Hey, thank you, I was gonna ask you, how do you deal with these evaluating ideas? When you’re a first to market? There’s no competitors. Absolutely blue ocean, we’re trying to create the market. So the problem is, we’ve asked advisors and nobody really knows what is possible.

Rajesh Nerlikar 35:20

Yeah, well, I think the question is, what are you? What are you trying to learn from this advisory group, right? If you’re trying to get competitive Intel, then I would agree that they’re probably not going to be able to provide meaningful insight. But if you’re trying to understand whether the product is solving the problem, or helping them achieve their outcome, in a way, unlike anything else they’ve ever seen, I think you could ask questions like that. And I think so, you might be able to say, hey, what’s working, what’s not those types of things, and I think you can get some good insights that way. I typically think actually, there’s a whole nother deck on sort of b2b customer discovery interviews. But I think that there’s different categories between like usability issues, or, you know, interviews, I want to get feedback on something we’ve already released or are about to release, versus what I would call we consider more like generative research, which is like, I just want to talk to you about your life, your business, your problem space, what products you’re using today. And this is where you get a little bit more into the competitive Intel. And maybe they’re not able to sort of like, articulate that, or the competitor isn’t another SAS solution. It’s a spreadsheet and a series of emails or like a bunch of meetings or like your idea, there’s always some hack together thing that, especially in the b2b space, it exists.

Unknown Speaker 36:32

Yeah, quick question. On particular, on your slider, you’re talking about, you know, how do you know if you’re ready, I kind of chuckled at can we get away from sales driven development? Because that describes us to a tee? I guess, like being particularly a startup, right? How do you go through that balance of like, hey, it’s a feature we need to build. And like, we kind of want the money. And we can talk ourselves into this being a roadmap item or something that shouldn’t be on the roadmap, right? Like, how do you go through that tension and, and kind of move either your sales organization, your organization as a whole in that direction? And then I guess the second piece is, particularly in the overall macro climate that we’re moving into, is that a wise thing to kind of consider or even, you know, start that journey?

Rajesh Nerlikar 37:15

Yeah, I am. So I think the answer is it’ll depend on where you are on the product lifecycle. And I think of the product lifecycle of like launch, to trying to find product market fit to find having found product market fit and ready to scale to like having a mature product to like sunsetting, a mature product, right. And we usually use the S curves of innovation to show how the cycle should repeat over many years, like kind of consistently. And so the reason I bring that up is I think if you’ve just launched or you’re circling, trying to find the product market fit, most b2b startups will start with a sales driven motion. And I think that’s okay and appropriate, right? How are you going to know whether the value you’ve created is enough? If you actually aren’t sending the contract out and saying, pay me money for this thing? Right? I think the slippery slope are the thing to watch out for is are you rebuilding the same feature or solution? are you solving the same problem in many different ways? Right, so you have like, now you maybe after a few years, you’re like, dang it, we have like four different features that do almost exactly the same thing in our customers workflow. And all of them work differently. And so they end up being like, really expensive. And so to me, the key question is the SAS Holy Grail, are we building a scalable solution that would be that would work for the next wave of customers we’re going to find, and I think without getting too far into, like product market fit and sort of like thoughts there, the target audience may change as you move through the lifecycle, right, what the early adopters are looking for might be different than the early majority might be different from the late majority. And so I think you also need to think about, can we reuse what we’re building for these early customers? And to me, there’s nothing wrong with having sort of customer funded development, especially really early in the product lifecycle for a b2b business.

Unknown Speaker 38:54

Thanks. Yeah.

Rajesh Nerlikar 38:57

Okay, maybe last question, Christina. And then I’m going to do the I’ll just walk through the case study, and then we’re gonna open it backup.

Unknown Speaker 39:03

Yeah, I had a question on it’s a little bit more tactical. And it’s really how do you communicate the roadmap to the company, like I’ve been following more of an OKR process, but I struggle sometimes with that process of being, you know, very clear on what your KPIs are, and being able to measure impact to be able to iterate. Like sometimes a feature takes two months to deliver. That’s like two thirds of a quarter. And so I think kind of balancing like, how do we define what our roadmap is? Do you recommend more quarterly based roadmaps? And if so, like, how do you get to the point where you can actually measure and then iterate within that same time horizon?

Rajesh Nerlikar 39:43

Yeah, I think it’s a great question. Hopefully, as I go through the case study, there’s an example of an outcome based roadmap at the end of the case study, but I think the short answer is you know what, I think you just have to be open and honest in some situations, some of the development effort may not produce an impact on the the metrics for some period of time when That’s within a quarter or over many quarters. I think that those are big bets. And you need to make sure that everyone understands that this is a big bet. And there’s risk that comes with every big bet, we may have built it the wrong way. Now, you might use an agile mindset and say, Can we slice this in some new way that we hadn’t thought about so that we could ship some incremental value and test that we’re headed down the right right path and like, create some kind of like checkpoints to make sure we’re investing the right amount. But I think it’s hard. And I think there might be some other ways I, you know, my take, depending where you’re on the product, like a company maturity stages, you might need a balancing act of like big bets versus small bets that are more sure. And you might have a steady stream of iterative changes that are producing incremental lifts in some of the metrics, knowing that there might also be parallel some major investment that’s going on, and that might hopefully produce a stepwise change on metric, right? Well, let me walk through the the example and then open it up again for questions. So this is an imaginary product, I’m going to use a consumer facing product solely for simplicity. I know a lot of y’all might work on b2b products. And I’ll happy to explain how I would adapt this concept for them. chuckwagon imaginary product, we use it in the book to illustrate how to implement the vision led framework. So I’ll ask you all this, how many of you cook home cooked meals? Okay, all right, it’s a few hands go up. All right, that’s okay. It’s not a whole exercise. But Chuck Wagon has a product for you. This is an example of a key outcome. And the key outcome that Chuck Wagon is intended to deliver to consumers, is reducing the amount of time it takes to spend on meals, we use a pyramid structure to kind of break our key outcome metrics down into further sort of like value props. And so in this case, kind of followed a workflow model here, which is like, what is it? What does it take to get a home cooked meal on the table, you got to plan and you got to cook. Within planning, you can think about that, like, Hey, I’m kind of like drafting a meal plan. These are the things we’re going to eat food on Tuesday versus Wednesday, and like stuff like that, and then I gotta go to the grocery store, make sure I have all the ingredients, right? If I break that further down, you know, I’m gonna make the plan, I’m gonna edit it, you know, my wife does this. And she’ll like, send me that list. Or she’ll ask me, what else can we make this week, those types of things. And then it takes, you know, you gotta go to the grocery store, you gotta shop. And also takes a lot of time to cook the meal, right? I gotta prep, I gotta cook, I gotta clean up. And these are the value props. And the general idea here is that we’re breaking down the outcome metric into sort of like sub metrics, and being able to show this sort of like vision. So this is an example of a vision. So the vision is to save millions of hours on deciding what to cook buying groceries or getting home cooked meals on the table, basically, strategically, their first phase was to build a feature related to recipe plans to really reduce the amount of time it takes to plan meals. The second phase was to reduce the amount of time it takes to, to spend on groceries or grocery shopping, the idea is to convert the rest of the meal plan into a grocery list. And with one tap, you can order. And then the last thing is to offer home chefs in the house, you can come make all your meal plans. And now we’re saving time on actually cooking and cleanup. Again, this is strategic in that there was a reason that this was sequenced in the way that it was, for example, you couldn’t go to the likes of Instacart or shipped and say I have 25 users in Austin, Texas, would you do an integration with me, you kind of needed to have a volume of consumers to make that an interesting integration or partnership for those grocery delivery services. Similarly, you can’t just have home chefs coming into your house making any single recipe under the sun. Having the recipe plans built first meant that there was a finite number of recipes that a handful of chefs needed to know how to make. And so there was some reasons why these things were phased the way that they were also with home chefs, it would be nice if the chest didn’t have to do the grocery shopping themselves, and that the groceries were already at home. So you know, here’s an example of some mock ups. Again, when I talked about the vision, we were a big fan of showing the mockups especially for consumer facing products. So here’s a mock up of what meal planning could look like, you know, tell us about your household hit a button and boom, every day of the week, you’ve got sort of a meal, you can replace it. The next phase was grocery deliveries. And now we’re converting the meal plan into a grocery list. And you can kind of check things off as you go through the grocery store. And then there’s a call to action to link up your shipped account or your Instacart account and have your groceries delivered to you to your door. So you don’t have to spend time in the final sort of strategic phase chefs, while you’re waiting for your groceries to be delivered, there’s a call to action to request a chef and you can book that person to come make all the meals on your meal plan. Okay, so this was the general idea behind the vision. Let’s imagine this was last year’s coming up on December. And this is the key for roadmap. This is an example of an outcome based roadmap. So I’m going to pull this up slowly. I’m going to call out a couple of things here. Right? Ignore the color coding for one second. I won’t talk a little bit about the reason we call these things different things is like we said sometimes there’s you know, big bets you need to place that we call innovation, like they’re gonna take a while. There’s iteration which is modest tweaks. But notice the left hand side of this roadmap. One it says that grocery delivery value props represent 60% of the product development capacity in the coming time to it’s actually broken out into different outcomes. Remember how we talks about there’s different personas who are looking for different key outcomes. The first one is the user facing persona, which is like how do we reduce the amount of time it takes for grocery shopping by 70% for our user base? The second one is the business outcome, how do we extract value from that value creating that sits at time savings for users by charging them money to do things like grocery delivery. And then you can look at sort of the actual roadmap initiatives and notice how like, you know, there’s sort of an alpha beta concept for Instacart. And it might take the good part of the first half of the next nine months. And meanwhile, we’re also starting to do some research and prep work to plan to do an integration with shipped in the second half of like, you know, next year, separately, doing some research on how to opt in how to price our grocery deliveries and running some experiments, right. And then the final alternative, like swim lane here is that, you know, there’s a meal planning feature set that presumably is in a more mature state, which is why it’s only 40%. Again, now we’re trying to like really figure out how to optimize the business outcome there, because it’s a mature feature set. So we’re going to run some experiments. There’s also a concept of saying, like, I don’t know what we’re going to do, but I know that after we run these experiments, we might want to reserve a little more capacity to do these things. And then finally, you might do some usability testing to like, continue to drop the amount of time it takes for meal planning. So when you see Joel, I think I saw your hand go up here.

Unknown Speaker 46:20

Yeah, thanks. Um, I apologize if you’re already mentioned this, but are you a fan of taking which portion of this do you publish publicly? And then to if you push if you publish the now soon, later piece publicly which I assume you do, that you recommend? How do you prevent the customers from zeroing in on But you promised Instacart delivery beta and halfway through 2022?

Rajesh Nerlikar 46:44

Yeah, great question. I would say. There are probably very few instances where I’ve seen public roadmaps published, I think, probably the product lead growth model is where I would imagine that being most common, so I’m selling, it’s a b2b product, but I sell to SMBs. And they’re curious about where this is headed. I don’t think I’ve seen a single example of a consumer facing company who’s ever shipped showed their product roadmap. And I think if you work in the big, like, kind of like with mid market, or enterprise b2b customers, that it’s pretty rare that you’re publishing anything publicly, because there’s not really a ton of traffic or audiences are really clamoring for that. In fact, I’d probably say it’s better to not. And so However, one thing I will say about more of that b2b side is, I think, 100%, you should share the roadmap with your customers. And I think that it’d be a good habit to just like be have some cadence by which this happens. When I was at Morningstar, I basically, I think, once a year, I got, you know, an hour from the quarterly business review or whatever with our clients to talk about product roadmap. The reason is, I want to make sure that they agree that the things that the value props, independent of the features, that the problem spaces that we’re trying to solve for them are the right ones that they would agree with, they’re nodding their head and saying like, Yes, I agree that that amongst everything else I could possibly think of I agree that these these look right, right? Because if not, I need to reconcile that quickly. Otherwise, they’re going to fire us or, you know, we’re going to have a very tense relationship for between customer success and product and stuff like that, right. Thank you. And the second part of the question, like how do you prevent them from dialing in and saying that you promised Instacart? Yeah. Well, I think, um, I don’t have a great template here. But what we’ve often talked about is, I think the visualizations of roadmaps is an extremely powerful thing when you have these one sliders right. So one thing I think a lot about are the swim lanes, obviously, in this example, I’ve got the outcomes as the swim lanes, which strongly communicates that we’re using an outcome based sort of like model here. I think the more you can communicate uncertainty or flexibility, the further you get out the better, right, and so you might, man, I wish I had this one from one of the other prettified visors, he had like a little cone that just a little gray line, that cone that got smaller as you went out and write right next to it. It was like note uncertainty goes up as you move, or like, you know, likelihood of change goes up goes or sort of fidelity goes down as you move out. And and like, you know, at Opower, we had a product operations team, they basically ran some analytics on our roadmaps and found that we were like, I don’t know, 70 80% accurate within six months, and like 30% accurate at six months. I think it’s it’s a natural thing. Most customers would understand this, but I think it’s worth calling out and saying, hey, I want I want to start the conversation. Because I’m curious what you believe, six or nine months or 12 months down the road, we should be working on to make sure that we’re aligned, because if you tell me something different than what we had planned, I want to have that conversation now rather than later. Um, one thing I was just gonna mention, just, you know, that that a lot of folks are b2b. And I think I had some of you submitted some questions upfront, and I think I had identified like one or two themes, but if this was a b2b roadmap, what I would do instead of having like kind of these consumer facing KPIs, I would think about the value props of what you’re delivering what metric what outcome are you trying to deliver to them and it might be three or For me be clear what, which one are these features connected to? Right? The whole spirit of the outcome based roadmap is like right there on the left, I see the metric. The prioritization is these are the things that we believe are going to move the needle the fastest or most efficiently on moving this metric up or down, or whichever way you’re trying to manage to move, right. So I think getting alignment first, unlike what are the metrics that matter to your customers, whether it’s time savings, or cost savings, or like revenue growth, or whatever, and then try to rationalize how these feature sets are going to deliver on that metric for them can be a powerful way of presenting this.

Unknown Speaker 50:37

Any other questions? Yes, I have. Yes. You could go back to the previous slide. Can you explain again, what you mean by operation?

Rajesh Nerlikar 51:00

Oh, yeah, sorry, I totally, totally glazed over that. This is something that comes out of our framework. We call it a roadmap balancing framework. The point was, we create a framework called Vision lead product management. Very few teams, once you have the vision and a multi year like roadmap or strategic plan to realize the vision could then just be like, Oh, we’re going to take 100% of our capacity. And we’re just going to try to go build towards that vision, right. It’s not realistic. And so what we realized is that product teams are juggling three categories of work simultaneously. There’s innovation, which is basically what we call progress toward the vision. There’s iteration, which is like, hey, we need to tweak the existing product, we have usability issues, we have conversion, funnel optimization, and like those types of things. And then there’s operation which I would say is this sort of like tax of modern SAS owning a modern SAS platform, right? performance, security uptime, like tech debt, right. And so the point of operation was, and in this example, the gray boxes were like internal facing tools effectively, I think, is the way that they would these were built in this example. And the point was that as you think about a roadmap, you have to acknowledge that that not all capacity is all user facing or customer facing, right?

Unknown Speaker 52:05

Yep. Okay, that makes sense.

Thank you. Sorry, I forgot to cover that.

Unknown Speaker 52:08

No, it’s okay. And then I for the outcomes KPI, you’re recommending like an annual goal.

Rajesh Nerlikar 52:17

that, I would say, it may depend on where you are in the private life site or like company lifecycle, I think you’d be needed to have quarterly OKRs. That’s fine. I think what I found with my product teams that use quarterly OKRs is by the time that OKR, is published, and everyone’s had a chance to look at them and digest them. And then like, think about the roadmap, you’re like, already a month into the quarter. So one of the things that probably thank you, I probably shouldn’t just emphasize this, the less these metrics change, the better. And I think that there is like this metric whiplash, and I’ve had a few clients where the metric, the KR actually changed from one quarter to another, or the first half of the year to the second half. That’s like a huge deal, right? As you can kind of tell from this model, what you’re trying to do is build a roadmap, that’s what are the highest priority things we can do to move the needle on that metric? If that metric is changing all the time, it’s like very expensive from a planning perspective, right? The way I think about this is like, or the analogy I use sometimes here is like, imagine if you were running a marathon and halfway through the race, they were like, by the way, the winner is going to be the person who took the fewest and amount of steps, not the one who ran the fastest. And now you’re like, well, shoot, my whole life strategy is gonna change as a result of that. Right? So I think that, you know, I think annual or quarterly is fine. But I would probably say the the metric goal is probably annual is good, because it might take a decent amount of time to actually move the needle. And this is the time that most of my clients are doing this December, January is when they’re looking at 2023 planning stuff.

Unknown Speaker 53:39

Awesome. Thank you appreciate it.

Rajesh Nerlikar 53:43

Well, I know Janessa sent the feedback form. So yes, if you have a few minutes now, and one of the other things I’ll do and I can stick around for a little bit is just mentioned, if you do have questions, email, Justin Janessa, one guide, or you can email me directly and I’m happy to answer any questions. I was also like I said, I was gonna share this deck a little bit more on our book, we have a template if you want to, like borrow this like thing and just like edit it directly. So you don’t have to recreate the whole concept. I was just going to call out one thing that was more of a b2b story. So this is a story from my time at Opower. With Ben and it became an HBs case study. The general idea was, we had this, we had a lot of when Ben inherited the product team, there were a lot of deadlines of what the sales team had sold. And in fact, the entire, like six months was just kind of like, littered with deadlines, and there was no flexibility. And him the head of sales, the CEO got went into a room and decided that to create a system where basically sales could buy some capacity of the roadmap. But they had to, they had to, they had to escalate the request all the way to the like VP of sales to make the token to pay for these like that thing. And what we found was that they got 50% of the roadmap, I think, is what what we ended up so we also talked about the ratio. And what we found was that we ended up with a much more scalable product area because the VP of sales was like, I’m not going to use my tokens on that like random stupid feature requests. because like it’s going to drive like, you know, $50,000 of revenue when we’re trying to hit like, multimillion dollar goals here, right? So anyways, in case it’s helpful, you can kind of look at it. And then the only other thing I was going to mention in the back of this was that b2b pyramid that I sort of mentioned that I think if you do work in a b2b product, this is the standard pyramid I ask clients to think about, which is like, what is the ROI math that’s happening in a kind of buyers head when they’re buying your product?

Justin Edwards 55:29

Thank you so much for Jewish this is this has been amazing. I think that if I have a couple of takeaways, one is, you know, just avoiding unreasonable ambitious planning would be a key point. And you do that through the outcome, focus, and really just balancing functionality, enhancements and tech debt. So this has been super helpful. I have one last question for you. And it’s, it’s regarding holding the people responsible for the product roadmap accountable. So it’s more of like an HR performance. And just maybe one brief one or two brief insights into how do you manage the team, when you’re missing key features as it relates to your updates and product roadmaps? Sorry, Justin, can you clarify that last part, like you’re missing key features and the subjective knowing the roadmap? It’s, it’s because of features not ready. You’re either doing iterative or you’re trying to incorporate a strategic approach approach. And then when you’re missing that, what is what is the recourse? A leader in Project product management has to make sure that the teams accountable for hitting those dates.

Rajesh Nerlikar 56:54

Yeah, yeah. I mean, I think that there are probably a couple things I would highlight one, the fewer outcomes that the company is pursuing, the better. And I think that if you want to build the cross collaborative team, for example, revenue should not just be owned by more sales, right? Like, obviously, marketing is usually like in there with sales generating the leads, I think that product should just feel as much accountability for the sales number as well. Because if you are going to b2b product, what other metric do you have to know that your product is actually valuable in the market, then revenue. And so I think that the more you can say that there’s sort of, I don’t know, if there has to be joint accountability. I’m not saying that like only you know, they should split the accountability, maybe it should be sales. But I think the product people should should feel the pressure and stress when the sales team is like off off course or something during the course of the year as well, and should be thinking about do we need to adjust our roadmap to better support, give them a product that will sell in the market, whether it’s because we’re missing out, you know, competitive feature, or whatever, right, that that’s causing them to not win deals. And I think that that’s sort of a critical point. I think the accountability, this is a little bit of a, this would require a shift as well, maybe even on the HR side, which is like, Hey, we’re moving to a world where outcomes is the thing that we’re actually showcasing on a sprint or sprint basis, or like when we come together and do all hands. I’m not like giving you I’m not giving each other high fives because you shipped a feature that was like supposed to ship in March or something. I want to know what the outcome was. And I think it requires this sort of like mindset shift, which is like all conversations start with where are we on these metrics? What’s plan? Are we on track? If not, what are we what are we planning to do? And like having that consistent thing, I think that can create the environment of accountability, like whether or not performance reviews and stuff should be like tied to that maybe there’s something interesting, there was a controversial article that went around, I think, earlier this year, where it’s some, some startup had actually connected the product teams, bonus to revenue goals or something like that. And I think it’s an interesting like, kind of thing to think about.

Justin Edwards 58:45

Yeah. Well, thank you so much for your time. It’s been very educational. Everybody that attended, please take a moment to fill out the survey. We love the feedback. We want to make these better. We want to know what topics we should be discussing as an investor to provide value for our portfolio companies. So thank you, Rajesh, if you haven’t read the book, pick it up. And I really, really appreciate the time. Thank you.

Rajesh Nerlikar 59:15

Yeah, of course. Thanks, everyone, for coming.