Crafting Code Podcast

[00:00:16] Allan Stewart: Welcome to the Crafting Code Podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately I've been thinking about the impact made by a friend who gave me a useful programming book back in the days when I was first learning and documentation was scarce.

[00:00:35] Dave Adsit: I'm Dave Adsit, a VP of engineering, and recently I've been thinking about the problem of specialization as an organization scales.

[00:00:42] Allan Stewart: Our topic for this episode is software costs. It's a perennial question. How much does this cost? How much does writing software cost generally? How much does it cost to build a product? If there's somebody who has this great idea for a startup and they just want to know, well, how much is this software going to cost me? One way or another, this question comes up a lot. So I think there's an obvious question of why the question comes up, but why does it matter for the cost? What are they really trying to get at, do you think, Dave?

[00:01:18] Dave Adsit: Well, I would say in general, the question that I'm asking when I want to know how much it costs is like, can I afford it? Can we build it? Is it viable? Is there actually a viable business inside this idea? Is it something that has potential? Also, typically when you're starting out, you have a limited amount of capital. Either you've got your own seed money or you've raised money from an investor or something, and you're like, we can go for this long with the cat. I want to know how much this costs because I want to know if I have enough capital to even do it. Or. Or if I'm just going to burn through everything in my savings and then we won't even have the product that I wanted anyway. Mm-hmm. So.

[00:02:02] Allan Stewart: Yeah. I mean, it matters a lot because software is expensive, right? Like the great thing about software is that it is very malleable. You can do lots of things. It's kind of incredible sometimes, all the things you can do with software, but it comes at a cost.

[00:02:20] Dave Adsit: Well, and if you're talking about established businesses, you like, we want to get into this new business line. We think it's going to return this much. We think that we can sell this feature for this much, or if we add this feature, we'll get these customers, which will bring in this amount of money. Like what will it cost us to build this? And a lot of times in software costs, time and money are synonymous because every hour that you have a software development team working is going to cost you something. Um, and it's important to know how, how do you even figure out what that cost is? So. There's various costs associated with building a software. You've got to think about the biggest one is going to be your, your team. You know, you've got, of course you've got costs associated with hardware and SO and you know, software that you're purchasing and things like that. But you think the biggest cost of building software is the team of professionals that are responsible for building the software. And usually for the teams that we work with, that's going to be several software developers, several software engineers. And. Um, Someone with a product management product owner type of a role, probably a designer. If you're doing anything that's UI intensive or online or whatever, if you're building a product that's customer facing, um, and it might include other roles like QA or DBA or dev ops. But at the very least, you're going to have a team of a handful of software developers and someone who is thinking about the product holistically. and what needs to be built and how to prioritize the features.

[00:03:58] Allan Stewart: Yeah. Yeah. I feel like thinking about it in terms of a rate makes a lot of sense, right? Because, yeah, the time and the cost are kind of interchangeable. And I think instead of thinking about one or the other, it's more like both, right? It's not how many miles. It's not how many hours. It's how many miles per hour that matters. Because it's so impossible to know, right? Like you might have ideas of what the product is going to be. You might have ideas of, you know, well, we've outlined these 40 features that we think need to be done. But it's very hard to estimate how long it will take to do a particular feature. And it's hard to know when the gremlins are going to reach out and you hit an unexpected hiccup that costs an entire day of work. And so, yeah, thinking about it as a rate makes a lot of sense. And I think it goes back to some of those ideas that came out like around like the Agile manifesto, right? It's like we're thinking about working software in small batches, short iterations where you can just say, hey, is this working for you? Is this working for you? You have to spend a little bit to get into it. But then at each point, hopefully you have something to show for it.

[00:05:22] Dave Adsit: Well, and that's, I mean, definitely I like to manage my bets when it comes to the software that I'm building. But I also like to do back of the envelope calculations. I don't get over precise when it comes to estimating costs. I think that, first of all, we're bad at estimating. We don't know how long it's going to take to do something. So being over precise on the, you know, the cost per hour or whatever when we don't know how many hours is kind of a fool's errand. So I, for the teams that I run, I usually have a PM, a designer, and about four, maybe four to six engineers. And I do back of the envelope math and come up with about $20,000 a week for my team's costs. And that could vary. That could be, you know, depending on your team size or your team's experience and compensation level, that could go higher. That could be 20 to 30,000. You might find a number that. Is easy to work with mentally, but is closer to your actual cost. And that, that could be good. You can ask finance or account or, or maybe HR to give you a real number if you really, really want to have something accurate. But I just think, you know, your typical, you're typically talking about $2,500 per person per week. And that engineers, product managers, whatever, it's all about the same. There's obviously going to be additional compensation factors. Like. Stuff that you do training or travel or whatever that comes up. And so you're going to might push that up to $3,000 a person per week. And then you've got to, and that's just for salary. So you've got to think about fully burdened. You've got to take into account. All of the HR costs, insurance and. Taxes and all of that. And in the U S that's another, about another 35%. And then you add onto that. Some kind of overhead, you know, you've got management. You've got managers, you've got leaders, you've got. Other roles that are in support of that's another 15 to 20%. And so I just, I just kind of do back of the envelope calculations, 20, 20,000 for a small team, 30,000 for a large team. And when I say small team, I mean, four devs and a PM and a UX and a large team is probably eight. Not a six to eight devs, maybe a PM in the UX. Right. But again, it's just rough numbers. Right. And I use those. I use those to find out to, to guess, like if, if we, if we think this is going to take us two weeks. Do I think this feature is worth 40,000 to $60,000? Is this feature going to return? Is the ROI there for this feature at a cost of 40 to $60,000? And you know, how long will it take to return that investment? And is there something else I could do that would return more sooner? So these, these are the kinds of things, things I think. Think about when I'm thinking about the actual financial cost of running that team.

[00:08:27] Allan Stewart: I don't remember exactly when it was, but I do remember that this idea of the fully burdened cost is something that stuck with me after I first heard you talk about it. Because I think previously I had thought about it often in terms of just my own salary, because that is something that I understood as just a developer on an average software developer team. And so when I started running the team, I was like, well, I don't, I don't know what everybody makes. I don't know what a lot of the costs are to a business. All I know is that this is how much money that I get. And so I think if you only look at it from the salary perspective, because that's what you get and that's what you're thinking about, it's really easy to miss a lot of the nuance of those overhead items. That don't get taken into the equation. often, right? Because your compensation isn't just salary, it's also benefits. But I didn't have to pay, they weren't paying me a salary and saying, okay, out of your pocket, you got to pay the AWS bill. And you also have to pay the HR lady for the fraction of time for your HR needs, right? Like it would be very different if that were the case. And so I really liked the fully burdened concept where you take those into account and you're saying, okay, salary times one point something, right? It's almost one and a half. It's almost one and a half. And that's just a good ballpark or like you said, across many teams, looking at that ballpark number, it often ends up around 20,000 a week.

[00:10:16] Dave Adsit: Yeah. For the teams that I run, I like to run small teams. And so, because that, that gives me a lot of flexibility. And, and so those teams tend to cost me about 20 to 25,000 a week. And so one of the places that I like to use this number is when I'm making a build versus buy decision. This is a really obvious one. You get an, you get a cost estimate for a SaaS product you want to use. And you're like, it's going to cost us this much. And we're like, well, but we could build it because we could always build it, right? We have the expertise. We know how to build software. But should we build it? Is it worth it for us to build it versus just pay somebody for it? And I think you and I, we, we leveraged this long ago when we were trying to decide whether to invest in some SaaS product that was going to help us control something in one of our systems. And we're like, well, we could build it. It would take us probably a month or two to build it. And

[00:11:12] Allan Stewart: that's no big deal. And in fact, we did build a version of it, right? It was sort of like the, the prototype version. Or like the, the initial version that was like, oh, well, we, we just throw this together real quick. But then we started to see is like, oh, well, we can't, we couldn't just rely on it running in perpetuity. Like we're going to have to invest more money in it to, to continue to make it grow, to improve the feature set. And so it's like, oh, well, should we just buy one instead?

[00:11:44] Dave Adsit: Yeah. And if you think it will cost us two weeks of developer work per year, across the year to maintain this software going forward, like that's $40,000 a year. Should we instead just buy the SAS product? That's going to cost us $6,000 a year. Yeah. Kind of, kind of a no brainer at that point, right? When you're making that comparison, but when you're saying, oh, it's not that big of a deal, it'll only be a few days a week or a few days a year working on this thing. We should just build it ourselves. You're like, oh yeah. And it'd be fun. I think that that's what gets us more often than not as engineers. And it would be fun a divergence from our standard work and, you know, it's a new puzzle and, you know, it'd be a

[00:12:28] Allan Stewart: lot. Yeah. Well, and oftentimes in, in cases like that, you are your own customer. And so, you know, what itches you want to scratch, you know, what you want the product or sub product to be like, and you don't have to ask anybody else. Like you can have your own vision. It's not, there's not questions about what will be needed. And so it's really tempting to say, let's do that because it's easier to understand and more fun to build. Yes. Then the hard thing where we have to do a lot more experimentation, where we have to do a lot more measurement of seeing, did customers actually use the thing? And even if they did use the thing, did it actually create the outcome that we wanted to create? Those are, those are much harder. Yeah, absolutely. I think the team cost definitely, it almost always outweighs the other costs. But it is worth thinking too, about your service provider costs. So if you're going to build software, well, what other things do you need? Are you going to be hosting it in a cloud? Well, it's going to cost you some amount of money. It's usually very small compared to the, to the fully burdened cost of the team, but it does exist. And, you know, if you're, they're going to also want to use some kind of time tracking or work tracking software and their, whatnot. Yeah. And whatnot. And whatnot. And whatnot. And whatnot. And whatnot. And whatnot. And whatnot. And whatnot. And whatnot. And whatnot. And whatnot. And whatnot. And whatnot. And whatnot. And whatnot. And whatnot. And whatnot. And whatnot. And whatnot.

[00:14:19] Dave Adsit: And I think like some of those things that you mentioned, for me, I consider to be bundled into my overall fully burdened, like the whiteboard and the time tracking and all that, I think as part of the overhead of having a team. And so that's part of that 10 to 20% additional overhead past the, but there's a lot of other costs. Like I don't include the cost to run the cost for the servers, you know, the compute and the network and the storage to actually run the thing that we built. You know, that's not part of your fully burdened cost. That's on top of. Yeah. So if we're, if we're, if we're running it, that's a good distinction. If we're running it in a cloud provider, then we pay for compute and we pay monthly for compute. And if we're running it in a colo, then we are paying one time for compute and storage and monthly for network and, you know, facility. And if we're running it locally, we're paying with our very self. No, we're, we're paying with, we're paying, basically paying for all of those things. And so you can, you can balance your costs in a lot of different ways when it comes to like, where do we run our systems? Where do we, like, do we, do we buy servers and stand up a rack and stand it in the corner of the room? Or do we have to build, what do we want to do? And, and that a lot of those considerations come down to, you know, what does your service look like? And what characteristics do you need of your cloud? And I, I've come down on because it's at the scale of businesses that I operate, it has been cheaper to just put it in the cloud because you, you can get away with having fewer operators. that, like, that's just the scale I'm at, right? If you were at very large scale, you would make different, different trade-offs because your cost characteristics would be so much different. But also I've already admitted, we're not very good at estimating. I'm not very good at predicting how much hardware I need and when I'm going to experience load spikes on my system. And I sure do like the fact that in a cloud, I can have elastic bursting, elastic capacity, and I can burst when there's a traffic burst and I can go back down to a lower cost when there's a, when the traffic lulls. And, you know, all of those things are beneficial from cost management from my perspective. So when we take these things, your, your team's fully burdened cost

[00:17:13] Allan Stewart: and your server's provider costs, well, now you can start to understand that rate we were talking about, like, okay, this much per week or divide it by five and this much per day or however you want to look at it or per month. But then we have to apply those rates across a number of different phases of development. And this is where I think this is the other side that gets people because we've, we've outlined a few different phases here. So we've got the design and discovery phase development, your operating costs, maintenance, enhancement, and then retirement or end of life. And each one of these phases costs you something. None of them come for free, right? So, so starting with design, what kinds of, what kind of costs do we have in design?

[00:18:03] Dave Adsit: Well, at this point, a lot of your, your, your people costs, your, your, your staffing costs are going to be lower because when you're doing your initial design and discovery, you're looking at maybe a product manager, someone who understands the product vision. You've got maybe a designer involved and maybe one probably pretty senior engineer involved for feasibility checking and, you know, helping, design and run experiments and things like that. So you've got a pretty small team at this point. So your, your cost is lower when it comes to your staffing, but you're also paying a lot for, you know, the research that you're doing. Maybe you're paying a research team to help you go out and do market research, or you are paying for survey tools and email blasts. And you're, you're doing a lot of things to discover what is the value of this potential product is the value that I think is there. And so we want to do a lot of discovery. We want to do a lot of work in discovery to find out if the thing that we are trying to build is going to have a return. And so one of the costs there is time like that can take a while to actually validate an idea and iterate on an idea until you have something that actually proves its value enough to, to start ramping up the build. You know, we, we talk about different types. We can, we can build software to earn money, but we can also build software to learn about something, the market, the users, the capacity capability of the systems. Right. And so we're in a built in the design and discovery phase. We're in a build to learn phase where we're trying to validate and prove a market before we ramp up the team and go into the next phase.

[00:19:53] Allan Stewart: Right. And so when we think about that from like a cost basis, you combine that with the, with the understanding that the best learning will come when you have a 50, 50 probability of it being useful. Right. And so you can kind of imagine that half your money is

[00:20:12] Dave Adsit: going to get thrown away. Well, yeah, half of your money is going to learn about things that don't work. Right. And that's your optimal learning curve. You are learning the fastest when half your experiments are, have a negative outcome. Right.

[00:20:26] Allan Stewart: And so you have to learn to recognize that the value is in the learning, not in the things that are being built for the experimentation. Yeah. Cause, cause that's all, it's all kind of like a scaffolding that goes away in the long term.

[00:20:41] Dave Adsit: Yeah. You're, you're building prototypes and you're building experiments and you're building things that are half done and half baked. And, and that's a lot of those things will end up getting thrown away. Ideally you learn and you take the designs, but you don't re try to, you don't want to fall for the trap of taking your prototype to production. Yeah. Now that I think we've all done it at least once, probably more than once where we built a prototype. It looked good. And somebody said, ship it. And we did, and it wasn't actually production quality software. So, so after design and discovery, we validated an idea. We're ready to ramp up and actually build, build to earn instead of building to learn. Now we go into the development phase, the one that we're most familiar with. That's where we have a full team and we are, you know, we're, we're building features, we're churning stuff out, we're getting, we're, we're meeting the needs of customers and we're

[00:21:38] Allan Stewart: building an actual product that has a positive ROI. Yeah. And I think that there's a lot of danger people over-indexing on this part, because it's easy to imagine what the design and discovery phase is going to be like. It's easy to come up with a bunch of ideas, and it's like, oh, these are our dreams of what it should be. But they're oftentimes not validated. They're not really ready to go. And so people over-index on this developments section. It's like, well, okay, but once I get people actually starting to write the code, how, how much is that going to, how much is that going to cost? And oftentimes I find it is rooted in a project mindset. Like, oh, well I, you know, I've got these 20 features that I really want built. I've got it all laid out in my mind. And all I need to know is how much is it going to cost to get that done? And then it's done and it's over and the costs go away, the costs go away. And that's just not the case. And in fact, you know, so we're going to go on to some of these other phases that inevitably come about, but in fact, even the design phase, right. It's like, it's not a one-time in most cases, it's not a one-time design discovery phase, but that's happening concurrently. Speaker 2 up front, like if you're starting a new project or if you're in a new startup or something up front, the design and discovery phase might, you know, take up a bigger proportion of your budget.

[00:23:10] Dave Adsit: Speaker 1

[00:23:10] Allan Stewart: But if you're developing a product that you want people to continue using over the course of years, you're going to be doing that design and discovery alongside development all the way. Plus you also get operating costs and maintenance. Speaker 1

[00:23:26] Dave Adsit: Speaker 2 Well, and I want to say one of the things that I've seen over and over and over is we get specifically when we get stuck in the project mindset, projects are easy to reason about because a project has a start, a scope of work, a team and an end. And then you're like, okay, great project done. We did the project. It finished fine. We're it's over. But what we mostly build these days, what a lot of us are building all the time is products that, Speaker 1 do not follow that same pattern of a defined start, a scope of work, a team that executes on it and a defined end. A product is perpetual until its end of life. And so like you were saying, we have ongoing operating costs and people are familiar with those. They run servers. Okay, great. And also maintenance costs. So let's, let's dive in a little bit on operating costs when it comes to a, Speaker 1 a SaaS product. The, um, the thing that happens often is at the beginning, we, we start capturing data, whatever we're throwing it in RDS, we're throwing it at S3, we're throwing it somewhere in the cloud and it's cheap because there's not very much of it. And so we don't create any kind of, uh, data retention policies, even internally, we just keep everything forever and the costs just start to go up. And now, Speaker 2 are, yeah, our cost is continuously increased and art, are we still getting value from data? That's seven years old? Probably not. Um, are we doing anything about it? Are we managing that? Or are we just letting it grow exponentially? Uh, it, it's one of those things that you need to be paying attention to over time so that you can manage your operating costs for your system. We've also whatnot to our servers because we didn't build in good auto-scaling. We are in the process of correcting that. It's, it's almost the project. The project is almost done. And after which we will have auto-scale that we will just have to maintain for the duration of the product. But, um, We, we intentionally over, over provision servers because it was complex to add new servers to the cluster. Uh, and now that we've corrected that problem, we can run at a lot lower continuous operating costs we can handle bursty traffic. And so those are things that you need to consider. Like, do we want to put an extra engineering effort now to manage our operating costs? Or do we want to pay higher operating costs so that we can apply our engineering effort

[00:26:09] Allan Stewart: to something else? Yeah. And also, is it worth the engineering cost to lower the operational cost? Right. Sometimes it's not. Sometimes it's better to just say, you know what, I'm going to just keep handing them over a bigger check because it's not the right time to deal with that. Yeah. So that's operating costs. And the kind of going hand in hand with that are your maintenance costs. This is where you get things like security patches. This is where your obsolete dependencies live that will eventually need to be upgraded or replaced. This is what happened. This is the entropy of you created something and it doesn't last pristine forever. It doesn't stay in isolation and you don't want it to. Because you want people to use this product. You know, it's not, at least I don't know of anybody who creates software like they would create, say, a Fabergé egg and say, okay, here it is. It's done. The end. You want people to keep on using the software. And if they're going to keep using it and you're in the world, well, the world changes and you're going to have to change along with it, which means you're spending time, which means you're spending money. On keeping all the things that you did in the past going. And so all the things that you've done in the past had better be valuable because they continually cost you more money.

[00:27:37] Dave Adsit: Right. There is an ongoing maintenance cost forever when you are running a software product. We've previously talked about it in terms of software rot, right? The bits don't degrade. However, the world around you changes. They release new versions of the SDK. They release the dependencies. They release new versions. People find new exploits for previous SDKs and versions of the things. And the third parties that you integrate with continuously release new API versions. And they expect you to move to the new API version. And there's new features that are available. Also, the old one may no longer be documented. And you may run into trouble of trying to figure out how to do maintenance on very old parts of your system when inevitably something breaks. Because you have failed to do maintenance. In manufacturing, they talk about it in terms of total productive maintenance. And I talk about it in terms of total productive maintenance with my dev teams. We need to be doing a certain amount of maintenance to keep the machines running. Yes, we built the feature. The feature works. We just need to keep maintaining the software so that the feature continues to work. If we don't maintain the software, the features will eventually stop working because things will not work. Things around them will change. And maintenance is different from enhancement. Maintenance is a cost you have to pay just to keep the existing features working. And almost inevitably, when we are building software, we want to enhance it. And that goes back to our original design and discovery and development. It's those phases over again on top of software that's already operating, which is cheaper, right?

[00:29:25] Allan Stewart: Unfortunately, it's... Often not. Just because you have all of these additional constraints that are upon you, right? Nobody wants to continue paying you for your product while you take it offline for two months to do a big change. And so oftentimes, the enhancements are a lot more expensive because you have to keep everything... I've heard it described in various ways. I like the idea that you've got a race car. And you need to change the engine and the tires and replace basically everything in the car while it's still driving. And also, you have to win the race. Yes. But there are some things that can help, right? I mean, we call this the Crafting Code Podcast. We think that the craft of code, or in other words, the quality and professionalism of code matters. And we do, in fact, see that... Yeah. You know, in the world of Crafting Code, well-factored code, depending on, you know, however you want to think about this, code that was intentionally crafted is cheaper to maintain and easier to enhance. Yes. And so it has a lower overall cost of ownership, even though it has a higher upfront cost to get it right, to get it good in the initial phases.

[00:30:52] Dave Adsit: Right. That's been my experience throughout my career. The systems that I've worked on. where we have done minimal maintenance, minimal, the enhancements have always been very challenging. You find, for example, lots of duplication of functionality. We've implemented this same concept five different times in the code base and I made an enhancement and I only caught four of them. And so now there is a defect in the code or inconsistency in the code and it takes time to discover and resolve, right? So those enhancements take longer if it's poorly factored. If it's well factored and I find that, oh, I've implemented this twice before, this is the third time I'm going to refactor all of these things into a single concept and give it a solid name and then reuse this from a single location. Now, later when I come back and I'm like, hey, I need to make an enhancement. Oh, great. Here's the place to make the enhancement. All the tests exist. I can, you know, I can add my tests for the new feature, the new functionality that I want. And then I can, I can run all of the tests for the test suite and look, they all pass and look, my code is good. That, you know, having well factored, well tested code seems like it's going to cost you more and it may cost more in the design or development phase, but it costs you less in maintenance and enhancement, which is where you spend most of your time.

[00:32:15] Allan Stewart: Right. Martin Fowler has a article that I really like. It's called, is high quality software worth the cost? I feel like it, it discusses this quite a bit. I also like thinking about software, a little bit like cooking. You're going to, you're going to get in and you're going to make a mess when you cook. And so you got to, you got to do it well. You got to clean up as you go because otherwise the craft gets overwhelming and now you can't actually do the things that you want to do because you've got, you've got the big dishwashing project that you have to do before you can cook the next meal. Yeah.

[00:32:54] Dave Adsit: Yeah. Yeah. That's, that's definitely a clean as you go makes for a much more pleasant environment overall. It makes more, a more habitable code base and a more pleasant kitchen. So the final phase that we have on our list for software products is retirement or end of life. And that should just be free, except that much like every other phase, it requires people to do things. And while people are doing things, costs are being incurred.

[00:33:22] Allan Stewart: Yeah.

[00:33:23] Dave Adsit: Right. Turning off your servers, shutting down the service, even if you've already managed to get all of your users off, which is not a guarantee, doing the end of life tasks so that you can stop paying for the software requires work and effort. And all of that comes at a cost. And it may be a small cost. It may be short in comparison to the overall value that you've accrued from the software, but it can't be completely ignored.

[00:33:48] Allan Stewart: And I think it does get ignored sometimes because we often think about, you know, to think about it in terms of the end of life of an entire product, or it's like, okay, this whole thing is done now. And even when that happens, it's not entirely true, right? We'll just ask Microsoft about their Windows 10 end of life and how that's going for them, or maybe their IE6 end of life. But I think it's a lot healthier to think about it in terms of a continuous retirement, not of the whole thing, but of pieces. And saying, hey, we've, as our product grows, as it evolves, what do we still need? Right? Or maybe it's a little bit like pruning your tree. If you want to shape it and have it look really nice and be healthy, well, sometimes you have to cut some stuff off. And software products are no exception to that. And if you account for that, you can pay that cost to retire something, which, then no longer produces maintenance costs, and it makes it cheaper to continue to build your new stuff. So I think that that is an important aspect that you should not forget about when you're wondering, how much is this going to cost me?

[00:35:07] Dave Adsit: Well, and to that point of continuous retirement of components of your system, I have a very relevant and timely example from my own team. One of my engineers came to me and said, hey, it looks like we're paying, you know, several hundred dollars a month for this memory database. We disabled the last feature that uses it about six months ago, but it's still running. Should I turn it off? And the answer is yes, please turn off that unused service inside of our cloud. Also, we skipped the retirement phase, or we took shortcuts where we said, ah, maybe something else uses it. Let's just leave it up. We, this may be the last feature that uses that memory database, and it may not. So let's just not turn it off yet. We'll remember to check later if it's been used. And we didn't remember to check later. And so we've been paying ongoing maintenance costs because we took shortcuts during our retirement phase for a piece of functionality in our system. And sometimes you catch those and sometimes you don't. If you have a big enough, and complex enough cloud bill, you may miss thousands or tens of thousands of dollars worth of wasted spend for ever indefinitely, because it's, you know, you took shortcuts in some of your cost management.

[00:36:35] Allan Stewart: So now we have an idea of how much things cost in terms of rate. And we think about these cost phases so that we're not, we're not, we're thinking about it holistically and not, myopically on one particular aspect of, especially the development aspect. And so now we can start talking about how it costs, what a, how much will a thing cost? Let's see where, knowing those costs, where does that help us? I think you already mentioned one, build versus buy. Yeah. Once you have a good idea of how much a thing will cost, and oftentimes it's just, it's ballpark, right? Like, oh, well, if we built it, we think it will at least take us a few, this many days per year, or this, you know, we'll probably have to do something with it once a month. So you get an idea and say, okay, well, is that more expensive or is buying the, is buying the thing more expensive? But you don't always, you don't always know, or there's not always a SaaS product or other, or non SaaS product that you can buy. It might not meet the needs. It might not, it might not be what, what is required. So the other one that you also mentioned, but let's talk about some more is making bets. And what I like about this one is that we're, we're moving it away from that fixed cost mentality. When you start thinking about software costs as a rate, then you can say, well, how much am I willing to invest in something or towards something? And, and I really liked the bet concept because not all bets pay off.

[00:38:23] Dave Adsit: Yeah. I like the concept of, I like thinking in bets because it introduces the concept of risk. We, sometimes we think that we have solved the problem and we know exactly what's going to happen. But when we say, well, how much are we willing to bet that we have the right answer? Now, all of a sudden you're, you're reintroducing the concept that this is actually not sure a sure thing there is risk involved, but also I think it gives us an opportunity. And we've talked about this many times, you know, many times of flipping the script from a scope based software development mindset to a time based. I want to bet two weeks of work that we can solve this problem. And then if we work in a, in a clear, in a clean way, we can say, Hey, we worked on this for two weeks. This is what we got done. Should we ship it? Versus, I mean, I don't know how many times I've experienced this, but you get to the end of a thing and a stakeholder comes in, and it says, I only thought this would take a couple of days. I thought it was simple. If I had known it would take over a month, we never would have done it. And then everybody feels demoralized because they thought they were doing great work that was benefiting the company. But the stakeholder had information about the value of the work product that they were working on that was not communicated effectively. Like I thought that I was betting $10,000 to on this feature. And. It was going to have a positive ROI, but when I ended up paying $80,000 for this feature, now the ROI is guaranteed to be negative. Right. So yeah, I don't even know how many times I've heard that every time it's, it's demoralizing to both leadership and to the engine, the engineers on the team, because everybody wants to be building value and creating value for the organization and the customers.

[00:40:12] Allan Stewart: It definitely takes a mind shift mindset to shift because it's so easy to think about, oh, well these are the features that we want and we start enumerating them and here are the different cases. And pretty soon we've built up a fixed scope. And as soon as we fixed the scope, we don't know how long it's going to take, except for that. We do know it's going to be longer than that. Whatever your, whatever your optimistic estimate is, it's wrong. Well, even your pessimistic estimate is, is most likely wrong as well. And so it's always going to be more, it's always going to be nebulous. Yeah. But time boxing is very like you get to the end of the time box and then you look and you say, okay, well what have we accomplished? And, and I think one of the mistakes I've made in the past is that I have fallen into the trap of, well, let's just take this scope and try to fit it into a time box. How much of the scope can we get done? And that it doesn't really work very well because it's like, oh, well we only got a small piece of it. Okay. Well we don't know if it's going to work or not. Still. So we still, we're going to keep adding more and more time boxes that are completely arbitrary, trying to get to a certain scope. Instead, you have to change the thinking process about it to say, Hey, here is an outcome that we want. We're willing to spend two weeks. We're willing to spend four weeks at the end of the four weeks. What, what, how has that outcome changed because of what we did during that time? Yeah. And then you say, Hey, I think this is good. Let's let's do more of it. Or you say, no, that's good enough. Or you say, you know what? That failed and we're not going to throw bad money after good. So we're going to change to something else.

[00:42:02] Dave Adsit: Right? Well, and, and so from my perspective, this is where the real engineering work happens in software development, because engineering is all about dealing with your constraints and the constraint of, we are only willing to invest. This much time, which equates to this much money to build this feature. Okay, well, great. We can't build it. We know we can't build the whole thing. What can we build of the superset that we want? Is there something valuable in there that we can get? What's the most important thing that we could deliver? If we could, you know, we often talk about how 20% of the features create 80% of the value. Well, let's find those 20% of the features and build them first. And then if we still have time, we can continue working on the other 80% of the features and trying to capture that other 20% of the value. But if we only get the, if we only get this one feature or this one feature, these two or three related features, they're the highest priority things are the most valuable, the most customers we use them. They'll bring in the most revenue and we only get those. Sometimes it was still a good bet, even though we did not get the entire scope that we wanted. And sometimes when somebody says, Hey, the bet, the bet that we're willing to make is two weeks. You look at the scope and you say, we can't get any thing of value done in two weeks. And so now you have a choice. Do we want to reevaluate the value? Do we want to just skip this thing entirely and work on something that's going to create more value or something that will create value within the bet that we are willing to make?

[00:43:39] Allan Stewart: Or are you willing to make a bigger bet?

[00:43:41] Dave Adsit: Yeah. Right.

[00:43:42] Allan Stewart: A risk, a much riskier bet. Mm-hmm.

[00:43:45] Dave Adsit: With the hope of a payoff. Yeah. So there's a lot to, a lot to think about. We could probably talk for an entire hour just about making bets in software and which ones we think are valuable and which ones are not, but let's move on for now and talk about other things that are expensive that we do in engineering like status meetings.

[00:44:07] Allan Stewart: Yeah. Status meetings. Well, so meetings in general can get expensive, right? Because we need to. We need to communicate. We need to be able to talk through ideas. We need to be able to get alignment on a vision of what we're trying to do because very few software products that are large and successful are done by just one person who can have it all in their head. So you're going to need the meetings. It's just, they're expensive, right? As soon as you, as soon as you put one person into the meeting, then you're taking that portion of their, their burden cost and you're applying it to the communication overhead of working with, with people. Right. And so, so some kinds of meetings I think are worthwhile, right? Like training meetings. Those are an investment. We're going to invest some money now to improve ourselves so that we can be more efficient so we can be better educated, better able to deliver features later. Right. So that's kind of, it's, it's like the sharpen your saw kind of concept, right? We keep, keep things working well, but status meetings are basically entirely pointless. It's it's, it's, it's just throwing money away because status reporting ought not to

[00:45:33] Dave Adsit: be done in a meeting. Yeah. One of our friends often says status is not a meeting. Status is something that should be visible. You know, we want to make our work visible. We want to share what's what the status is to anyone who wants to know it at any time without having to meet. If you think about, if you say, Hey, it's $20,000 a week to run my development team and we work for 40 hours a week, that is $500 an hour for a meeting. If you've got everybody there is updating one person on the status of a product or project or feature enhancement or whatever, is it worth $500 or is there some other way that you can communicate status? Yeah. And the, the, the truth is there is definitely, there are definitely many other ways you can communicate status. My teams all use virtual Kanban boards. They use virtual Kanban boards because we are distributed across many geographies. If we were in person, I would still prefer to just use a physical Kanban board on the wall of the dev room. But a Kanban board tells you at a glance, what is completed, what's in progress and what's up next. That is the, that is generally enough status. If we need more status than that. We can look at things like higher level epics and things like that. But you don't need to call a meeting to communicate status. And if you do need to call a meeting to communicate status, you're probably doing something wrong. You may need meetings for things like alignment and planning. Those are critical aspects of delivery. I think it's critical for my teams to do a retrospective at least once, you know, once a week. And that allows us to improve on our work and continue to work in a better and better way. You know? And I think it's worth investing in improving my team's mechanisms for work. The same is true for training. Training is invaluable. You can often get well more than you invest out of a training meeting. If you, you know, do something, if you do a training that is on point with what the team is doing.

[00:47:32] Allan Stewart: Yeah. And even if you, even if you don't do the work to, to make status visible, which I think is the ideal, right? You've got to use the means to make a good decision. to make a good decision to make a good decision to make a good decision to make a good decision to make a good decision to make a good decision to make a good decision to make a good decision to make a good decision to make a good decision to make a good decision to make a good decision And meanwhile, I'll do something else. You can do it at your leisure. And then we find out. And that's better than calling the meeting.

[00:48:19] Dave Adsit: Yeah. What's the meme? This meeting could have been an email. A status request could have been an email or better yet, a Slack message. Yep. So you and I are both fans of lean software development, lean methodologies. Lean talks a lot about waste and eliminating waste. And there are seven types of waste in lean. And they each apply to software development in one way or another. And the wastes are inventory, overproduction, extra processing, transportation, waiting, motion, and defects. So let's go through each of those in order and see how those map to software processes.

[00:48:57] Allan Stewart: The first one, the waste of inventory maps to partially done work or work in process. Um, so basically while you are. Building the feature, it's not worth anything until you can ship it. And so the longer that it sits around partially done worse, it is. No, I think while you're working on it, if you're actively working on it, well, that's just, that's a cost you have to pay. The danger is that we have this thing that's partially done and we set it aside. It's not delivering value, but it exists as inventory. And so software inventory is often well, that code is. There. And if we make changes, the compiler is going to complain unless we change that code too, or, um, or, or we have to. Con you know, spend costs to get back up, up to speed on it when we want to think about it again later when we're, we're done having it in process. So that's how I think about the inventory a lot of times, but although, but I also think about it in terms of design inventory. Oftentimes be because. Yeah. It's much easier to imagine what the product could be like than it is to actually build it. It's very easy for product managers and UX designers to go off and build out years worth of work that you may never actually get to.

[00:50:25] Dave Adsit: Yeah. And one of the things I think is that, you know, in manufacturing, you can't do manufacturing without some amount of inventory. So it's actually excessive inventory that becomes waste. Right. Um, but you could consider all. Yeah. You could consider all of the inventory to be waste until it's actually turned into usable product and sold because even a completed product is still inventory until you have exchanged it for currency. So in software, the thing that I always say is it is better to have one feature completely done 100% done and delivered than it is to have five features, 80% done one feature that's a hundred percent done reach has a positive RO has the potential to return. Yeah. On the investment and five features that are 80% done. Do not. This is why you get these concepts of it's done, but it's not done, done. Right. Like I did all the coding, but I haven't tested it. I haven't deployed it. I haven't released it, whatever. If it's not in the hands of customers, it's not, it is not capable of returning value. And so all of that is inventory and needs to be managed very carefully. This is why it's so critical that we limit our work in process and focus our energy. focus on delivering completed work in order to minimize that waste. So the second waste is the waste of overproduction in, in, you know, manufacturing systems. This is, you know, we have one system that runs really quickly. And so it creates a whole bunch of intermediate parts. It's similar to, you know, it's similar to the waste of inventory, but overproduction, we have stuff that we can't sell whatever. And that maps to delivering extra or unneeded features. You know, sometimes gold plating is what you would call it, where I spend a bunch of time working on unnecessary features. So if I have a continuous delivery system and I have a, a, and I say, Oh, well, we, sometimes we want to change the text on this button. Okay. Well, I can just change the text on that button and then ship it. Or I can, I can build a system that allows someone else to change the text of that button by a UI that writes to a database that's read from the database and cached and this and that and the other. That is all of that is, can be considered waste because it was unnecessary to provide a way for someone else to change the thing without talking to a developer. If we could just update the text

[00:52:58] Allan Stewart: and release the code anytime we want it. Yeah. I don't think I have anything to add to that one. So let's move. On to extra processing, the waste of extra processing in manufacturing would be when you take longer or you're, you're spending more resource than was really necessary in order to create a thing. And that maps in software to relearning because our creation is very thought based, right? It's knowledge work. It's not re it's really not about your keyboard typing speed. It's about how you're applying what you know, into code. And anytime that you have to go and relearn something because you're, you're revisiting some piece of code that you haven't, haven't touched in a while and there were bugs in it, or because it got set aside or whatever these reasons are where you have to say, Oh, well, I don't know. Let me go and relearn this. That's waste.

[00:53:57] Dave Adsit: Right. And the waste of transportation in manufacturing, it costs money to move things. And so if you can arrange your system to minimize transportation, you can save, you can reduce waste and save money, whatever, save money, time, whatever it takes in engineering that maps to in software development and maps to handoffs. If I, if the PM does all the discovery work with no engineer present, and then hands off the completed specs to the engineer who then has to understand it and then build something and then hands it off to the, the QA team who has to go back and understand all the specs and understand the software and validate all of these handoffs are, are waste. If we instead arrange our teams in a way that we are working collaboratively, we can reduce the cost and frequency of handoffs. We can, we just eliminate those handoffs entirely because everyone knows what we're doing the whole time.

[00:54:55] Allan Stewart: The waste of waiting maps to delays. And I think this one is pretty simple. This is, this is the, this is the situation where the team is working on something and all of a sudden a question comes up, right? There's a, there's a technicality or there is a something about the user interface, maybe that like, oh, well, did they want this? Did they want that? We don't know. We have a question and the person who can answer the question is not there. And so we have to delay. We, we wait and we wonder what, what is this, you know, what are we supposed to do about this? And that,

[00:55:32] Dave Adsit: is waste. Well, and in my experience, that waste often turns into other kinds of waste. That one is a, is a catalyst for creating work, the waste of partially done work and context switching and handoffs and all kinds of others. So be very, very aware of the waste associated with delays. The next is the waste of motion, which maps to context switching, which context switching. One of my favorite things I, at one of the conferences I went to a while back, they shared some research that, When we contact switch, When we contact switch, When we contact switch, When we contact switch, When we contact switch, When we contact switch, When we contact switch, When we contact switch, When we contact switch, When we contact switch, When we contact switch, When we contact switch, When we contact switch, Switching becomes very, very expensive when we are trying to work on multiple products or multiple projects simultaneously.

[00:56:38] Allan Stewart: And it feeds into the relearning waste that we talked about, too, and the delays. Right. Because if you have to switch contexts, then you have to pick up what you were doing. And sometimes sometimes it's just, OK, I'm switching context. I've got a rough idea of what it is. But other times it's, well, I didn't write it down very well. Like maybe I maybe I wrote my code really poorly. And this variable is just called X. And I don't know what it is for. Well, now I'm relearning because of the context switching. And so they yeah, they like to gang up on each other. Yeah, for sure. And then the final one, defects are defects. So manufacturing defects are one kind of thing. And we call them software bugs. And ultimately, that is there. There are a lot of reasons for bugs. But I think more often than not, most of the bugs that I have seen is just that something wasn't thought through very well. It wasn't it wasn't crafted. It was just kind of thrown out there. Hey, I got it working once. And so it must be done. And it was never refactored. It was never cleaned up. The naming was never changed. And so you end up with a problem. And now you have to go and fix the problem that you created for yourself.

[00:57:58] Dave Adsit: Yeah. And I have strong opinions about software defects. I feel like a lot of us don't work in an environment where we could actually know if we have defects because we don't write robust enough specs for the software. And so a lot of times what gets labeled a defect is really previously undefined behavior that was only only became defined when a user tried to do something and we realized what the software did. But regardless, at the end of the day, you're going to have to make a change. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Because we failed to meet the primary need. So you need a feature. I haven't given you a feature. Now we have a status meeting. That status meeting is secondary work. It was completely unnecessary if I had just given you a completed feature that you could use. Right. And we see that happen throughout the system in software, where we build sophisticated processes to handle all kinds of secondary work.

[00:59:20] Allan Stewart: Yeah. My favorite example of this is the backlog grooming meeting. Because we get lots of ideas for software and we say, yes, we should do these things. These ones are bugs that somebody has reported. So obviously we've got to fix all the bugs. Plus there's these features that we want to build. There's that really good idea that we had. Oh, and also there's these technical things that we need to do. And we're going to just write them all down in a big old list. So we don't forget. So we don't forget. Because you wouldn't want to forget anything. Right. You've got to keep that mental load perpetually. So you write it down somewhere. And I guess you're offloading the mental load onto a paper so that you can load that mental load up again. So that every two weeks or month or whenever you do your backlog grooming, you can try to remember what all of these things were. Try to guess what their relative priorities are to each other. And make sure that, hey, yes, we have groomed this backlog. Because we don't want to throw anything. And we're going to create this list, this ordered list. And anything that's past, it's like past the page fold doesn't matter. Right. It's like, I don't know how many it is. It's going to be maybe, it's small. It's probably like only 10. Maybe five. Yeah, five. And anything after that, if you spend any time on it whatsoever, even to look through it and see, hey, are any of these, should any of these get lifted up? Then you've just wasted time.

[01:00:51] Dave Adsit: Yeah.

[01:00:52] Allan Stewart: Because all those things at the bottom. You're not going to work on. You're not going to get to it.

[01:00:57] Dave Adsit: Backlog grooming seems like an important part of the process. Right. But backlog grooming only exists because a backlog exists, which only exists because we fail to deliver on the needed software. And so it's, you know, it's a secondary effect, a secondary work of a secondary work. And there are ways to completely eliminate that need. Right. We just don't maintain a backlog. Every, if each time a new idea comes up, we say, is it more? More important than the ones on the list. Now, if yes, one of them gets bumped and it gets added. And if not, if we, if we have a whip or a, a cap on the number of items we'll keep on our backlog or to do list, we can completely eliminate all of that. Yeah. And the thing is, you're never going to forget the good ideas. You're probably going to forget some of the like mediocre or bad ideas, but they're, they were mediocre or bad to begin with, if it was a really good idea. Yeah. Yeah. You would have moved it to the top of the list and bumped something else off. Yep. Let's see. What other kinds of secondary work are important to touch on? A perpetual favorite of our teams are estimation. How many times have we said just in this podcast that we're bad at estimating? Probably a lot. Just in this one conversation, we've mentioned being bad at estimating multiple times. Software developers are bad at estimating. First of all, they're all way too optimistic. Right. Right. Right. Right. And when they try to be realistic, they get pressure to be optimistic instead, which doesn't actually make the software easier or faster to develop. It just makes the estimates less accurate. And none of us has ever really gotten good at estimating well, because we're always doing something new.

[01:02:39] Allan Stewart: And I think that to get better at estimating is not necessarily worth the cost, because it takes a lot of time and effort to refine and improve your estimation. And oftentimes you have to actually start doing the thing to know what it was going to take. Right. You can only know it in hindsight. And so doubling down on, well, we're just going to do better and better at estimating our estimation process, I feel like is just a losing battle. Right. It's a, it's diminishing returns.

[01:03:13] Dave Adsit: Yeah. Well, and so I've, I've heard that there are some doctors that are very good at diagnostics, you know, diagnosing. A patient and there's some doctors that are not very good at diagnosing a patient. And the difference is the ones that go follow up after the fact to find out if they were right. And if we want to be, if we wanted to be good at estimating, we would have to spend extra time reviewing our estimates after the thing was finished to see how much we were off and by what and why. And then we'd have to take all of those things into account. And we would have to be working on small enough projects that we could reason about. Right. So we'd have to be working on a whole project while we were doing the estimating. And often that's not the case either. Right. And so these things become very challenging. And I think you're right. In most cases, it's not worth the cost to improve at estimating. So can I talk about one of my favorite kinds of unnecessary secondary work? Absolutely. CapEx. Capital expenditure. This is definitely a quirk of the U S tax code and will be a project driven by your finance team. Okay. Okay. So I want you to determine which, what, what work is capital expenditure, which is building new features versus maintenance, which includes things like bugs and operations and whatever. And why do they want to do it? Because you can do fun things in accounting. If you know what your capital expenditures are. However, this is a huge waste of time for a few reasons. One of which is that software bugs. Are virtually impossible to define without a spec. And the reason I say that is because you don't know if it's a defect, if nobody ever wrote down what it was supposed to do in the first place. And so all you ever know is we have software that works this way. And we want it to work that way. And we have to do work to get from here to there. Is that a feature? Is that a bug? I don't actually care. It's work that has to be done to move from a to B. We want to be at B we're at a, we've got to do work. Okay. I would categorize personally, I would categorize every single thing we do in software development as a new feature because none of the teams I work with have the super robust written specifications. Like you would have in a, in an engineering. Environment.

[01:05:37] Allan Stewart: And it might be cheaper to do it that way too, right? Like the tax write-off that you're going to get from your. CapEx allocation may not actually warrant all of the time that it takes to get all of your people to do it. the first place. How do you tell them that this was an operational expense versus a new feature

[01:06:27] Dave Adsit: development? It's almost impossible to say. Well, and the good news is there are actually certain heuristics that you can use that will just allow you to escape by any auditor. They'll just accept it. And my understanding is it's 80-20. You say 80% of our work is building new features and 20% is addressing defects or doing operational work. And nobody will question that. And if you want to get more aggressive than that and say it's 90-10, then you better be able to back it up. And that means that you're tracking every piece of work that you do has to be tracked back to a ticket that was approved by someone who had the approval ability to make software change, blah, blah, blah, blah. It's a bunch of work. It's a bunch of work that if you do it well, it falls almost exclusively to leadership and not to the engineers on the team. And then the engineers can stay focused on their value add, which is, building new features and enhancing the system at hand. But yeah, to me, the whole thing is just a very fancy form of secondary work, AKA waste. So software costs, like I said at the beginning,

[01:07:35] Allan Stewart: it comes up a lot. When people are thinking about starting something new, when people are wondering what the new feature is going to be or the new app that they want to build, anytime that you're already developing a product and you want to know what the new feature is going to be, what the next feature is going to cost, all these things are costs, right? Like eventually you're taking, you're paying your developers to do some software work and you want to know how much is this going to cost? So hopefully some of the things that we've talked about today will equip you to better have that discussion. I know that they have improved my ability to discuss and make rational arguments and decisions with leadership in the places that I've worked. So that you can say, Hey, this is, this is the rate. We can start thinking about things in perhaps a different way. Instead of scope-based, we can think about it more time-boxed approach. Maybe you can think about places where you can address some low hanging fruit, like not spending money on services that you're not using or eliminating some waste within your product or a status meeting that isn't helping you out. But I hope that something that we've said here is useful. As you're considering what it costs and what it's going to take to build software.

[01:08:55] Dave Adsit: Well, and to paraphrase Don Norman in Principles of Product Development Flow, if you want to have an effective conversation where you influence a manager or business leader, speak to them in terms of money, reduce it to money, and you can have a much greater impact on decisions. And that's what this is all about.