Crafting Code Podcast

[00:00:16] Allan Stewart: Welcome to the Crafting Code Podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately I've been thinking about the impact made by a friend who gave me a useful programming book back in the days when I was first learning and documentation was scarce.

[00:00:34] Dave Adsit: I'm Dave Adsit, a VP of engineering, and recently I've been thinking about the problem of specialization as an organization scales.

[00:00:42] Allan Stewart: Our topic for this episode is software costs. It's a perennial question. How much does this cost? How much does writing writing software cost generally? How much does it cost to build a product? If there's somebody who has this great idea for a startup and they just want to know, well, how much is this software going to cost me? One way or another, this question comes up a lot. So I think there's an obvious question of why the question comes up, but why does it matter for the cost? What are they really trying to get at, do you think, Dave?

[00:01:18] Dave Adsit: Well, I would say in general, the question that I'm asking when I want to know how much it costs is like, "can I afford it? Can we build it? Is it viable? Is there actually a viable business inside this idea? Is it something that has potential?" Also, typically when you're starting out, you have a limited amount of capital. Either you've got your own seed money or you've raised money from an investor or something. And you're like, "we can go for this long with the cat." I want to know how much this costs because I want to know if I have enough capital to even do it, or if I'm just going to burn through everything in my savings and then we won't even have the product that I wanted anyway.

[00:02:01] Allan Stewart: So, yeah, I mean, it matters a lot because software is, is expensive, right? Like the, the great thing about software is that it is very malleable. You can do lots of things. It's kind of incredible sometimes all the things you can do with software, but it comes at a cost.

[00:02:20] Dave Adsit: Well, and if you're talking about established businesses, you like, "we want to get into this new business line. We think it's going to return this much. We think that we can sell this feature for this much, or if we add this feature, we'll get these customers, which will bring in this amount of money. Like, what will it cost us to to build this?" And a lot of times in software, time and money are synonymous because every hour that you have a software development team working is going to cost you something. And it's important to know how do you even figure out what that cost is? So there's various costs associated with building a software product. You've got to think about the biggest one is going to be your team. You know, you've got, of course, you've got costs associated with hardware and, and, you know, software that you're purchasing and things like that. But you think the biggest cost of building software is the team of professionals that are responsible for building the software. And usually for the teams that we work with, that's going to be several software developers, several software engineers, and someone with a product management, product owner type of a role, probably a designer, if you're doing anything that's UI intensive or online or whatever, if you're building a product that's customer facing. And it might include other roles like QA or DBA or DevOps. But at the very least, you're going to have a team of a handful of software developers and someone who is thinking about the product holistically and what needs to be built and how to prioritize the features.

[00:03:58] Allan Stewart: Yeah. Yeah. I feel like thinking about it in terms of a rate makes a lot of sense. Right. Right. Because, yeah, the time, the time and the cost are kind of interchangeable. And I think instead of thinking about one or the other, it's more like both. Right. It's it's not how many miles, it's not how many hours, it's how many miles per hour. Yeah. That matters because we it's just it's so impossible to know. Right. Like you might have ideas of what the product is going to be. You might have ideas of, you know, well, we've outlined these 40 features that we think need to be done, but it's very hard to estimate how long it will take to do a particular feature. And it's hard to know when the gremlins are going to reach out and you hit an unexpected hiccup that costs an entire day of work. And so, yeah, thinking about it as a rate makes a lot of sense. And I think it goes back to some of those ideas that came out like around like the Agile manifesto. Right. It's like we're thinking about working software in small batches, short iterations where you can just say, "hey, is this working for you? Is this working for you?" You have to spend a little bit to get into it. But then at each point, hopefully you have something to show for it.

[00:05:22] Dave Adsit: Well, and that's, I mean, definitely I like to manage my bets when it comes to the software that I'm building. But I also like to do back of the envelope calculations. I don't get over precise when it comes to estimating costs. I think that, first of all, we're bad at estimating. We don't know how long it's going to take to do something. Thing. So being over-precise on the, you know, the cost per hour or whatever, when we don't know how many hours is kind of a fool's errand. So I, for the teams that I run, I usually have a PM, a designer and about four, maybe four to six engineers. And I do back of the envelope math and come up with about $20,000 a week for my team's costs. And that could vary. That could be, you know, depending on your team size or your team's experience and compensation level, that could go higher. That could be 20 to 30,000. You might find a number that is easy to work with mentally, but is closer to your actual cost. And that, that could be good. You can ask finance or account or, or maybe HR to give you a real number if you really, really want to have something accurate. But I just think, you know, you're typical, you're typically talking about $2,500 per person per week. And that engineers, product managers, whatever, it's all about the same. There's obviously going to be additional compensation factors, like stuff that you do training or travel or whatever that comes up. And so you're going to might push that up to $3,000 a person per week. And then you've got to, and that's just for salary. So you've got to think about fully burdened, you've got to take into account all of the HR costs, insurance and taxes and all of that. And in the US, that's about another 35%. And then you add onto that some kind of overhead, you've got managers, you've got leaders, you've got other roles that are in support of, that's another 15 to 20%. And so I just kind of do back of the envelope calculations, 20, 20,000 for a small team, 30,000 for a large team. And when I say small team, I mean four devs and a PM and a UX and a large team is probably eight, not eight, six to eight devs, maybe a PM and the UX. Right. But again, it's just rough numbers. And I use those, I use those to find to, to guess, like, if, if we, if we think this is going to take us two weeks, do I think this this feature is worth $40,000 to $60,000? Is this feature going to return? Is the ROI there for this feature at a cost of $40,000 to $60,000? And how long will it take to return that investment? And is there something else I could do that would return more sooner? So these are the kinds of things I think about when I'm thinking about the actual financial cost of running that team.

[00:08:27] Allan Stewart: I don't remember exactly when it was, But I do remember that this idea of the fully burdened cost is something that stuck with me after I first heard you talk about it. Because I think previously I had thought about it often in terms of just my own salary, because that is something that I understood as just a developer on an average software developer team. It's like, well, I don't know what everybody makes. I don't know what a lot of the costs are to a business all I know is that this is how much money that I get, and so I think if you only look at it from the salary perspective because that's what you get and that's what you're thinking about, it's really easy to miss a lot of the nuance of those overhead items that don't get taken into the equation as often, right? Because your compensation isn't just salary. It's also benefits, but there are also, but I didn't have to pay that. They weren't paying me a salary and saying, "okay, out of your pocket, you got to pay the AWS bill." And you also have to pay the HR lady for the fraction of time for your HR needs, right? Like, it would be very different if that were the case. And so, so I really liked the fully burdened concept where you, where you take those into to account and just, and you're saying, "okay, salary times one point something," right? It's almost one and a half. Yeah.

[00:10:03] Dave Adsit: It's almost one and a half. 1.5.

[00:10:05] Allan Stewart: And that's just a good ballpark, or like you said, across many teams, looking at that ballpark number, it often ends up around 20,000 a week.

[00:10:17] Dave Adsit: Yeah. For the teams that I run, I like to run small teams. And so, because that, that gives me a lot of flexibility. Um, and, and so those teams tend to cost me about 20 to 25,000 a week. And so one of the places that I like to use this number is when I'm making a build versus buy decision. This is a really obvious one. You get an, you get a cost estimate for a SAS product you want to use. And you're like, "it's going to cost us this much." And we're like, "well, but we could build it because we could always build it." Right. We have the expertise. We know how to build software, but should we build it? Is it worth it for us to build it versus just pay somebody for it? And I think you and I, we leveraged this long ago when we were trying to decide whether to invest in some SaaS product that was going to help us control something in one of our systems. And we're like, "well, we could build it. It would take us probably a month or two to build it. And that's no big deal."

[00:11:13] Allan Stewart: And in fact, we did build a version of it, right? It was sort of like the prototype version or like the initial version that was like, "oh, well, we just throw this together real quick." But then we started to see it's like, "oh, well, we couldn't just rely on it running in perpetuity. We're going to have to invest more money in it to continue to make it grow, to improve the feature set." And so it's like, "oh, well, should we just buy one instead?"

[00:11:44] Dave Adsit: Yeah. Yeah. And if you think it will cost us two weeks of developer work per year spread out across the year to maintain this software going forward, like that's $40,000 a year. Should we instead just buy the SaaS product that's going to cost us $6,000 a year? Yeah. Kind of a no-brainer at that point, right? When you're making that comparison. But when you're saying, "oh, it's not that big of a deal, it'll only be a few days a week or a few days a year working on this thing. We should just build it ourselves," you're like, "oh yeah." And it'd be fun. I think that that's what gets us more often than not as engineers. And it would be fun because it's a divergence from our standard work and, you know, it's a new puzzle and, you know, it'd be a lot. Yeah. Well, and oftentimes

[00:12:31] Allan Stewart: in, in cases like that, you are your own customer. And so, you know, what itches you want to scratch, you know what you want the product or sub product to be like, and you don't have to ask anybody else. Like, you can have your own vision. It's not, there's not questions about what will be needed. And so it's really tempting to say, "let's do that," because it's easier to understand and more fun to build. Yes. Then the hard thing, where we have to do a lot more experimentation, where we have to do a lot more measurement of seeing, did customers actually use the thing? And even if they did use the thing to actually create the outcome that we wanted to create. Those are much harder.

[00:13:12] Dave Adsit: Yeah, absolutely.

[00:13:13] Allan Stewart: I think the team cost, it almost always outweighs the other costs, but it is worth thinking too about your service provider costs. So if you're going to build software, well, what other things do you need? Are you going to be hosting it in a cloud? Well, it's going to cost you some amount amount of money. It's usually very small compared to the fully burdened cost of the team, but it does exist. And they're going to also want to use some kind of time tracking or work tracking software. And they might want to have an online whiteboard software and you want to have your online video conferencing software. Those things do add up, and it's worthwhile to consider that, that especially like you were saying in a build versus buy, sometimes choosing a product like a sass product might make a lot more sense because if you were to build it, you would spend that much

[00:14:14] Dave Adsit: just in the in the service costs. Yeah, yeah, for sure. Well, and I think I like some of those things that you mentioned, I, uh, for me are, I consider to be bundled into my overall fully burdened, like like the whiteboard and the time tracking and all that, I think is part of the overhead of having a team. And so that's part of that 10 to 20% additional overhead past the, but there's a lot of other costs. Like, I don't include the cost to run the cost for the servers, you know, the compute and the network and the storage to actually run the thing that we built. You know, that's not part of your fully burdened cost. That's on top of. So if we're running it in a cloud provider, then we pay monthly for compute and storage and network. And if we're running it in a colo, then we are paying one time for compute and storage and monthly for network and facility. And if we're running it locally, we're paying with our very self. No, we're, we're paying with, we're paying, basically paying for all of those things. And so you can, you can balance your costs in a lot of different ways when it comes to, like, where do we run our system? Where do we, like, do we, do we buy servers and stand up a rack and stand it in the corner of the room? Or do we have to build, what do we want to do? And, and that a a lot of those considerations come down to, you know, what does your service look like and what characteristics do you need of your cloud? And I, I've come down on, because it's at the scale of businesses that I operate, it has been cheaper to just put it in the cloud because you, you can get away with having fewer operators who are doing patching and repairing and replacing servers. And And because that cost is such a small fraction of my overall budget, I would rather just pay a cloud provider than build those servers and run them myself. And part of that, that's just the scale I'm at, right? If you were at very large scale, you would make different trade-offs because your cost characteristics would be so much different. But also, I've already admitted we're not very good at estimating. I'm not very good at predicting how much hardware I need and when I'm going to experience load spikes on my system. And I sure do like the fact that in a cloud, I can have elastic bursting, elastic capacity, and I can burst when there's a traffic burst, and I can go back down to a lower cost when the traffic lulls. And all of those things are beneficial from cost management from my perspective. So when we take these things, your team's fully burdened cost

[00:17:12] Allan Stewart: cost and your service provider costs. Well, now you can start to understand that rate we were talking about, like, okay, this much per week or divide it by five and this much per day or however you want to look at it or per month. But then we have to apply those rates across a number of different phases of development. And this is where I think this is the other side that gets people. Because we've outlined a few different phases here. So we've got the design and discovery phase, development, your operating costs, maintenance, enhancement, and then retirement or end of life. And each one of these phases costs you something. None of them come for free. Right. So starting with design, what kind of costs do we have in design?

[00:18:03] Dave Adsit: Well, at this point, a lot of your people costs, your staffing costs are going to be lower, because when you're doing your initial design and discovery, you're looking at maybe a product manager, someone who understands the product vision. You've got maybe a designer involved and maybe one probably pretty senior engineer involved for feasibility checking and helping design and run experiments and things like that. So you've got a pretty small team at this point. So your cost is lower when it comes to your staffing, but you're also paying a lot for the research that you're doing. Maybe you're paying a research team to help you go out and do market research, or you are paying for survey tools and email blasts. And you're doing a lot of things to discover what is the value of this potential product. Is the value that I think is there actually there in the market? It. And so we want to do a lot of discovery. We want to do a lot of work in discovery to find out if the thing that we are trying to build is going to have a return. And so one of the costs there is time. Like, that can take a while to actually validate an idea and iterate on an idea until you have something that actually proves its value enough to start ramping up the build. You know, we talk about different types of build. We can build software to earn money, but we can also build software to learn about something, the market, the users, the capability of the systems, right? And so we're in a built, in the design and discovery phase, we're in a build to learn phase where we're trying to validate and prove a market before we ramp up the team and go into the next phase.

[00:19:53] Allan Stewart: Right. And so when we think about that from like a cost basis, you combine that with the understanding that the best learning will come when you have a 50-50 probability of it being useful. Right. And so you can kind of imagine that half your money is going to get thrown away.

[00:20:13] Dave Adsit: Well, yeah, half of your money is going to learn about things that don't work. Right. Right. And that's your optimal learning curve. You are learning the fastest when half your experiments are have a negative outcome.

[00:20:26] Allan Stewart: Right. And so you have to learn to recognize that the value is in the learning, not in the things that are being built for the experimentation. Yeah. Because because that's all it's all kind of like scaffolding that goes away in the long term.

[00:20:41] Dave Adsit: Yeah. You're you're building prototypes and you're building experiments and you're building things that are half done. And half-baked. And that's, a lot of those things will end up getting thrown away, ideally. You learn and you take the designs, but you don't try to, you don't want to fall for the trap of taking your prototype to production.

[00:21:02] Allan Stewart: Yeah.

[00:21:03] Dave Adsit: Now that, I think we've all done it at least once, probably more than once, where we built a prototype, it looked good and somebody said, "ship it," and we did, and it wasn't actually production quality software. So after design and discovery. We validated an idea. We're ready to ramp up and actually build to earn instead of building to learn. Now we go into the development phase, the one that we're most familiar with. That's where we have a full team and we're building features. We're churning stuff out. We're meeting the needs of customers and we're building an actual product that has a positive ROI.

[00:21:41] Allan Stewart: Yeah. And I think that there's a lot of danger, people over-indexing on this part. Because it's easy to imagine what the design and discovery phase is going to be like. It's easy to come up with a bunch of ideas and it's like, oh, these are our dreams of what it should be. But they're oftentimes not validated. They're not really ready to go. And so people over index on this developments section. It's like, "well, okay, but once I get people actually starting to write the code, how much is that going to cost? Is that going to cost?" And oftentimes, I find it is rooted in a project mindset. "Oh, well, I've got these 20 features that I really want built. I've got it all laid out in my mind. And all I need to know is how much is it going to cost to get that done? And then it's done. And it's over. And the costs go away. The costs go away." And that's just not the case. And in fact, In fact, so we're going to go on to some of these other phases that inevitably come about. But in fact, even the design phase, right? It's like it's not a one-time, in most cases, it's not a one-time design discovery phase, but that's happening concurrently with the development. Upfront, like if you're starting a new project or if you're a new startup or something, upfront, the design and discovery phase might take up a bigger proportion of your budget. It. But if you're developing a product that you want people to continue using over the course of years, you're going to be doing that design and discovery alongside development all the way. Plus you also get operating costs and maintenance.

[00:23:25] Dave Adsit: Right. Well, and I want to say one of the things that I've seen over and over and over is we get, specifically when we get stuck in the project mindset. Projects are easy to reason about because a project has a start, a scope of work, a team, and an end. And you're like, "okay, great. Project done. We did the project. It finished. Fine. It's over." But what we mostly build these days, what a lot of us are building all the time is products that do not follow that same pattern of a defined start, a scope of work, a team that executes on it, and a defined end. A product is perpetual until its end of life. And so like you were saying, we have ongoing operating costs, and people are familiar with those. They run servers. Okay, great. And also maintenance costs. So let's dive in a little bit on operating costs when it comes to a SaaS product. The thing that happens often is at the beginning, we start capturing data, whatever. We're throwing it in RDS. We're throwing it in S3. We're throwing it somewhere in the cloud. And it's cheap because there's not very much of it. And so we don't create any kind of data retention policies, even internally. We just keep everything forever. And the costs just start to go up. And now, yeah, our costs just continuously increase. And are we still getting value from data that's seven years old? Probably not. Are we doing anything anything about it? Are we managing that? Or are we just letting it grow exponentially? It's one of those things that you need to be paying attention to over time so that you can manage your operating costs for your system. My system, we have consistently over-provisioned our servers because we didn't build in good auto-scaling. We are in the process of correcting that, the project is almost done. And after which we will have autoscale that we will just have to maintain for the duration of the product. But we intentionally over-provisioned servers because it was complex to add new servers to the cluster. And now that we've corrected that problem, we can run at a lot lower continuous operating costs because we can handle bursty traffic. And so those are things that you need to consider, like, do we want to put an extra engineering effort now to manage our operating costs, or do we want to pay higher operating costs so that we can apply

[00:26:08] Allan Stewart: our engineering effort to something else. Yeah, and, and also, is it worth the engineering cost to lower the operational cost? Right, sometimes, sometimes it's not. Sometimes it's better to just say, "you know what, I'm gonna just keep handing them over a bigger check because it's not the the right time to deal with that." Yeah. So that's operating costs, and the kind of going hand in hand with that are your maintenance costs. This is where you get things like security patches. This is where your obsolete dependencies live that will eventually need to be upgraded or replaced. This is what happened. This is, this is the entropy of you, you created something and it doesn't last pristine forever. It doesn't stay in isolation, and you don't want it to because you want people to use this product. You know, it's not a, at least I don't know of anybody who creates software like they would create, say, a Fabergé egg and say, "okay, here it is. It's done, the end." Yep. You, you want people to keep on using the software, and if they're going to keep keep using it, and you're in the world, well, the world changes, and you're going to have to change along with it, which means you're spending time, which means you're spending money on keeping all the things that you did in the past going. And so all the things that you've done in the past had better be valuable because they continually cost you more money.

[00:27:38] Dave Adsit: Right. There is an ongoing maintenance cost forever when you are running a software product. We've previously talked about it in terms of software rot, right? The bits don't degrade. However, the world around you changes. They release new versions of the SDK. They release new versions of the dependencies. They release new version. People find new exploits for previous SDKs and versions of the things. And the third parties that you integrate with continuously release new API versions and they expect you to move to the new API version. And there's new features that are available. Also, the old one may no longer be documented and you may run into trouble of trying to figure out how to do maintenance on very old parts of your system when inevitably something breaks because you have failed to do maintenance. In manufacturing, they talk about it in terms of total productive maintenance. And I talk about it in terms of total productive maintenance with my dev teams. Like, we need to be doing a certain amount of maintenance to keep the machines running. Yes, we built the feature. The feature works. We just need to keep maintaining the software so that the feature continues to work. If we don't maintain the software, the features will eventually stop working because things around them will change. Well, and maintenance is different from enhancement. Maintenance is a cost you have to pay just to keep the existing features working. And almost inevitably, when we are building software, we want to enhance it. And that goes back to our original design and discovery and development. It's those phases over again, on top of software that's already operating, which is cheaper, right?

[00:29:22] Allan Stewart: Right. Unfortunately, it's often not just because you have all of these additional constraints that are upon you. Right. Nobody wants nobody wants to continue paying you for your product while you take it offline for two months to do a big change. And so oftentimes the enhancements are a lot more expensive because you have to keep everything. I've heard it described in various ways. I like the idea that you've got a race car and you need to change the engine and the tires and replace basically everything in the car while it's still driving. And also, you have to win the race. Yes. But there are some things that can help, right? I mean, we call this the Crafting Code Podcast. We think that the craft of code, or in other words, the quality and professionalism of code matters. And we do, in fact, see that clean code, well factored code, depending on however you want to think about this, code that was intentionally crafted is cheaper to maintain and easier to enhance. And so it has a lower overall cost of ownership, even though it has a higher upfront cost to get it right, to get it good in the initial phases.

[00:30:52] Dave Adsit: Right. That's been my experience throughout my career. The systems that I've worked on where we have done minimal maintenance, minimal, the enhancements have always been very challenging. You find, for example, lots of duplication of functionality. We've implemented this same concept five different times in the code base, and I made an enhancement, and I only caught four of them. And so now there is a defect in the code or inconsistency in the code, and it takes time to discover and resolve, right? So those enhancements take longer if it's poorly factored. If it's well factored, and I find that, "oh, I've implemented this twice before. This is the third time I'm going to refactor all of these these things into a single concept and give it a solid name and then reuse this from a single location." Now, later when I come back and I'm like, "Hey, I need to make an enhancement. Oh, great. Here's the place to make the enhancement. All the tests exist. I can, you know, I can add my tests for the new feature, the new functionality that I want. And then I can run all of the tests for the test suite and look, they all pass and look, my code is good." That, you know, having well-factored, well-tested code seems like it's going to cost you more, and it may cost more in the design or development phase, but it costs you less in maintenance and enhancement, which is where you spend most of your time.

[00:32:15] Allan Stewart: Right. Martin Fowler has a article that I really like. It's But I also like thinking about software a little bit like cooking. You're going to get in and you're going to make a mess when you cook. And so you've got to do it well. You've got to clean up as you go, because otherwise the craft gets overwhelming and now you can't actually do the things that you want to do because you've got the big dishwashing project that you have to do before you can cook the next meal.

[00:32:53] Dave Adsit: Yeah. Yeah. Yeah. That's, that's definitely a clean as you go makes for a much more pleasant environment overall, makes more, a more habitable code base and a more pleasant kitchen. So the final phase that we have on our list for software products is retirement or end of life. And that should just be free, except that much like every other phase, it requires people to who do things. And while people are doing things, costs are being incurred, right? Turning off your servers, shutting down the service. Even if you've already managed to get all of your users off, which is not a guarantee, doing the end of life tasks so that you can stop paying for the software requires work and effort. And all of that comes at a cost. And it may be a small cost. It may be short in comparison to the overall value that you've accrued from the software, but it can't be completely ignored.

[00:33:48] Allan Stewart: And I think it does get ignored sometimes because we often think about it in terms of the end of life of an entire product, or it's like, okay, this, this whole thing is done now. And, and, and even when that happens, it's not, it's not entirely true, right? It was, we'll just ask Microsoft about their Windows 10 end of life and how that's going for them, or maybe their IE six end of life. But I think it's a lot healthier to think about it in in terms of a continuous retirement, not of the whole thing, but of pieces. And saying, hey, as our product grows, as it evolves, what do we still need? Or maybe it's a little bit like pruning your tree. If you want to shape it and have it look really nice and be healthy, well, sometimes you have to cut some stuff off. And software products are no exception to that. And if you account for that, you can pay that cost to retire something, which then no longer produces maintenance costs, and it makes it cheaper to continue to build your new stuff. So I think that that is an important aspect that you should not forget about when you're wondering, how much is this going to cost me?

[00:35:07] Dave Adsit: Well, and to that point of continuous retirement of components of your system, I have a very relevant and timely example from my own team. One of my engineers came to me and said, "hey, it looks like we're paying several hundred dollars a month for this memory database, we disabled the last feature that uses it about six months ago, but it's still running. Should I turn it off?" And the answer is yes, please turn off that unused service inside of our cloud. Also, we skipped the retirement phase, or we took shortcuts where we said, "ah, maybe something else uses it. Let's just leave it up. This may be the last feature that uses that memory database, and it may not. So let's just not turn it off yet. We'll remember to check later if it's been used." And we didn't remember to check later. And so we've been paying ongoing maintenance costs because we took shortcuts during our retirement phase for a piece of functionality in our system. And sometimes you catch those and sometimes you If you have a big enough and complex enough cloud bill, you may miss thousands or tens of thousands of dollars worth of wasted spend forever indefinitely because you took shortcuts in some of your cost management.

[00:36:35] Allan Stewart: So now we have an idea of how much things cost in terms of rate. And we think about these cost phases so that we're thinking about it holistically and not myopically on one particular aspect of, especially the development aspect. And so now we can start talking about how it costs, how much will a thing cost. Let's see where knowing those costs, where does that help us? I think you already mentioned one, build versus buy. Yeah, once you, once you have a good idea of how much a thing will cost, and, and oftentimes it just is ballpark, right? Like, oh, well, if we built it, we think it will at least take us a few, this many days per year, or this meant, you know, we'll probably have to do something with it once a month. So you, you get a, you get an idea and say, okay, well, is that more expensive or is buying the, is buying the thing more expensive? Um, but you don't always, you don't always know or there's not always a sas product or, or other, or, or non-sas product that you can buy it might not meet the needs, it might not, um, it might not be what, what is required so So the other one that you also mentioned, but let's talk about some more, is making bets. And what I like about this one is that we're moving it away from that fixed cost mentality. When you start thinking about software costs as a rate, then you can say, "well, how much am I willing to invest in something or towards something?" And I really like the bet concept because not all bets pay off.

[00:38:24] Dave Adsit: Yeah. Yeah. I like the concept of, I like Thinking in Bets because it introduces the concept of risk. Sometimes we think that we have solved the problem and we know exactly what's going to happen. But when we say, "well, how much are we willing to bet that we have the right answer?" Now, all of a sudden you're reintroducing the concept that this is actually not a sure thing. There is risk involved. But also, I think it gives us an opportunity, and we've talked about this many times, of flipping the script from a scope based software development mindset to a time based. I want to bet two weeks of work that we can solve this problem. And then if we work in a, in a clear, in a clean way, we can say, "Hey, we worked on this for two weeks. This is what we got done. Should we ship it?" Versus, I mean, I don't know how many times I've experienced this, but you get to the end of a thing, and a stakeholder comes in and says, "I only thought this would take a couple of days. I thought it was simple. If I had known it would take over a month, we never would have done it." And then everybody feels demoralized because they thought they were doing great work that was benefiting the company. But the stakeholder had information about the value of the work product that they were working on that was not communicated effectively. Like, "I I thought that I was betting $10,000 on this feature and it was going to have a positive ROI. But when I ended up paying $80,000 for this feature, now the ROI is guaranteed to be negative." Right? So yeah, I don't even know how many times I've heard that. Every time it's demoralizing to both leadership and to the engineers on the team, because everybody wants to be building value and creating value for the organization and the customers.

[00:40:12] Allan Stewart: It definitely takes a mind shift, a mindset shift, because it's so easy to think about, "oh, well, these are the features that we want." And we start enumerating them, and here are the different cases. And pretty soon we've built up a fixed scope. And as soon as we've fixed the scope, we don't know how long it's going to take. Except for that we do know it's going to be longer than that. Yeah, whatever your optimistic estimate is, it's wrong. Well, even your pessimistic estimate is most likely wrong as well. And so it's always going to be more, it's always going to be nebulous. But timeboxing is very, like, you get to the end of the timebox, and then you look and you say, "Okay, well, what have we accomplished?" And and I think one of the mistakes I've made in the past is that I have fallen into the trap of well, let's just take this scope and try to fit it into a time box. How much of the scope can we get done? And that it doesn't really work very well because it's like, "oh, well, we only got a small piece of it. Okay, well, we don't know if it's going to work or not still, so we still we're going to to keep adding more and more time boxes that are completely arbitrary," trying to get to a certain scope. Instead, you have to change the thinking process about it to say, "hey, here is an outcome that we want. We're willing to spend two weeks. We're willing to spend four weeks. At the end of the four weeks, how has that outcome changed because of what we did during that time?" And good. Let's, let's do more of it. Or you say, "no, that's good enough." Or you say, you know what, that failed and we're not going to throw bad money after good. So we're going to change to something

[00:42:01] Dave Adsit: else. Right. Well, and, and so from my perspective, this is where the real engineering work happens in software development, because engineering is all about dealing with your constraints and the the constraint of, we are only willing to invest this much time, which equates to this much money to build this feature. Okay. Well, great. We can't build, we know we can't build the whole thing. What can we build of the superset that we want? Is there something valuable in there that we can build? What's the most important thing that we could deliver? If we could, you know, we often talk about how 20% of the features create 80% of the value. Well, let's find those 20% of the the features and build them first. And then if we still have time, we can continue working on the other 80% of the features and trying to capture that other 20% of the value. But if we only get the, if we only get this one feature or this one feature, these two or three related features, they're the highest priority things, are the most valuable, the most customers we use, they'll bring in the most revenue. And we only get those. Sometimes it was still a good bet, even though we did not get the entire scope that we wanted. And sometimes when somebody says, "hey, the bet I'm willing to make is two weeks," you look at the scope and you say, "we can't get anything of value done in two weeks." And so now you have a choice. Do we want to reevaluate the value? Do we want to just skip this thing entirely and work on something that's going to create more value or something that will create value within the bet that we are willing to make?

[00:43:39] Allan Stewart: Or are you willing to make a bigger bet?

[00:43:41] Dave Adsit: Yeah. Right.

[00:43:42] Allan Stewart: A risk, a much riskier bet with the hope of a payoff.

[00:43:47] Dave Adsit: Yeah. So there's a lot to, a lot to think about. We could probably talk for an entire hour just about making bets in software and which ones we think are valuable and which ones are not, but let's move on for now and talk about other things that are expensive that we do in engineering, like status meetings.

[00:44:07] Allan Stewart: Yeah. Yeah, status meetings. So meetings in general can get expensive, right? Because we need to talk. We need to communicate. We need to be able to talk through ideas. We need to be able to get alignment on a vision of what we're trying to do. Because very few software products that are large and successful are done by just one person who can have it all in their head. So you're going to need the meetings, it's just they're expensive, right? As soon as you, as soon as you put one person into the meeting, then you're taking that portion of their, um, their burden cost and you're applying it to the communication overhead of working with, with people, right? Um, and so, so some kinds of meetings, I think, are, are worthwhile, right? Like training meetings, those are an investment. We're We're going to invest some money now to improve ourselves so that we can be more efficient, so we can be better educated, better able to deliver features later. Right. So that's kind of, it's like the sharpen your saw kind of concept. Right. We keep, keep things working well, but status meetings are basically entirely pointless. It's, it's, it's, it's just throwing money away because status reporting ought not to be done

[00:45:34] Dave Adsit: in a meeting. Yeah, one of our friends often says, "status is not a meeting. Status is something that should be visible." You know, we want to make our work visible. We want to share what's what the status is to anyone who wants to know it at any time without having to meet. If you think about Now, if you say, "hey, it's $20,000 a week to run my development team and we work for 40 hours a week," that is $500 an hour for a meeting if you've got everybody there. Is updating one person on the status of a product or project or feature enhancement or whatever, is it worth $500? Or is there some other way that you can communicate status? And the truth is there are definitely many other ways you can communicate status. My teams all use virtual Kanban boards. They use virtual Kanban boards because we are distributed across many geographies. If we were in person, I would still prefer to just use a physical Kanban board on the wall of the dev room. But a Kanban board tells you at a glance what is completed, what's in progress, and what's up next. That is generally enough status. If we need more status than that, we can look at things like higher level epics and things like that. But you you don't need to call a meeting to communicate status. And if you do need to call a meeting to communicate status, you're probably doing something wrong. You may need meetings for things like alignment and planning. Those are critical aspects of delivery. I think it's critical for my teams to do a retrospective at least once, you know, once a week. And that allows us to improve on our work and continue to work in a better and better way. You know, And I think it's worth investing in improving my team's mechanisms for work. The same is true for training. Training is invaluable. You can often get well more than you invest out of a training meeting if you do something, if you do a training that is on point with what the team is doing.

[00:47:32] Allan Stewart: Yeah. And even if you don't do the work to make status visible, which I think is the ideal, deal, right? So somebody can just come by and they can see the status or they know where to look. But even then, like, just asking the question, right? Sending a Slack message, sending an email, sending, you know, asking one person rather than calling a meeting to find out the status of all the people, and everybody stands around and is mostly just waiting for their turn to say that they're still working on the thing. Yep. At least you can, at least you can do it on the a cheat, but say, "Hey, when you get a minute, tell me how this is going." Yeah. And meanwhile, I'll do something else. You can do it at your leisure. And then we find out. And that's still,

[00:48:17] Dave Adsit: that's better than calling the meeting. Yeah. What's the meme? "This meeting could have been an email." The status request could have been an email, or better yet, a Slack message. Yep. So you and I are both fans of lean software development, lean methodologies. Lean talks talks a lot about waste and eliminating waste. And there are seven types of waste in Lean, and they each apply to software development in one way or another. And the wastes are inventory, overproduction, extra processing, transportation, waiting, motion, and defects. So let's go through each of those in order and see how those map to software processes. The first one, the waste of

[00:48:59] Allan Stewart: inventory maps to partially done work or work in process. So basically, while you are building the feature, it's not worth anything until you can ship it. And so the longer that it sits around partially done, the worse it is. Now, I think while you're working on it, if you're actively working on it, well, that's a cost you have to pay. The danger is that we have this thing that's partially done, and we set it aside. It's not delivering value, but it exists as inventory. And so software inventory is often, well, that code is there. And if we make changes, the compiler is going to complain unless we change that code too, or we have to spend costs to get back up to speed on it when we want to think about it again later, when we're done having it in process. So that's how I think about the inventory a lot of times. But I also think about it in terms of design inventory. Oftentimes, because it's much easier to imagine what the product could be like than it is to actually build it, it's very easy for product managers and UX designers to go off and build out years worth of work that you may never actually get to.

[00:50:25] Dave Adsit: Yeah. And one of the things I think is that, you know, in manufacturing, you can't do manufacturing without some amount of inventory. So it's actually excessive inventory that becomes waste. Right. But you could consider all inventory to be waste until it's actually turned into usable product and sold. Because even a completed product is still inventory until you you have exchanged it for currency. So in software, the thing that I always say is it is better to have one feature completely done, 100% done and delivered than it is to have five features, 80% done. One feature that's 100% done has a positive ROI, it has the potential to return on the investment and five features that are 80% done do not. This is why you get these concepts of "it's done, but it's not done, done," right? Like, I did all the coding, but I haven't tested it. I haven't deployed it. I haven't released it, whatever. If it's not in the hands of customers, it's not, it is not capable of returning value. And so all of that is inventory and needs to be managed very carefully. This is why it's so critical that we limit our work and process and focus on delivering completed work in order to minimize that waste. So the second waste is the waste of overproduction. In manufacturing systems, we have one system that runs really quickly, and so it creates a whole bunch of intermediate parts. It's similar to the waste of inventory, but overproduction, we have stuff that we can't sell, whatever. And that maps to delivering extra or unneeded features. Sometimes gold plating is what you would call it, where I spend a bunch of time working on unnecessary features. So if I have a continuous delivery system and I say, "oh, well, sometimes we want to change the text on this button." Okay, well, I can just change the text on that button and then ship it. Or I can build a system that allows someone else to change the text of that button by a UI that writes to a database that's read from the database and cached and this and that and the other. That is, all of that can be considered waste because it was unnecessary to provide a way for someone else to change the thing without talking to a developer if we could just update the text and release the code anytime we wanted.

[00:53:00] Allan Stewart: Yeah. I don't think I have anything to add to that one. So let's move on to extra processing. The waste of extra processing in manufacturing would be when you take longer or you're spending more resource than was really necessary in order to create a thing. And that maps in software to relearning because our creation is very thought based, right? It's knowledge work. It's not it's really not about your keyboard typing speed. It's about how you're applying what you know into code, and any time that you have to go and relearn something because you're, you're revisiting some piece of code that you haven't, haven't touched in a while and there were bugs in it, or because it got set aside or whatever, these reasons are where you have to say, "oh, well, I don't know. Let me go and relearn this." That's waste.

[00:53:57] Dave Adsit: Right. And the waste of transportation in manufacturing, and it costs money to move physical goods around. And so if you can arrange your system to minimize transportation, you can save, you can reduce waste. You can save money, whatever, save money, time, whatever it takes. In engineering, that maps to, in software development, it maps to handoffs. If I, if the PM does all the discovery work with no engineer present and then hands off the completed specs to the engineer who then has to understand it and then build something thing, and then hands it off to the QA team who has to go back and understand all the specs and understand the software and validate it. All of these handoffs are waste. If we instead arrange our teams in a way that we are working collaboratively, we can reduce the cost and frequency of handoffs. We just eliminate those handoffs entirely because everyone knows what we're doing the whole time.

[00:54:55] Allan Stewart: The waste of waiting maps to delays. And I think this one is pretty simple. This is the situation where the team is working on something and all of a sudden a question comes up, right? There's a technicality or there is something about the user interface maybe that, like, "oh, well, did they want this? Did they want that? We don't know." We have a question and the person who can answer the question is not there. And so we have to delay. We wait and we wonder, "what are we supposed to do about this?" And that is waste.

[00:55:32] Dave Adsit: Well, and in my experience, that waste often turns into other kinds of waste. That one is a catalyst for creating the waste of partially done work and context switching and handoffs and all kinds of others. So be very, very aware of the waste associated with delays. The next is the waste of motion, which maps to context switching, which context switching. One of my favorite things, one of the conferences I went to a while back, they shared some research that when we context switch, each additional project that we pick up causes a 20% cognitive overhead load. And some smart Alec in the audience, probably me, asked, "does that mean if I'm working on five projects, I'm getting nothing done?" And the research suggests that yes, if you are working on five different projects, you are probably making zero progress on any of them because context switching becomes very, very expensive when we are trying to work on multiple products projects or multiple projects simultaneously.

[00:56:38] Allan Stewart: And it feeds into the relearning waste that we talked about too, and the delays, right? Because if you have to switch contexts, then you have to pick up what you were doing. And sometimes it's just, "okay, I'm switching contexts. I've got a rough idea of what it is," but other times it's, "well, I didn't write it down very well. Like, maybe I wrote my code really poorly, and this variable is just called X, and I I don't know what it is for." Well, now I'm relearning because of the context switching. And so they, yeah, they like to gang up on each other. Yeah, for sure. And then the final one, defects are our defects. So manufacturing defects are one kind of thing, and we call them software bugs. And ultimately that is, there are a lot of reasons for bugs, but I think more often than not, most of the bugs that I have seen is just that something wasn't thought through very well. It wasn't crafted. It was just kind of thrown out there. "Hey, I got it working once, and so it must be done." And it was never refactored. It was never cleaned up. The naming was never changed. And so you end up with a problem. And now you have to go and fix the problem that you created for yourself.

[00:57:58] Dave Adsit: Yeah. And I have strong opinions about software defects. I feel like a lot of us don't work in an environment where we could actually know if we have defects because we don't write robust enough specs for the software. And so a lot of times what gets labeled a defect is really previously undefined behavior that was only, only became defined when a user tried to do something and we realized what the software did. But regardless, at the end of the day, you're going to have to make a change. If the software doesn't behave as, as desired, you're going to have to make a change. So in addition to waste, the seven types of waste that we've just covered, there's also the concept of secondary work, which is one of the most sophisticated kinds of waste that we have in a software system, because often we don't think that secondary work even is waste. And so to provide an example, secondary work is any work we have to do because we failed to meet the primary need. So you need a feature. I haven't given you a feature. Now we have a status meeting. That status meeting is secondary work. It was completely unnecessary if I had just given you a completed feature that you could use. And we see that happen throughout the system in software, where we build sophisticated processes to handle all kinds of secondary work.

[00:59:20] Allan Stewart: Yeah, my favorite example of this is the backlog grooming meeting. Because we get lots of ideas for software, and we say, "yes, we should do these things." All of these, you know, these ones are bugs that somebody has reported. So obviously we've got to fix all the bugs. Plus there's these features that we want to build. There's that really good idea that we had. Oh, and also there's these technical things that we need to do. And we're going to just write them all down in a big old list. So we don't forget. So we don't forget because you wouldn't want to forget anything. Right. You've got to keep that mental load perpetually. So you write it down somewhere, and I guess you're offloading the mental load onto a paper so that you can load that mental load up again. So that every two weeks or month or whenever you do your backlog grooming, you can try to remember what all of these things were. Try to guess what their relative priorities are to each other and, and make sure that, hey, yes, we we have groomed this backlog, because we don't want to throw anything away, and we're going to create this list, this ordered list. And anything that's past, it's like the, past the page fold doesn't matter, right? It's like I don't, I don't know how many it is. It's going to be maybe it's small, it's probably like only 10, maybe five. Yeah. Five. And anything after that, if you spend any time on it whatsoever, even to, to look through it and see, "Hey, are any of these, should any of these get lifted up?" And then you've, you've just wasted time. Yeah. Because all those things at the bottom, you're not going to work on, you're not going to get to it.

[01:00:57] Dave Adsit: Backlog grooming seems like an important part of the process, right? But backlog grooming only exists because a backlog exists, which only exists because we failed to deliver on the needed software. And so it's, you know, it's a secondary effect, a secondary work of a secondary work. And there are ways to completely eliminate that need, right? We just don't maintain a backlog. Every, if each time a new idea comes up, we say, "is it more important than the ones on the list now? Now, if yes, one of them gets bumped and it gets added." And if not, if we, if we have a WIP or a, a cap on the number of items we'll keep on our backlog or to-do list, we can completely eliminate all of that. And the thing is, you're never going to forget the good ideas. You're probably going to forget some of the, like, mediocre or bad ideas, but they're, they were mediocre or bad to begin with. If it was a really good idea that you really, really needed in your software, you would have moved it to the top of the list and bumped something else off. Let's see, what other kinds of secondary work are important to touch on? A perpetual favorite of our teams are estimation. How many times have we said just in this podcast that we're bad at estimating? Probably a lot. Just in this one conversation, we've mentioned being bad at estimating multiple times. Software developers are bad at estimating. First of all, they're all way too optimistic. And when they try to be realistic, they get pressure to be optimistic instead, which doesn't actually make the software easier or faster to develop. It just makes the estimates less accurate. And none of us has ever really gotten good at estimating well, because we're always doing something new.

[01:02:39] Allan Stewart: And I think that to get better at estimating is not necessarily worth the cost, because it takes a lot of time and effort to refine and improve your estimation. And oftentimes you have to actually start doing the thing to know what it was going to take, right? You can only know it in hindsight. And so doubling down on, "well, we're just going to do better and better at our estimation process," I feel like is just a losing battle, right? It's diminishing returns.

[01:03:12] Dave Adsit: turns. Yeah. Well, and so I've, I've heard that there are some doctors that are very good at diagnostics, you know, diagnosing a patient. And there's some doctors that are not very good at diagnosing a patient. And the difference is the ones that go follow up after the fact to find out if they were right. And if we want to be, if we wanted to be good at estimating, we would have to to spend extra time reviewing our estimates after the thing was finished to see how much we were off and by what and why. And then we'd have to take all of those things into account. And we would have to be working on small enough projects that we could reason about the whole project while we were doing the estimating. And often that's not the case either. And so these things become very challenging. And I think you're right. In most cases, it's not worth the cost to improve at estimating. So can I talk about one of my favorite kinds of unnecessary secondary work? Absolutely. CapEx. Capital expenditure. This is definitely a quirk of the US tax code and will be a project driven by your finance team. They want you to determine what work is is capital expenditure, which is building new features versus maintenance, which includes things like bugs and operations and whatever. And why do they want to do it? Because you can do fun things in accounting if you know what your capital expenditures are. However, this is a huge waste of time for a few reasons. One of which is that software bugs are virtually impossible to define without a spec. And the reason I say that is because you don't know if it's a defect, if nobody ever wrote down what it was supposed to do in the first place. And so all you ever know is, we have software that works this way, and we want it to work that way. And we have to do work to get from here to there. Is that a feature? Is that a bug? I don't actually care. It's work that has to be done to move from A to B. We want to be at B, we're at A, we've got to do work. Okay. I would categorize, personally, I would categorize every single thing we do in software development as a new feature because none of the teams I work with have the super robust written specifications like you would have in an engineering environment.

[01:05:37] Allan Stewart: And it might be cheaper to do it that way too, right? Like the tax write-off that you're going get all of your people to do this extra work. And from what I've heard, this could be wrong, but from what I've heard is that nobody wants to just say, "oh, well, let's just make it a flat rate. And we'll say, oh, well, it's a 50% or a 30% or a 70%." Because as soon as you tell the IRS that, oh, well, I'm writing off this money, then you run the risk of being audited. And if you can't say why this was capital expense, which goes back to what you were saying in the first place. How do you tell them that this was an operational expense versus a new feature development? It's almost impossible to say.

[01:06:30] Dave Adsit: Well, and the good news is there are actually certain heuristics that you can use that will just allow you to skate by any auditor. They'll just accept it. And my understanding is it's 80-20. You say, "80% of our work is building new features and 20% is addressing defects or doing operational work." And nobody will question that. And if you want to get more aggressive than that and say it's 90-10, then you better be able to back it up. And that means that you're tracking every piece of work that you do has to be tracked back to a ticket that was approved by someone who had the approval ability to make software change, blah, blah, blah, blah. It's a bunch of work. It's a bunch of work that if you do it well, it falls almost exclusively to leadership and not to the engineers on the team. And then the engineers can stay focused on their value add, which is building new features and enhancing the system at hand. But yeah, to me, the whole thing is just a very fancy form of secondary work, aka waste.

[01:07:32] Allan Stewart: So software costs, like we said at the beginning, it comes up a lot. When people are thinking about starting something new, when people are wondering what the new feature is going to be or the new app that they want to build, anytime that you're already developing a product and you want to know what the next feature is going to cost, all these things are costs, right? Like, eventually you're taking, you're paying your developers to do some software work, and you want to know how much is this going to cost? So hopefully some of the things that we've talked about today will equip you to better better have that discussion. I know that they have improved my ability to discuss and make rational arguments and decisions with leadership in the places that I've worked so that you can say, "hey, this is the rate. We can start thinking about things in perhaps a different way. Instead of scope-based, we can think about it more time-boxed approach." Maybe you can think Think about places where you can address some low hanging fruit, like not spending money on services that you're not using or eliminating some waste within your product or a status meeting that isn't helping you out. But I hope that something that we've said here is useful as you're considering what it costs and what it's going to take to build software.

[01:08:55] Dave Adsit: Well, and to paraphrase Don Norman in Principles of Product Development Flow, if you want to have an effective conversation where you influence a manager or business leader, speak to them in terms of money. Reduce it to money and you can have a much greater impact on decisions. And that's what this is all about.