Crafting Code Podcast

[00:00:15] Allan Stewart: Welcome to the Crafting Code Podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately I've been thinking about the difficulty of trying to coach and code at the same time.

[00:00:31] Dave Adsit: And I'm Dave Adsit, a VP of engineering, and recently I've been thinking about immutability, append-only data structures, invariance, and using coding techniques to enforce system

[00:00:42] Allan Stewart: integrity. Our topic for this episode is lean software development. Lean has been something that I've heard about a lot over the years. It seems like maybe 10 or so years ago, it was very... Very popular. A lot of people were talking about it. And so we're going to discuss what it is and why we might care about it in the software development space.

[00:01:06] Dave Adsit: Yeah. I think this is one of the things that's a driver of value in the organizations I work in. It's one of the things I care a lot about. You would have found me 15, 20 years ago around an Agile roundtable or two talking about Agile concepts. I feel like lean is the next evolution towards... Getting what we hoped to get out of the Agile movement that was super popular for quite a while. So as we get started talking about lean, there's a few things that need to be defined because they're critical to understanding the concepts. And the first one is efficiency, which is strictly just the ability to do something or produce something without wasting materials, time, or energy. It's just a simple definition, basically being able to get something done without creating waste or without waste as part of the process. So that's efficiency. And when we talk about lean, we're talking about efficiency and we're talking about delivering value. So at its very simplest, the definition of lean is to... It requires us to contrast two types of efficiency. So the first one is resource efficiency and the second is flow efficiency. So resource efficiency is a measurement of the amount of time a machine or worker is working per unit of elapsed time or wall time. For example, Bob works 38 hours a week, which is really Bob is working 38 out of the 168 hours in a week. Which means Bob is only 22.6% efficient as a worker. And we are going to take note of that for his next performance review. Right. So that, that is a definition of resource efficiency. You could also say he works 38 out of 40 hours, whatever. I think the point stands.

[00:03:16] Allan Stewart: So in contrast, flow efficiency is a measurement of the amount of time a job or a work unit, like the stuff that needs to get done is being worked on. Uh, per unit of wall time. So an example there is we can start a story card on Monday and by Friday, uh, between all the meetings and working on other tasks. If we only logged five hours of work on that story, well, then we were only five out of 40 hours. So 12.5% flow efficient at delivering that card. Right.

[00:03:53] Dave Adsit: And so you can see the, these two types of. Of efficiency are they're measuring related, but different concepts. Um, one of the things, one of the examples or ways of describing it that comes from one of the books that I really like, which is, um, this is lean. The, they describe the focus of the camera in the first example with Bob, we had the camera pointed at the worker to see if he was working. And in the second example of the story card, we had pointed the camera. At the work item to see how much it was moving or being worked on or whatever during the elapsed time. Right.

[00:04:34] Allan Stewart: And that can be very, very important way of shifting your focus because there are some ways that we can look at this differently, right? So for resource efficiency, we look at Bob and his work hours. Well, if we're looking at him over the, the full 168 hours, well, it doesn't look very good on a daily. We expect him to work 40 hours in a week. Well, now he's a much higher percentage, but probably once you start doing that, it's unlikely that that's going to change one way or the other. Right? So if you're always going to measure him over the total number of hours or over the, you know, the work hours in the week, probably he's there working, but it doesn't tell you anything about. What is getting done and whether any output or outcome. are being achieved. But with our story card example, where we're focused on the work, okay, well, it took five hours out of a 40-hour week to get it done. But we weren't really measuring the 40-hour week. That just happened to be that you started it on Monday and you finished it on Friday. If you started it on Monday and you didn't get distracted and you put those five hours in all on Monday, maybe you had lunch, maybe you did something else. So at the end of the day, Monday, it gets done. Well, then you have five divided by eight hours, let's say, and your efficiency goes up significantly as you're looking at it. And so the numbers, the raw numbers and the percents don't really mean anything by themselves. You can game those by changing the measurement, but it is telling us something important. About how long does it take to get value, which you just don't get out of resource efficiency measurements.

[00:06:29] Dave Adsit: Right. And one of the things that I've found is that it's easy to measure whether or not somebody is showing up to work, whether they're there doing their job, right? So we can say, oh, Bob, he's not a very good resource. He's not here as early as I am and he leaves before I do and all that stuff. And so it's easy to measure, easy to assign blame, but really, maybe Bob is super effective during the time when he's working. And, you know, being effective of course is similar to efficient, but effective is just the ability to get anything done. And so the whole point of lean is that we are trying to, in the choice between being resource efficient and flow efficient, we are prioritizing work on increasing flow efficiency. We want to flow value through the system and reduce the waste. And the time spent waiting is considered waste. So. I mean, in the book, This is Lean, one of the things that they talk about is the resource efficiency paradox. They talk about resolving the efficiency paradox, which is really interesting because when we start measuring Bob and tracking Bob and watching what Bob's doing, we're going to start doing a lot of extra activities to make sure that Bob is showing up his 40 hours and he's in his chair and he's not being distracted. And we've, you know, blocked access to YouTube and for good measure, the rest of the internet, because no programmer ever needed the internet to do their job. And so the efficiency paradox is basically that when we focus on utilizing resources efficiently, we tend to increase the amount of work there is to do without adding value. Some examples of that are if the work, the value that we want to deliver is working software. Well, Bob hasn't delivered working software today, so we're going to have a status meeting and find out why not. I don't, I don't think any status meeting has ever gotten us closer to delivering value. But the good news is that what we found out in the status meeting is that he hasn't even started some of the work we put in his backlog. And so now we're going to have a backlog grooming meeting where we prioritize the items in the backlog. So we make sure he starts the right one and remove some of the ones that are, you know, not relevant anymore, or we missed the market window or something. Right. And then we're going to have you. So you can see you, we've all experienced that when we fail to deliver the primary good, which is working software or solutions to business problems or whatever, right? The, when we fail to deliver on that primary value, we start doing extra things to track or to manage the work to make sure that it's the right stuff. And so we tend to. Yeah. Increase the amount of work done without increasing value, right? Nobody cares how many status meetings you had. None of your customers asked you if you did all of the scrum ceremonies perfectly. No one is ever looking at that kind of thing. Once the software is out the door. But those are all efficiency killers. And so in a lean system, we want to focus on how do we maximize the flow of the value? Units or the work units, you know, how do we take problems and ship solutions in the most efficient way?

[00:10:00] Allan Stewart: Right. And so, yeah, the flow units are those work. I usually think about it in terms of cards on the Kanban board that I use.

[00:10:10] Dave Adsit: Right.

[00:10:11] Allan Stewart: It's not a coincidence that people who think about lean also think about Kanban. But I also like to think about it in terms of like a value stream map. And so if I think about any. It's like, oh, well, when I'm doing the coding work, value is high, right? Like stuff is actually getting done. But then I submit my pull request and I have to wait for an hour before somebody looks at it. And then there's nothing getting done. And so we're low. And then it goes high again as somebody reviews the pull request. And then it goes low again when you're doing coding. Or when they've, excuse me, it goes low again once they have submitted it. And then eventually you look at it again and you say, oh, okay, now I can start coding again. And value goes high again. And then if you have to pass it over to a testing group, then the value is low while you're waiting for them to be available to test. And so you want to keep that as high as long as you can and get rid of the waste in between, right? So thinking about these two different kinds of efficiencies, another example that I like to think about is a printer, right? So you buy a new printer. And you want to have a really good return on investment in your printer. And so most people would start looking at resource efficiency and say, oh, well, how often is it printing? If pages are constantly flying off the printer, then it must be well utilized, right? It's got really high resource efficiency. But if it just sits there idle all day, well, then why do we bother? But the flow efficiency tells you why people really buy printers. It's that when they want to print something. They want to click print and it just pops out, right? If it takes 10 seconds to print the document, but I had to wait for 10 minutes because there was some other work in the print queue, then I'm not particularly happy, right? That's not very good flow efficiency. Because I want to say print now, and I don't want to wait 10 minutes. I want to only wait the 10 seconds that it's required. So for me, that's one way that I think kind of keep these two. Two concepts straight, right? Resource efficiency. We're just printing documents all day or it's sitting idle. Flow efficiency is, hey, from the time when I click print, how long does it take before I actually have the result in my hand?

[00:12:57] Dave Adsit: Yeah, it's all about looking at either the machines doing the work or the work to be done. And given that our goal as software developers working in businesses is to create value, deliver value. What we want to do is. Continuously flow that value through the system. And so when you start talking about lean, there's some things that come up that are very important. You immediately get an introduction to queuing theory. Right. Because it turns out that everything, everything in lean is queuing. You are either in queue or you are being processed by the queue processor, or you're, you know, queued up to queue up to queue up and somewhere other spot, right? It's all queues all the time. And, and so you have to learn a little bit about queuing theory. And. And some of the things that are in there. There's, there's basically three laws that get talked about regularly. The first one is little's law, which is that increasing cycle time, how long it takes to get something done or increasing the number of concurrent flow units will increase the throughput time. So how do you think about that? Basically think about that as people in line. If you think, hey, if I increase the number of people in line ahead of me. I have to wait longer. So throughput time is the time it takes me to get from the back of the queue through the process. So if there's more people in front of me, I have to wait longer. It's pretty obvious.

[00:14:28] Allan Stewart: Right. Right. And you also have to wait longer if it takes long time to complete the task. So when the, uh, the person at the grocery store is having struggles with their payment and you're waiting and waiting, it takes longer. Even though the average time might only be a few seconds to, to tap their card.

[00:14:50] Dave Adsit: Right. So if the, if the, if the job processor is slow, it takes longer to get through the queue. If the number of items ahead of you is slow or the average number of items in the queue is slow. It takes a long time to get through the queue. Both of these are, are straightforward and apparent. Um, but they're very critical when understanding why things aren't flowing well in your system. And so that is. Little's law. Uh, the second is the law of bottlenecks, which is easy. The law of bottlenecks is that there is always a bottleneck and a bottleneck will increase the throughput time. So imagine an airport. That's an easy one. The security checkpoint is a bottleneck. You know, it's a bottleneck because there's a queue formed in front of it. There's a queue of people waiting. That is, that is your number one indicator that there's a bottleneck is that there are a bunch of things waiting. To get through it. If there was no bottleneck, nothing would be waiting. So we have a bottleneck and when there's a bottleneck, it takes time to get through because of queue forms. And now you're waiting in queue. And so little's law applies and the longer the queue, the longer the throughput time, et cetera, et cetera. Yep.

[00:16:02] Allan Stewart: And then the third law is the law of variation. And it says that variation increases throughput time and the effect increases as the process gets closer to. 100% utilization. So essentially there's all of these little things that cause differences in the world, right? Uh, natural or human causes that affect that affects things. So it's not, it's not always exactly the same, right? Uh, I can imagine a system in which the highway runs and everything is exactly the same. And you can imagine the cars are just exactly going on, except for that eventually some of them. Yeah. Want to get off and other ones want to get on. And sometimes when you want to get on, there's nobody in your way and you just get right on other times. There's people in your way and now you have to slow down a little bit and, and they have to make room and there's these adjustments and not everybody goes exactly the same speed. And sometimes there's a hill and not all the cars can climb the hill and maintain a smooth speed. All of these things are variation. And as soon as you have variation. It increases your throughput time. Now, if you've got that car that's struggling to go up the hill, but there's two lanes and you can just switch lanes and go around them. Well, then that's not, not a big deal, but if there's only one lane, well, now you're stuck behind them.

[00:17:27] Dave Adsit: When I think about King, uh, law, the law of variation, also known as Kingland's formulation, basically it's the idea that variation increases report time. One of the things that comes immediately to mind is riding the bus in a developing nation. Yeah. If you ride the bus in a developed nation, mostly it's low variation. It's people getting on and off the bus. If you ride the bus in a developing nation, it's, it's people getting on and off the bus, people with children getting on and off the bus, small family units, right? It's also people carrying all their produce for their small shop. Um, they are either bringing to market or taking back to a shop to sell. It's sometimes different. It's people trying to bring livestock onto the bus. I I've experienced chickens and pigs. Uh, the chickens were allowed to stay. The pigs were not, um, all of these things, all of these variations increase the amount of time it takes to load the bus and unload the bus. And so the bus waits and it waits and that increases your throughput that increases the time it takes you to get anywhere of value. Um, it's actually one of the things that led to the creation. Of the shipping container. Uh, apparently in the battle days of shipping, they just had a boat with a big hole in the middle of it and people would manually carry everything on and put it on the boat. And it would take up to a month to load a boat before you could sail somewhere and up to a month to unload the boat before it could sail back. And, um, with the advent of the shipping container, all of the variation in the system went away and now shipping containers are huge. Uh, uh, and they're loaded by robots because everything's the same. There's no variation. It doesn't matter how long it takes you to load your shipping container. Once your shipping container is on the lot, it gets loaded onto the boat and it's easy, easy, smooth sailing. So, so law variation, basically very, uh, variation increases throughput time. And the closer you get to fall, the worse it gets when you try to load a crate of chickens onto a bus. It's already got 85 people onto it. You're going to have a bad time.

[00:19:46] Allan Stewart: I think most programmers are familiar with this concept when we think about CPU usage, right? Like everybody knows everybody that I've met, they all know that we do not want to peg the CPU at 100% on the server, that that, that that would be bad. And this is basically why. So all of these three laws tell us something about how our throughput time, how long it takes. Can go up. Right. Um, and inversely how we can make it go down. So any of those things that cause a delay, well, it causes a negative effect because we don't get our value as quickly as we want. Right. And that in turn creates secondary needs, right? These additional needs, right? So like if, um, if things just happened instantly, you wouldn't need a status meeting because they say, Hey, could you do this? Oh yes, here you go. And they would know the status because it was done. It was completed right before their eyes. But unfortunately, most requests that I get for features are going to take quite a bit of time. And in fact, I can't even tell you how long it's going to take because we got to talk about what it even means first. Um, and so then we get these additional needs, right? Right. We want to do status updates. We want to do tracking. We want to do other things that are necessary now. And some of them, some of them are more necessary than others. Some of them are just not necessary. But. Um, you have this tendency to create these additional needs because it's taking a long time. Um, but then there's other things that happen, right? So like people are trained when they're working, don't sit and do nothing. You're supposed to be working. And so if they are blocked on some bit of work, what do they do? They start the next thing. Well, that increases your work in process, right? Little's law told us that if you have more concurrent flow units, more things in the. So like you, it's going to take longer. Well, there you go. So starting on something new might actually make it all take longer instead of making it shorter.

[00:21:51] Dave Adsit: Right. I mean, that's definitely the, uh, the case in most, most instances, because if I have. You know, card a, and I'm working on it and I get stopped because I'm waiting for someone and I start card B. Well, maybe somebody finished the thing on card a, but I'm in the middle of process, uh, middle of working on B. I'm not going to go back to a immediately. So. So a is sitting idle even longer. Um, and so yeah, increasing whip while it feels good from a resource efficiency perspective is often the wrong thing to do from a flow efficiency perspective. Um, one of the net effects is. We also, when we're, when things are waiting in queue, when they're not done, we incur a cost of delay. And we've talked a little bit about cost of delay in the past, but cost of delay is basically. What happens if we can't deliver this value by a certain date. Right. So you can measure things on a, like the, the, I think the full definition is like, how much money does the business lose by not having this done? And sometimes you have items that are low cost of delay. Like we can deliver it today, next week in six months. And whenever we deliver it, we'll achieve all of the same value. We'll get. All the same customers we would have gotten. We'll get all the same revenue that we would have gotten no matter when we deliver it. Often there are things that if we get them done before a certain date, we will see a larger impact on our business than if we get them done after that date. Right. And so maybe we missed the opportunity to serve a certain giant client that would be very valuable to our growth, or maybe we don't, we don't get something. between the two things that you're trying to work on. Right. I've, I've been told, I've read that there's the research that suggests that there is a 20% penalty in context switching costs for each additional project you pick up. And that suggests that there's a limit of five projects at which point you're not doing anything valuable ever. But, and I mean, I think some of us have had experiences that bear that out as well. All you do all day is go to status meetings and planning meetings and meetings about why the meetings aren't effective. And we need to have another meeting to figure out how to get some work done. When the reality is that the problem is too much context switching and too much work in process. And so I think that there are, there are small context switches, right? If, if we have one project that we're working on and there's three or four stories that roll up to one feature that we're going to deliver, the context, which there is very low, very small. But if we're working on several different unrelated projects, that's when we see this, this very high cost for context switching. So you can take one person and say, I'm going to assign you 50% to this project and 50% to that project. And you really only get 40 and 40, or more likely from my observations, you're going to get 60 and 20 because one project is going to be more interesting. One project is going to have more answer's already solved. One project is going to attract most of the work and the other one is going to get almost none of the work. And you still have all this wasted context switching cost.

[00:25:48] Allan Stewart: Well, that's interesting. It's almost like the projects aren't exactly the same, that there's something different about them, something variable about them. Indeed. Indeed. So all of this is in service of keeping flow high, right? So we see some negative effects if we have low flow or delays, right? And I think as soon as you start talking about it or thinking about it in terms of delay, well then of course that's not good, right? So how do you keep flow high? One of the ways is to limit your working process. Or whip, right? So a lot of Kanban boards have a feature that let you establish whip limits. And what that's all about is saying, hey, we are only going to allow this many things to be in progress at a time. And the reason is to keep flow high. Once you hit that whip limit, instead of starting new work, well, you should instead ask yourself, what can you be doing to help someone? And that's something else that's already been started so that it gets done instead of starting new things. One of the things that I like to think about with this is I've heard a lot of times people talk, when they talk about establishing whip limits for the team, they'll often think about it in terms of a multiplier. It's like, we've got four developers on our team. And so they'll say something like, well, we've got the four developers and sometimes work gets stuck. we want the developer to be able to work on something else. So let's set our whip limit to two times the number of developers, or maybe 2.5 to give us a little bit of slush room, or maybe three. And the higher you make that whip limit, the more opportunities you're giving for the thing to fill up and slow down the process overall. Even though it feels like you're saying, oh, no, no, it should be, it should be high. Personally, I like to, look at the multiplier as something less than one. So if I've got four members of the team, I would like to set a whip limit of maybe two. Now there can only be two things happening at once. And that changes the dynamics of the team. You start thinking to yourself, well, how are we going to do that? Well, maybe we're going to pair program and there's two things going on at the same time, or maybe we're going to mob or ensemble program. Everybody's working on one thing. And if it gets we'll hop onto one other thing while we wait for the product manager or the CEO or the marketing department or whoever you're having to wait on, we'll work on one other thing while we're waiting to get the answers to that. But we're not going to keep looking for more and more work. We're not going to divide it out with a multiple greater than one. Exactly. And one of the things about

[00:28:49] Dave Adsit: working in a highly collaborative style with pair programming or ensemble programming is that you are intentionally into the concept of high flow efficiency at the expense of resource efficiency. And I think that what I have found on my teams is generally resource efficiency doesn't go down that much. It's a false economy to have people all working separately on separate things and the waiting for coordination points to deliver value. The teams that I have that work in very collaborative ways with ensemble program mobs or ensembles or even if they're pairing, they tend to get each individual thing done much sooner and much higher quality because we have had a continuous review of that code, right? We're not going to get too much into the benefits of collaboration, but I agree with you that your WIP limit should be lower than the number of people on your team as a mechanism for encouraging collaboration. And as a mechanism for encouraging flow of value through the system, you may have one card, and this is like the lowest form of collaboration, perhaps, but you may have one card where you say, okay, we're going to deliver this one feature and the two of us are going to work on it together. And you're going to go work on the front end and I'm going to work on the back end. And that's not great collaboration. That is still more flow efficient than if we each picked up our working through the front end issues and the back end issues, then coming back and connecting the two and whatever, we're doing it together in parallel. And so we can finish this feature sooner by working together. And so that increases our flow efficiency.

[00:30:47] Allan Stewart: Right. I think if somebody, you know, if this idea is a little bit too radical, well, maybe try setting it to exactly the number of people on your team. And so. So most of the time people will work solo on their own thing and they'll get their thing done as soon as they can. But as soon as somebody gets stuck on their thing, whatever work you were doing that, that is now stuck and you're on pause. Well, see if you can help somebody else out with something else. And even that, even that is going to increase the flow efficiency for a lot of teams.

[00:31:18] Dave Adsit: For teams that I've worked with that really struggle with collaboration and really struggle with flow. What I have, I have often set their limit to the number of people that I've worked but the most effective thing you can do is take the thing that you need an answer to and walk over to the person who can answer it and sit there until they give you an answer. And now you've unblocked the flow of value. Because as long as you have the perception of resource efficiency or busyness, then people will let you just keep working. But if they realize that it's actually important to solve this problem, to answer this question so that we can continue to deliver value and actually get this thing done and deliver it within this sprint, or in many lean systems, you just do continuous delivery instead of waiting for the end of a sprint to ship things. But you can actually make visible the state of the work in a way that is very visceral and allows you to unblock work and flow value in a system that would otherwise start building up queues. If you let yourself If you let yourself block any number of things and just let them all build up in a queue, well, you'll never get anything done.

[00:32:53] Allan Stewart: So aside from WIP limits, another thing that I think we can do is look for bottlenecks. Identify and address what the bottlenecks are in your system. And sometimes that can be done by looking for parallel work, things that people can do that have low overlap. So you're not going to be as likely to have a merge conflict. So if you've got two developers who they're not collaborating, they're working on separate things, or even in your example, they are collaborating, but one person's doing front end and one person's doing back end, well, they're probably not going to get a merge conflict from each other because they're working in different places. Or if you're working in two different parts of the system, that is going to be more effective than trying to do two things at the same time that work in the same area of code. If I'm working on invoicing and you're working on user profiles, probably not going to overlap. But if we're both working on something related to invoicing, there's a lot more chance that we'll have a diverging solution. We'll have some other problem, even if it's not a merge conflict. So some other thing that we did is like, oh, well, I wrote this code to behave this way. You wrote it to behave that way. And we just didn't realize that we had done similar but different code. And now we've got a problem.

[00:34:15] Dave Adsit: We talk about bottlenecks in a software system. There can be a lot of them. We've already talked about one, getting questions answered. That can often be a bottleneck for delivery. One of the bottlenecks that we see when we're operating with continuous delivery is the build pipeline. If we don't have a build pipeline and we just ship once a week, we won't notice that that's a bottleneck. But if we have a continuous delivery system and it takes 45 minutes or an hour to build and deploy to staging, that can become a bottleneck. If we have our teams trying to deliver more often than that. So we can address that and increase the throughput of the system and decrease the wait time and increase the flow efficiency. One of the things that I have had the most success with after limiting work and process has been reducing scope, working in small batches. The way that we have to do this, however, is to flow of tasks. If you are looking at a bunch of tasks and and taking them all apart, but you can't deliver any value until every task is done. Well, the batch size is still the total amount of work. But if we can take a large feature and find a small feature within it that might allow us to validate the customers want to validate that it solves a certain problem, and deliver that small thin feature. that works, but is not everything we've ever imagined. And we do that. We deliver that small batch. We can flow value a lot sooner and faster. And we could actually start in the context of a business. We can start to sell that feature and say, Hey, look, we've got this feature. It's, it's not everything you ever imagined, but you know, we're shipping the color red. We know you want a whole rainbow, but for now, here's the color red. You can do all kinds of stuff with red while you're waiting for green and blue and yellow and purple, right? Like there's stop signs need red. They don't need green and blue and yellow. Right. So if you think about it in terms of, we need to, we can take thin slices and deliver those continuously. I like to tell my teams deliver, set a target of delivering two, at least two slices a week. And sometimes they only get one. Sometimes you pick something up and you're like the atomic, the atomic unit of this feature is too big to get done in three days. It's going to take us three weeks. And that is on it's less common than you think it is when you get, when you're getting started. And also it sometimes does happen, but if we can divide things into thin slices and deliver them more regularly, we can achieve value. We can create value sooner. We can flow value through our system more consistently. We can unlock a lot of, a lot of potential. In our organizations.

[00:37:16] Allan Stewart: I think it's important to think about the concept of vertical slices here. And a vertical slice means basically if you're thinking about like all, all of the layers that have to go through delivering the software, right. Thinking about it like a layer cake, maybe you want to get a slice with all the layers. You don't want to say, well, the easiest tasks here are going to be the database tasks because there are, you know, 15 tables that we're going to have to write. And that is, that is the focus of our gefallen gefallen gefallen gefallen gefallen gefallen gefallen gefallen gefallen gefallen gefallen gefallen gefallen gefallen hits all of the layers of the stack and can actually deliver value. Right. And yeah, the other thing that I've noticed with smaller batch sizes is also easier to understand. When you've got a big old batch, you have to, there's more of work, right? There's those secondary concerns, those secondary needs come up because you have to keep track of all the things that it has to do. It needs to do X and Y and Z. And in the special case, it's going to do this. And if you already did one thing, then it has to do another different thing because, you know, if you've already, I don't know, invoiced it, then you can't edit it, right? Like all of these restrictions, because you're trying to do all of it at once. As soon as you start breaking up the work, I found it gets a lot easier. You don't have to think about all the edge cases. You don't have to think about so many details and people can actually get it done faster because there's less that they're. Worried about at any given point.

[00:39:07] Dave Adsit: Yeah. I've found that large batches tend to attract more work. They tend to get gold plated. They tend to, people tend to sneak their own special project into the large batch because it's already taking so long anyway. What'll it hurt if I do, if I spend two or three days doing this refactoring that nobody wants, but me, um, and none of that can happen in a small batch. You stay focused on what is critical and what is core. And so you, you do another thing that is. Related to lean, which is you maximize the work, not done. You, you eliminate waste, if you will, you know, things that were unnecessary, get left out of the batch when it is small enough that you can reason about the whole thing at once. Um, so, you know, we, we've talked about some of the kinds of waste that we see in software development. One of them is context switching, uh, context switching. It causes a loss of productivity and it causes a loss of quality when we contact. Switch. We forget things. And because we are not computers, we are people, you know, we put down one memory and pick up the other. It's not going to be perfect. Um, so we lose again, we've talked about how you lose productivity. You lose roughly 20% productivity per additional project that you're working on. And you lose quality because you can't keep all the details in your head all the time. Um, so we want to eliminate waste to the extent possible so that we can. Flow value through more consistently. If we want to have high flow through our system, we want to eliminate all the waste. So one of the other things that we've talked about when we talk about lean is prioritization. And the reality is prioritization is an orthogonal concern. When it comes to lean, lean is about flowing value through your system. Lean doesn't necessarily have an opinion on whether or not you're doing the high value or low value thing, or, you know, whether this is the most important. This is whatever prioritization category this is like, right. Priority. You have to do prioritization to make lean super effective, but lean itself doesn't necessarily have an opinion on how you prioritize the work you're doing. Yeah. It, but it does make it easier for you to create a lot of value and get high priority things done and delivered in, in the hands of customers so that you can pick up the next highest priority thing. Yeah.

[00:41:36] Allan Stewart: I think, I think that that's one of those concepts that lean expounds upon. I feel like that originated or that we saw a lot in the agile movement, right? Like we want to have regular deliveries so that people of working software, so that people can actually see it and use it. And then, and then they can decide, right. So like more often you can choose what you want next is what is really what it comes down to is this, this iterative approach. And so. If lean helps you deliver value more quickly, then you have more chances to play the prioritization game. And, and so then at some point it might even just be we're prioritizing at random, but because we get a lot more stuff done, it's better than if we spent a lot of time trying to figure out what the most highest priority thing was. But since value doesn't flow in our system, it doesn't matter what we chose because nothing gets finished. And so. And then we're going to change our mind. Right. And then when the CEO has the idea of the day that you have to, to build. And so every day it's something new, something different, and it's always changing. Well, then we're just going to be thrashing and nothing gets done.

[00:42:50] Dave Adsit: Right. I've seen that too many times where, because we have failed to deliver value, we've been working on feature X, Y, or Z for six weeks or six months, and we haven't gotten it done. Now people get really frustrated. And now they start. Demanding that we work on something else because they were certain we would be done because we estimated that it would only take, you know, six weeks and now it's six months and, and, and we can't keep waiting for this other thing. And so we'll switch priorities entirely. And now we've got a whole bunch of in-process work. That's never going to get finished and a whole bunch of new work. That's probably also never going to get finished because we'll change priorities again before we get done with it. And that is the danger of having a system where no value is flowing.

[00:43:34] Allan Stewart: Yeah.

[00:43:34] Dave Adsit: So we talk about efficiency. We talk about how you can have high, you know, it's, we've got two things and they're unrelated or they're orthogonal to each other. And so we of course are going to create a quadrant diagram. We can have high flow efficiency or low flow efficiency. We can have high resource efficiency or low resource efficiency. Uh, I think one of the things we talk about when we talk about high resource efficiency is a freeway, you know, we talk about traffic. A freeway that is a hundred percent utilized. Nothing is flowing anywhere. That's high, high resource, low flow efficiency. It's where we all don't want to be. Part of the benefit of work from home is you don't have to go out onto the freeway and rush hour.

[00:44:20] Allan Stewart: Well, yeah, I think the freeway is a nice natural way of thinking about it. Right. Because you want things to flow, right? If you're going to have to get on the freeway, you want to go and you want to go fast. That was the whole point.

[00:44:33] Dave Adsit: Yeah.

[00:44:34] Allan Stewart: But if that's not happening, right? Like as soon as soon as you see that there are lots of cars on the freeway, you know, right? Like even a picture, right? There's a photograph and you see there are many cars. You can be relatively certain that when there are that many cars, it must be slow, not fast. And it's telling you something kind of almost intuitive about this idea.

[00:44:59] Dave Adsit: Yeah. So on a freeway, you can only have. High flow when there is low utilization, when you have extra capacity, that's how you create high flow. Yeah.

[00:45:10] Allan Stewart: So the efficiency matrix then, right? If we, if we take this quadrant diagram between flow and resource efficiency, people have given names to the different quadrants that help us to understand kind of a categorizing them. Right. So in the bottom left quadrant where. Your flow and resource efficiencies are both low. That's called the wasteland. Pretty easy to understand. Nothing's getting done. Nothing's flowing. Also, nobody's really working. So it's pretty awful. It's pretty bad. Very few places are existing in there. Most places that I'm aware of head straight up the resource efficiency ladder where resource efficiency is high flow is low. And this is called the efficient islands. Right. Right. Efficient because all the people are working. Right. Our, our resource efficiency is high. It's just like that printer that never stops churning out pages, but they're islands, right? Like when you get high efficiency, you naturally start to silo.

[00:46:17] Dave Adsit: Yeah. Yeah. You've got the, you've got the, the island of the DBA and the island of the product owner or manager and the island of the front end developers and the island of the backend developers and the island of QA. Okay. And. It takes an act of God or Congress to get anything to move from one island to the next. And you know, you have to visit all of the islands to get something out into production. And so those, those islands. Cause cause or are caused by very low flow. You know, one of the things that a reaction to we're not getting anything done, the DBAs aren't getting anything done. We'll put all the DBAs in a room and we'll make them work together. Okay. Well, now you've got all the DBAs in a room and they just conveniently locked the door because they don't know what's going on. They don't want to talk to the developers anymore. So now anytime you have a database task, good luck to you. Right. So those are the islands. The islands are those silos that you're talking about.

[00:47:11] Allan Stewart: And the sign line isn't limited to like these different cross-cutting concerns, right? Different functional areas. Like it can even happen in the team, right? So we're a team of a couple of developers. And so you and I are working together and you ask me, Hey, Allan, what do you think about this? Or can you review this or can you help me with this problem? And I say, Oh, sorry. Let me finish up this thing that I'm working on real quick because I've got my head in this other problem. Let me finish that up. And meanwhile, being resource efficient, you're like, okay, no problem. I'll just start on this other task. And so, you know, 10 minutes later, I'm like, okay, sorry, Dave. I finished up that thing. All of my tests are finally passing. So what was it that you needed?

[00:47:54] Dave Adsit: Oh, I don't remember because I've started this new thing. Exactly. Let me get back to you once I'm done with this. And I remember.

[00:48:01] Allan Stewart: Exactly. So the efficient islands, their islands, we have made silos. Across, diagonally across from there, then we have the efficient ocean. Another land of efficiency, this time flow efficiency. Flow efficiency is high and resource efficiency is low. This is where things get done, but nobody's particularly busy working on them. Right? Like any individual, the individual resource. Or, or people may not be particularly busy.

[00:48:36] Dave Adsit: I've got to say, I don't necessarily understand the efficient ocean metaphor. The best I've come up with is the efficient ocean is the, the, the great currents that go around all the continents. And I, I keep thinking of like the salt cycle in the North Atlantic, right? It takes a long time for like, it is. The salt is constantly flowing. Toward the surf. I think it's north towards the surface and it sinks and comes back. I don't.

[00:49:06] Allan Stewart: Maybe the efficient river would have been a better name. Yeah.

[00:49:10] Dave Adsit: It's, it's flowing, but nobody's really that involved in it.

[00:49:16] Allan Stewart: And nobody's pushing it along. I don't know. But the, the, the thing that I think about for efficient ocean. That helps me to, to, to understand it is firefighters. Right? The idea of there are some jobs like a firefighter. Where if a fire happens, you do the, the one thing you do not want to hear from them is sorry. We're busy. Yes. You don't want hold music on nine one. You want them to be able to drop whatever they're doing and come and fight the fire. And so at least historically, I know that firefighters would often have various things that they, they work on. And these little side jobs, you know, we're washing the fire truck or we're. Doing some gardening in the, in the backyard or, or something that if we had to drop everything and throw on our uniform and slide down the fire pole, it would be okay. Because we are intentionally not being busy. Right. Or not so busy that we can't drop it all and do this important thing that needs to be done urgently.

[00:50:24] Dave Adsit: Well, and in a software system, I think of this as my tech enablement group or some group, some companies report, refer to it. As the center of excellence. These are the people who are working on the important non urgent things. Like I've got my architect in front and engineer are working on enhancing the design system. Well, if a team has a need that comes up urgently, they can put that on pause. It's fine. It's fine. If we don't get whatever they were doing, basically. I want to have some. Fire. Firefighters. Software firefighters who are actively engaged in improving the code base, but not from the product perspective. Not from the perspective of we have deliverables in the product that. You know, people are counting on. I want them working on other things. Improving the reducing the bottleneck or improving the throughput of our CI CD system. Increasing the test quality and coverage for the legacy part of the system that we wrote before we knew that we needed to be doing test. Driven development. So. Upgrading. Or doing maintenance on things like. You know, library versions or, you know, researching. New potential uses of AI tooling in our system. Right. There's a lot of that very valuable, but not particularly urgent things that can be done. That leaves these people. Free to be very reactive and responsive. When. A need arises. Yeah.

[00:52:03] Allan Stewart: This is also one of the reasons why I like collaborative coding, whether it's pairing or mobbing ensemble is because it kind of, it, even if it's not working very well, it puts you into this state naturally. Right? Like even if you haven't built up the skills for doing ensemble programming, you are more likely to end up in the efficient ocean because. You are working on just one thing together. Single piece flow. Trying to get this one work done and maybe. Right. So like the, the thing that a lot of leaders will often fear is that only one thing is getting done. And if I've got four developers in a mob, only one of them is contributing at a time. Right. That's the fear. But the thing is that if, if there's anything that interrupts that. One person, well, everybody else was on this same task. It can continue to move forward. You might end up in the efficient ocean. Even, even if you're really bad at, at mobbing. But if you get better at it and the people are engaged, well, then that pushes you up from the efficient ocean. So you have high flow and high resource utilization into what is called the perfect state or lean. Mm-hmm.

[00:53:30] Dave Adsit: Yeah. And I think it's critical to point out that we refer to that perfect state as lean because there can be a misperception that because we are optimizing for flow efficiency, we don't care about resource efficiency. Right. Which is not actually the case. You know, we ask people, would you rather be busy or get things done? Well, probably both. I would like to get things done and also be busy. There is nothing. I don't know if you've ever worked one. working hard. We, we actually have a pretty fun job. Writing software is actually pretty cool. Solving problems, working with computers. Like there's a reason why most of us got into it and it's because we enjoy it. Um, so we want both to be busy and get things done. And if we have to choose, it's better to get things done than to be busy all the time. Right. So when we talk about lean, there is a philosophy of lean that's been developed over who knows how many years. Uh, people talk about it coming out of Toyota and the Toyota Kata. Um, it's possible that it predates that even, but it's been in improved and enhanced sense. So lean philosophy talks about a couple of things like just in time, which is an option opportunity to reduce work in process, to work, reduce inventory, things like that. We want to work in a just in time manner so that we reduce planning waste. We reduce planning waste. Right. One of the most critical things, time is continual improvement. Uh, often you'll hear it referred to as Kaizen. Like we want to be continually improving our system. Uh, that means improving our own skills, improving our delivery, improving our product, improving value, improving flow. We want to be improving all the time. And so that means you'll stop and analyze what you've done and, and, and think of ways to make concept that gets talked about far less. I think it's a Kaikaku, which is radical improvement, which is talked about less because it has a tendency to be more risky and perhaps the kind of thing that we were already trying and failing at before we decided to adopt small steps and do Kaizen with continual small improvements versus trying to do one giant improvement once a year. Right. So, but you know, other things that are part of this lean philosophy, we have situational awareness or visibility of what's going on. Uh, often referred to as Jidoka. We want to basically make work visible. We want to, that's what Kanban means, right? Kanban is the signboard where the work is posted. Mm-hmm. We want to make it visible to everyone. What is going on? Honestly, we can avoid status meetings by making our work visible to anyone who's curious and wants to look at it.

[00:56:55] Allan Stewart: Yeah. What if we had the status and it was on a signboard? What was, visible for everything.

[00:57:02] Dave Adsit: What if indeed we did that and then we didn't have to have status meetings? Happy day.

[00:57:09] Allan Stewart: Yeah. Yeah. I think that that, that philosophy also helps with the, with the situational awareness concept, right? Like once, once you've made things visible, then you can know what's going on and not just so that you don't have to report status, but also so that you can interact better, right? Like focus on that focus on that focus on that focus on that focus on that focus on that focus on that focus on that focus on that focus on that focus on that focus on that focus on that focus on that focus on that focus on that focus on that focus on that focus on that focus on that focus on that focus on that focus on that focus on that awareness. And if some, and if there's a problem, we're aware of it and people can react to the problem instead of it being siloed and say, oh, well, we didn't, we didn't realize that you had

[00:58:08] Dave Adsit: a delay. Now everybody knows. Right. So there's a lot of practices. Some of them come out of agile or XP or any other, uh, software practice, uh, you know, philosophy that align with and help support working in a lean way. Uh, one of them that is critical is the regular retrospective, the team retrospective, the retrospect, the, you know, the project retrospective, like having a regular retrospective is a critical strategy or tool for delivering Kaizen or continual improvement. Uh, I encourage my teams to have a retrospective every week. And I've worked with teams who do what they call continuous retrospective, which happens at the end of every day, or sometimes more than once in a day where we are doing continuously doing small experiments in, in the retrospectives that I have my teams do every week, they are encouraged to be experimenting with an improvement on the system and then measuring the results and determining whether to make that part of their standard operating procedure or abandon it and try something else. Right. So we, we do, we leverage retrospective as a way of understanding what has happened recently, what things we would like to improve and then selecting and executing and measuring on experiments to improve our system.

[00:59:35] Allan Stewart: Another practice that goes along with lean that I'm a big fan of test-driven development, right. It helps us break down our work for one thing. Like even, even the one work task that we're doing gets broken down into a lot of smaller steps. Um, and I feel like it helps me with that just in time concept, because when, when I practice test-driven development and I'm really focused on, okay, I'm writing the test first and I'm thinking about the objective, I'm less likely to go off and start gold plating and doing other things that turn out to be unnecessary because I just, what, what is the bare minimum that is needed? I think through the test, I say, okay, well, I need it to do this. And I'm not worried about it from like a general purpose. Well, what might it need to do later? Like, no, just what passes the test, get that done just in time, making the changes that I need until the feature is done or the card is finished and it gets shipped. And then if we need to make other changes to it later, well, we'll add the tests for that as at the time that we actually need it. Instead of trying to guess ahead of time,

[01:00:44] Dave Adsit: what might be needed. You can eliminate the waste of over-engineering, which is definitely a thing in software development.

[01:00:51] Allan Stewart: Yeah.

[01:00:52] Dave Adsit: You know, we've already talked extensively about collaborative programming, both on this episode and others, but that is one of the practices that allows us to maximize flow and lean into the concepts of lean, lean software development. Related again, we've, we've talked about it quite a bit is using a Kanban board and applying whip limits to our Kanban board. By doing that, we can, again, increase the flow of value. And, and a lot of these things come as, as we apply these practices, we can create more value with the same number of people, the same resources, if you will, we create more value or we deliver more value sooner. And that is phenomenal from the perspective of, you know, investing in a software development team.

[01:01:49] Allan Stewart: One of my favorite places, is to set a whip limit on the Kanban board is also the backlog. So you can't keep adding more stuff to that backlog. Just delete it. Just delete it. There's a few things you need on there to keep because you're going to actually work on them. But once it's full, something else has to get deleted. Don't, don't try to keep, keep it. Cause you'll remember if it was important,

[01:02:11] Dave Adsit: it won't go away. If it's important. Yes. That's the critical thing. If it's important, you'll keep bringing it back. And if it's not important, enough to prioritize sometime in the next couple of months, you're probably not going to get it. It's not that important and you're not going to ever get around to it because you're going to keep putting other stuff in front of it. It's just delete it.

[01:02:32] Allan Stewart: So we've talked about all these concepts with lean. I want to kind of wrap us up with some discussion of the benefits of lean. Like why, why would you do this? And, and I think we kind of led with that concept of, well, we're going to deliver value sooner. Yeah. And so I think about it, like, let's take an example of a team that focuses on resource efficiency. And so they regularly deliver 10 features in every two week sprint. Well, we can contrast that with a more lean team or, or even the same team, but focusing on flow efficiency now, well, they might only release one feature to production every other day. Let's just, let's just take that as an example, right? So it's not even particularly, it's not even particularly resource efficient. And at first blush, we say, oh, no disaster. This is a problem because before we got 10 features every two weeks, now we're getting one every other day. So we're only getting five features every two weeks. So this is clearly bad. It's terrible and awful and we shouldn't do it. It's 50% less stuff, but on average, any single feature So can this increase is this increased speed? Can it offset the drop in feature output? Yes. Yes, it can. So first of all, we recognize that not all features provide equivalent value. Some are much more valuable than others. Some can even yield negative value. It turns out. And often it's impossible to tell the difference until you've released it. So releasing a feature sooner gives you faster, faster, faster, faster, faster, faster, faster. If it's not delivering the value that you thought, well, now you have the opportunity to make course corrections and you can do it faster. Right. There's less lag in the system. So you might need to take a second pass at the feature or you might decide, Hey, you know what, actually this feature isn't valuable. And so the other five things that we were going to do related to this feature, we're not going to do those. Well, that might save you a week of development. Cause if those five features that they were going going to do, we found out within two or four days that they're not worth doing. Well, you just saved yourself an entire week. But then secondly, now you can also capture market share and money sooner. Instead of waiting two weeks to get any value, now you're getting some value after the first two days, right? So if I release a feature on day four, which yields $100 a day in revenue, well, now we're going to get $600 more than if we had waited till the end of the sprint. Small features also help us measure impact, right? So if we release all those 10 features at the end of the sprint all at once, it's hard to know what changed, right? If there's a problem, which thing had the problem? Which one of them is causing the bug that's causing the outage? Or even if there's no problem, which one, was the one that actually made a difference? It's harder to measure that because they all happened at once. And so in my mind, you add up some of these things and even if the team appears on paper to be doing, you know, 50% less output, it might be better output that is

[01:06:05] Dave Adsit: more valuable to the company. Well, one of the things I'd like to add to that is that you are, you're actually hiding signal if you release large batches every few weeks. Imagine that you are an organization that is focusing extensively on reducing churn of customers. You know, you're like, we are good at closing customers. We're not good at keeping them. We want to reduce churn. So if you release your 10 features at once and all of them are intended to improve churn, and in fact, five of them each reduce churn by 1%, but five of them increase churn by 5%, at the end of the sprint, you'll say, well, crap, nothing mattered. We got to go back. to the drawing board. If instead we had released those one at a time, we would have seen, oh, each of these five is helping and each of these five is not. And so we'll turn these five off and double down on the five that are helping. And so we've obscured some signal by operating in a larger batch with less frequent delivery with, you know, slower flow. You know, the system is not flowing as well when we're releasing less frequently.

[01:07:16] Allan Stewart: Yeah. Work expands like a gas, right? It's going to expand to fill all the available space. So it's not, I've never been in a situation with software where we said, oh, well, there's just not enough to do. There's always more to do. And lean helps you be more effective with an inherently limited capacity. Also, software development tends to be knowledge work and knowledge work doesn't jive well with resource efficiency, generally speaking. And it certainly doesn't fit with the, uh, like assembly line style. Hey, we'll just replace somebody if they screw up or, or, uh, get tired or whatever. Right. Um, knowledge workers are non fungible. They can't just be can't swap out

[01:08:06] Dave Adsit: placed interchangeably widget, widget makers. Yeah. That's what we talk about. The, uh, productivity dip from adding a new person to a team. Uh, it takes time. Yeah. And effort to onboard someone to a team, to train them, to be a productive and effective member of your organization. And if we are not willing to take that time, you know, we, we can't just swap people out. And so if somebody calls in sick, we can't grab a developer from a different team and say, okay, work on this team for the day while so-and-so is out. Like we just lose that productivity permanently to some extent. So, so you and I have both worked in many different ways. Uh, I, I think I'm the only person I know who's actually worked in real waterfall, but like by the book waterfall, as opposed to, I read about this waterfall. Like we actually did it. We released once a year, we spent three months on planning six months on execution and three months on QA and then probably two weeks trying to ship the thing. That was the, uh, that was the

[01:09:08] Allan Stewart: book where they said, and here's the example of what we shouldn't do. Right. That's correct. In

[01:09:14] Dave Adsit: what, in the original waterfall, we can blame a page fold for the fact that so many people adopted the wrong way of working. Um, so we've worked in various kinds of chaos, various kinds of agile and a few kinds of lean. And I will say that for my teams, we have been most productive, felt best about the work we were delivering and gotten the most done when we have used lean techniques, but we were able to focus on flow efficiency and getting work done, even at the expense of the

[01:09:51] Allan Stewart: resource efficiency. I recommend it. I can't think of a better summary or recommendation.