Crafting Code Podcast

[00:00:00] Allan Stewart: Welcome to episode two of the Crafting Code podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately I've been thinking about how to make software developers resilient to distractions.

[00:00:16] Dave Adsit: I'm Dave Adsit, CTO, and recently I've been thinking a lot about the potential for using bottlenecks to create value in a system.

[00:00:24] Matt Baker: Them. I'm Matt Baker, software architect. Lately, I've been thinking about forming teams around data

[00:00:31] Allan Stewart: domains. And our topic for this episode is rescuing a failing project. So, to get things started, what is a failing project? How do you recognize one? How do you know if you're in a failing project?

[00:00:49] Matt Baker: Oh, bad answer, but I think it applies. You know, you just know, you know, you know, when you dread going into work and when you leave work defeated, you know, you know, when the release cycle is 27 hours or bugs galore. I feel like I could go on and on here, and I'm watching the grimacing faces of my co-host. It's kind of confirmation that I'm hitting some nerves.

[00:01:15] Allan Stewart: So this is the Justice Potter Stewart version of "I know it when I see it."

[00:01:20] Dave Adsit: Yeah, it is. I think that it's like being sick in that there are a lot of symptoms, but you feel it all the way through you. Some of the symptoms I look for are a slowdown in delivery of value. I look for increased demand for time and investment and giving to the company. I hesitate to use the term death march because it's overused in a lot of contexts, but man, I've lived them. And when people are asking you to put in longer and longer days, or when the manager more subtly just sits there working from 7 a.m. until 9 p.m.

[00:02:16] Matt Baker: Yeah, I've worked a 36-hour shift. I've slept on server room floors.

[00:02:22] Dave Adsit: So you wrote really good code the last 24 hours of that shift, right?

[00:02:28] Matt Baker: That was great.

[00:02:30] Dave Adsit: Best code I've ever written. Best code. The parts where I was lucid, man, you wouldn't believe.

[00:02:37] Matt Baker: Another thing I think that occurs is deadlines. I believe that there's some relationship between a failing software project and the increased demand for deadlines. Tell me when it's going to be done. And I also believe that the death rattle is the "give me two weeks." "I just need two weeks." The perpetual, "I just need two weeks."

[00:02:56] Dave Adsit: I think you're right about the deadlines for sure. People start asking for deadlines or start demanding code be done by a point in time because they don't believe that otherwise they're going to get it at all. Right. That's a pretty good sign that your project is failing. Oh, man. Allan, what do you think?

[00:03:19] Allan Stewart: Well, I think just the fun aspect of it. It's like if you're not enjoying going into work anymore, that may be related to the fact that the software project is failing. And so you're not tackling interesting problems. You're not feeling that delivery of value and saying, "hey, I worked on something and customers are using it now." And whatever that is that's stopping that, whatever is lengthening out that deployment timeline, whatever it is that, you know, maybe it's, maybe it's really bad code that's slowing you down. Maybe it's the team. Maybe it's something else. You start to feel that in just the enjoyment that you're having with the project.

[00:04:08] Dave Adsit: I was just thinking that one of the ways I know a project that I'm working on is failing is when I go home and the people who are there waiting for me tell me that my project is failing because I bring that baggage with me. And they get to experience it too. And man, that's worse. That feels worse even than just letting the project fail. Like bringing the baggage of failure home and having whoever it is that you meet at home know that things are not going well and haven't been for some period of time.

[00:04:50] Matt Baker: Yeah, I think there's this amazing thing that happens when we sleep in this case and it resets, maybe at least my temperament. So when I'm in a failing project, oftentimes I show up at the end of the day, I show up at home just defeated. I don't know what to do. I can't see a direction. I just feel like all hope is lost, but I calm down, I relax. I spend some time with people that I love, and then in the morning, I'm ready to go again. What I've noticed is that when I'm in a failing software project, that's a daily thing for me. I, in the evening, I'm defeated. In the morning, I'm rejuvenated. I'm ready to go, but without fail eight hours later, I'm back to that defeated state. So, let's talk about some of the different aspects

[00:05:33] Allan Stewart: of parts of the system that might be failing. So one of them could be code, right? And I think especially in the software crafting community, we talk a lot about, like, the big ball of mud, all this terrible code that gets written. And it's not a value judgment against the people who are writing it, but it's just these different approaches to writing software that are very additive, tacking it on, just get it working, band-aid it, slap it together, and before you know it, you have this code base that's just really hard to maintain. You know, it's buggy, it's hard to change. What other aspects of code have you noticed, like like, how do you know, um, so Matt, if you went into a software project that you know you had never seen before, a new code base, how would you know if that code was a mess or not?

[00:06:39] Matt Baker: Yeah, you know, I think we all developed little heuristics to, to answer this question, and some Some of those are going to be preferential. Let's see. Some of the things that I look for, lack of tests. I look for that. I'm looking for size of individual files or size of functions. I'm looking for degrees of coupling. Maybe some, those are the ones that come to mind immediately.

[00:07:04] Dave Adsit: I start looking at a software system from the outside. I want to know how many bugs have been reported but not resolved. Is the product slow? And what does that mean? Like, who knows what it means to be slow? Different systems have different slowness characteristics, right? But if you ask people, "is it slow?" They will have an answer. I want to know, how hard is it to change? For the last feature that you developed, how long did it take? How long does it take on average? I want to know, how often do you have failures in production? And how long do they last and who fixes them and how do they know what to do? And then one of the big ones that's typically hard to measure or impossible on a failing project is, are your customers happy? I mean, you might have a hint at it, but you might not even have customers yet. You may be failing before you even get there.

[00:08:05] Allan Stewart: So it sounds like there's a lot of aspects to this, that it's not just the code, right? There are people involved with this. There are people that are writing the code. So is it the team? Is the team the problem? What signs do you look for to see whether there is a problem with the team that is causing a failing project?

[00:08:27] Dave Adsit: So in most management books that I've read, they always say that every problem is a people problem. And I think to some extent that's true and to some extent that's the only hammer I have is people, so I'm going to pound on them as hard as I can. I mean, something about nails. But there is some truth to the fact that you have to have the right team. Does your team have skills? Have you given them the training to have the skills? Is the team aligned? Are they working together towards a purpose? Are they working against each other? Do they collaborate, or does everybody sit facing the wall with their headphones on and they talk at lunch for two minutes maybe if at all, uh, like, does the team actually get stuff stuff done? One of the big ones, one of the easy ones to feel but not necessarily to measure, is the morale of a team.

[00:09:28] Matt Baker: Yeah, you know, what you said about being able to feel the morale really speaks to me. I was on a team one time where we decided to do a leap year kata. And if you've done that kata, you know that it's not a huge ask and you should be able to solve it relatively relatively quickly. And we took an hour to try and do the Kata. We didn't finish it. We didn't really get far at all. And in fact, we spent the entire time arguing. We just had a lot of communication issues. We weren't able to, to move forward because no one could agree to anything. So when we were in that mob, we did it, you know, with mob programming, you, you definitely could feel a low morale.

[00:10:05] Allan Stewart: One of the things that that makes me think about is that maybe what's going on with a team is not so much about the individuals because I've worked with a lot of teams over the years, and almost everybody that I work with, I like as far as as working with people. But not every team has had the same skills. Not every team has been aligned around the same practices. So it might not be the team, like, it might not be, like, the people. And so when we talk about teams, I think it's important to recognize it's, like, it's not a value judgment against any person. Occasionally, occasionally there's one person that is just hard to get along with, your personalities clash, you just can't stand being around a particular person. And in that case, yeah, okay, maybe it's, maybe it's a person, maybe it's the one person who is just a jerk, and there's no two ways around it, but most of the time that's that's not the case. Most of the time, there's something around the team, whether it's practices or management or something else that is contributing to the failing software project more than the individual team members and their software coding skills.

[00:11:24] Dave Adsit: Yeah, there's definitely a lot of things in the system, which is basically, like you say, that's everything around the team that's delivering the software. There's something in that system that is not working, and that could be so many different possibilities. It could be things like the team room. It could be the team's practices that they've chosen or been chosen for them previously. We talked about how the CTO at one of your companies, Allan, told everyone they were going to TDD but not how to do that, right? That could be a problem. It could be, how are decisions made? Are decisions actually ever made? Are we in a constant state of flux and turmoil and we don't ever have a decision that we can work toward? There's so many things. How do you interact with other parts of the company if you do at all? Again, a lot of these problems come from not having the communication pipelines. Basically, anything around a team that affects their motivation influences their ability to be successful could be part of the problem for that team.

[00:12:41] Matt Baker: Yeah, this really speaks to me, and I want to add just some more concrete examples. Oftentimes you can find yourself in organizations where the team does not choose the rate of input of new work. So they're just sort of given new work. And if it's really bad, they're given new work without any consideration for current work. So, you know, if the team's already, let's call it, they have this utilization. If the team's 100% utilized, meaning they're doing as much as they can do, and you add more work, it's not going to do what you think it's going to do. You're going to slow everything down further. But on this subject, what we're talking about, which is how do system things affect team morale, that sucks for team morale. I mean, you're already at capacity. Now you're going to come work weekends. So I believe that that's it. And I also think that there's another concrete that really speaks to me lately is trust. If the team doesn't feel trusted to be able to go execute on the problem that they're trying to solve, morale is just going to be low. And I don't know all the ways that's going to manifest itself. But having been on teams that were not trusted, I know that it does, well, I shouldn't say, in my experience, it's definitely affected the morale of the team.

[00:14:01] Dave Adsit: I worked on one team where we had so many items in the backlog that we finally decided to do the mega grooming or whatever we called it at that point, where we quickly sized every card in the backlog log without much context beyond what it said in the title. And then somebody did some back of the envelope math, and we came up with an approximate completion deadline or completion date for the items we had so far of about five years in the future. And we all looked around at each other and nobody felt like they were going to be there for five years. We all got real bad, real quick. And then we started going through and, and, you know, eliminating some items, right? We started trying to do some things to save that failing product project. Um, so I could talk about some of the things that I've done, but what about you guys? What are some things that you've done? Once you've identified a failing project, what do you do next? Like, everybody's been on one, right? I'm sure people are like, "yeah, yeah, I get it. Now, now, what, what are you telling me? What are you you telling me?" Oh, so you want, you want some answers.

[00:15:21] Allan Stewart: At least you want to know how to rescue a failing project. Uh, well, for me, one of the big things that I have found works wonders is getting a continuous integration and or a continuous delivery pipeline set up. Um, just that ability to get code off of an individual developer's machine, integrated with all the other code that that is in your trunk or master branch and test it, run all the tests that you have, which maybe is just the one that you just added just now, but it runs the tests, and you can deploy it. That's such a huge thing. And now it's to the point for me that if I'm starting a new project, that's one of the very first things I do. It's like, okay, we're gonna build a thing. Oh yeah, maybe it's gonna have a database. Maybe it's going to have an API or whatever. It's like, okay, great. There's going to be a thing. I don't even care. What I'm going to do is deploy Hello World with a test and make sure that it does continuous integration and we deploy it into our staging environment and that we have it set up so that we could deploy it into production. And sometimes I do now. I'll deploy it all the way into production because nobody's got that URL because this is a brand new thing. Just go ahead, write on out to production, and then every other thing you do, you have that ability to get things out, because with these projects that are dying, that are failing, I feel like there's, there's no momentum. There's no way that you get stuff into production, and, and putting Putting that change in place is huge.

[00:17:08] Matt Baker: Yeah, it sounds like feedback. You've recognized that a project needs feedback, and it almost seems like there's this dopamine hit that occurs when you experience that feedback loop. When I ship something to production, especially when I see someone use it, that's incredible.

[00:17:25] Dave Adsit: Right, even if it's not the thing that they really, really, really wanted, even if it's just a hint at that, something that they can see and say, Say, "oh, by the way, now that I see this, I have an idea for something that would get me some value quicker or easier." One of the things I was going to make a snarky comment about, what if it's really hard to deploy my system? Then how do I do a CI pipeline? But, you know, like they say in software, "if it hurts, do it more and automate it more," right?

[00:18:02] Allan Stewart: So yeah, if it's, if it's hard to deploy then you probably want it automated, because more often than not, I find the projects that are suffering in the deployment phase have the problem because they are trying to get code out to production, and there's these all these manual steps, and it's like, oh well, we forgot to run the one script. Oh, yeah. Did you remember to? To reset the server processor or something that a human forgot to do.

[00:18:34] Dave Adsit: If you are in the team room or the war room, worse, and you are trying to do a release and it sounds the same as when your team is sitting around with their phones playing Space Team, you have done it wrong. My teams currently deploy once or twice a day, no less frequently than that. And sometimes the deploys are just small, but we know that they did something that is potential value for someone.

[00:19:11] Matt Baker: Yeah, you mentioned war room. I want to offer a positive take on that for a minute. One of the things that I think can be very effective, depending on the situation, when you walk into a failing software project is to take all the people that are working on the project and go sit in a room and just kind of ask a simple question, what's the first thing we need this thing to do? Maybe it's allow users to sign in or maybe it's allow someone to send an email, whatever it is. Then, as the group, solve that problem. Solve the first one. And all the way, code it, ship it, test it, design it, whatever needs to happen, but do it as a group, and, and you do that a few times. And something I've noticed is that there's this reorientation that occurs where the team stops saying, "well, it's Matt's solution versus Dave's solution." And then it's, it seems to change to "it's Matt and Dave versus the problem." And we get into this mode of, "all right, we need to get a user to sign in, we need to get a user to send an email," and, you know, you kind of go through the list, but but there's, I don't know exactly what it is, but there's something kind of magical about war rooming that. Now, war rooms are typically in response to firefights, and that's not, you know, I'm not necessarily saying that, but what I'm saying is get the necessary party, people, rather, go into a room and just start working on it. And I think you might see the morale improve.

[00:20:46] Allan Stewart: Yeah, the positive aspect of a war room, I think, is that collaboration, right? But the crazy thing is that we tend to do war room at the end when everything is on fire, everything is broken, everyone is sad. That's when we say, "well, I guess maybe we should work together instead of working all individually."

[00:21:06] Dave Adsit: But there's no way you could ever work together all the time. That would be far too effective.

[00:21:13] Allan Stewart: But Dave, don't get me started talking about that topic, because today we're talking about rescuing a failing project.

[00:21:19] Dave Adsit: And I think one of the things that is interesting about a failing project is that typically it's failing for one of those three reasons. It's the code, it's the team, or it's the system. If it's the system, it's got to be pushed up to a management level. Managers have to recognize the problem and start working it. They should do so with input, right? If it's the code, we know what to do. There's no better way to improve the quality of your code than to start testing it with automated tests. And there's no better way to improve the quality of your team than to start collaborating regularly. By which I mean not pull requests. I mean pair programming or mob programming, like, get people to work together on the same problem, not adjacent problems.

[00:22:14] Matt Baker: Yeah, and I, I think you can grow out of this. So maybe you start with this, you know, everyone pile into the same room and work for a little while, and maybe from there you, you graduate into a remote PR thing or, or something else, right, where the collaboration is a little bit less to accommodate some other variable in the situation. I think, you know, that that can be fine but I do think it's important to start together and figure out your ways of working and then start to split up if that's what you intend to do. But I tend to agree with you, Dave. The further you get from that face-to-face, hands-on-the-same-keyboard collaboration, the harder it gets.

[00:22:58] Dave Adsit: It's too easy. I've read somewhere that the half-life of trust is something like six weeks. It's too easy to blame the other guy on the team if you're not working with him. If the two of you are sitting together at one computer, then you're working against the problem. And that does, it builds something that is essential to rescuing that failed project. If we're working together against the problem, then we have a chance. If we're working against each other, we have no chance.

[00:23:37] Allan Stewart: Yeah. And going back to continuous integration, I really like that as an enabler, right? Because continuous integration, especially when we think about CI not as just a set of tools, right? I'm not just talking about Jenkins or Bamboo or whatever your favorite, you know, favorite CI tool is. But continuous integration, the concept, is about, hey, get all of your code together more often. And when you're doing that, then it naturally leads you to, to collaborate more, no matter how exactly you're, you're collaborating. I'm a big proponent of pairing. I'm big proponent of mobbing, and, and you're getting that when you, when you mob programs, you are creating continuous continuous integration, because now you don't need a server because everybody's writing the code at the same computer and it's all integrated and you can see whether it's working or not. So it enables you to do those things. And if code is the problem, then it enables you to have tests. Because if you have tests but not continuous integration, then most likely what happens is that only the one person runs the tests and everybody else breaks the tests.

[00:24:58] Dave Adsit: So you've got to get into that pattern. That's a good recipe for not running the tests anymore.

[00:25:03] Allan Stewart: Yeah, then you don't run the tests anymore, and then you comment out all the code, and then you just don't have tests anymore.

[00:25:09] Dave Adsit: I've worked on a project where it had all the tests at one point, and they were just excluded from the build file because they started failing, and that just made it so much easier to keep writing more and more code faster and faster until all of a sudden it made it impossible to continue writing any code.

[00:25:28] Matt Baker: Yeah, who wants blinky tests? So if you get a blinky test, just delete it.

[00:25:31] Dave Adsit: Just delete it. In fact, just delete the whole test suite. The hard part is that once you stop running your tests, you can never start again until you start over. Yeah, you basically have to start over. But the tests give you confidence. And they, once again, improve that feedback cycle. That feedback loop is even shorter with tests than it is just with CI.

[00:25:53] Allan Stewart: And then all of a sudden you feel like, oh, I can change code. Because in a lot of projects that are headed downhill, in my experience they the developers are scared of changing the code. And so it's like, well, instead of changing that bit of code, I'm just gonna add a new different code over here. And so then it grows. It starts to spread, almost like a cancer, throughout the system. It's like, no, we're just band-aiding all these different, like,

[00:26:21] Dave Adsit: lumps of code together. But for what other purpose am I gonna use switch

[00:26:25] Allan Stewart: statements? That's, that's not what they meant by open close principle, Dave. Oh, we'll have to have

[00:26:33] Matt Baker: a conversation about that at some point. No, I, I think there's truth to that though. I, I most software projects, there's, there's a module or a set of modules that are known to be kind of the don't touch that module, you know, or that, that one, there's so and so is the only person that knows how to work on that module, and, and to Allan's point, I, uh, I, it's almost like that dirty module just perpetuates itself, because there's some precedent that's set in that module that it's a comment on the rest of the system, and the people just continue that. I worked on one piece of software that, um, someone had written their own ORM, like, their own query language, the whole thing and it was used in some core functionality. And if you coupled to it, you had to also use it. So similar to that saying, "given a barrel of water and a drop of urine, you have a barrel of urine." That's what it was. It just spreads and left without interference. It'll just, your whole code base is now based on a custom ORM, and good luck. That sucks. When that thing gets to be a few hundred engineers and a million lines of code, then it's rather unwieldy.

[00:27:52] Dave Adsit: But it does keep people employed working on that custom ORM all the time. Imagine the value. Wait, that's not value.

[00:28:01] Matt Baker: I mean, it's a value. Right.

[00:28:05] Allan Stewart: The other thing that that confidence that the tests bring in for me, one of the big things there is your ability to refactor code. And if nothing else, as soon as you can start changing the names of things, you have just enough confidence now between your tooling, between your testing, that you can rename something. Then all of a sudden you can go in and say, "okay, what even is this code? Like, I don't even, I can't even understand what's going on here." And so you think about it for a little bit, and you figure out what it is, and then you change the name of it. You say, "okay, now you have a name. Now this other thing has, this method has a name. Now I've extracted a method and it has a name." And all of a sudden you start building up, or you find the thing. It's like, "oh, that says user ID, but it's actually the administrator user ID." And now all of a sudden you have more context, and you can understand better, and you don't have to think about a hundred lines of code. You can just worry about a few lines of code, and suddenly your your system starts to heal itself. It's like the, uh, you know, putting new coral in to a dead coral reef, and it starts to grow again and heal. Yeah, it's pretty amazing what you can do once you've

[00:29:35] Dave Adsit: built some amount of confidence around changing your code, and it starts to snowball to the, the more of your code base you put tests around. So, kind of going, kind of going back to the idea of, of crafting, my family has always done woodworking. My dad was a contractor, and so we always did that stuff. And so one of the things that for me is important is when you're working on a piece of of wood, you make a jig, or you use some clamps, or you use something to hold it together while you're making a change, while you're attaching one piece to another, or while you're cutting one piece off from another. You want it to stay stable. And that, to me, is one of the big advantages of starting to add those tests. You've added braces and clamps and support to the code so that you can change it, because you know it needs to be changed. The code is a mess, and now we need to start fixing it. So adding in testing. And there's a lot of techniques for that. And I don't know that we really have time to dig into all the possible techniques. But if you find yourself in this situation, there's a lot of things you should research. Like test-driven development, effective unit tests, good unit tests. Things like contract-driven testing, acceptance testing, UI testing. There's just so many different options. Think about how you could white box test it or black box test it so that you can create some of those clamps and some of those jigs that are holding your precious project in place so that it will be safe while you are busy ripping some of its guts out and replacing them with much, much better guts.

[00:31:29] Matt Baker: Yeah, I'm now thinking about the refactor versus rewrite debate, and it's interesting to me that I believe that what we've been talking about is a refactor. And for me, one thing that's important when I go into a project like this is to remember that it's just going to take a lot of time. It was a lot of little decisions that got you there and it's going to be a lot of little decisions that get you out of there and it gets further complicated by the fact that you have to try and balance, typically anyway, the continual delivery of value with this refactoring. You can't just shut down the warehouse for a year and rebuild the thing and then flip it back on and go back to work. You can't do it. So it's hard, and it's a slog. And I, for me, it's important to remind myself to settle in and, and Allan, like you've been saying, get that one thing under test, do that one rename, and then, using your CICD pipeline, get it out, get it shipped. And then just get into that flow, and it's going to take a while, but, you know, you'll start to see the progress. And like Dave said, it does snowball. Um, you know, a month into that kind of work, you'll start to, to really start to reap the benefits of it. Yeah, and I think you, you

[00:32:43] Allan Stewart: You mentioned refactor versus rewrite. And more and more, I find that with a lot of projects, you actually end up needing both. But the problem is when we say "rewrite," what we usually mean is, "well, let's throw all of this garbage away because I'm just frustrated with it. I can't figure out what to do about it. And so let's just throw away everything." And the problem is that you can never catch up from that, right? It's the whole tiger team question. It's like, "oh, well, we're going to start building all the new code from scratch." And you just can't catch up to all the features that the other thing did. But if you cut it down small enough and you say, "well, I'm just going to rewrite this one little piece. It's just this one method or just this one thing." Sometimes that's easier than refactoring. Refactoring requires tests to do it properly. Otherwise, you're just changing things. And so sometimes it's easier to figure out, "hey, what is this intended to do? Let's do that instead. Let's write the little piece that does that thing, and then we'll insert it into the project." And so ultimately, I think you need both. Sometimes you refactor. Sometimes you rewrite. But the key is that you're just rewriting really small things.

[00:34:05] Dave Adsit: I would completely agree with that. I, anytime I've been on a group that has discussed the rewrite, the question that always comes to mind for me is, "but we're the same people that made the first mess. How do we know that we're going to do better this time around since we've forgotten all those little decisions that we've been making for the last two years?" I'm always really hesitant about rewriting in the large because we are the same people. We may have learned a little bit more, but we're likely to make the same mess again. But if we refactor, we give ourselves an opportunity to make safe, repeatable, reversible, incremental improvement in the system that we have, which enables us to learn and improve the system at the same time without risking the business by doing the, "pull the ripcord, full stop. We're not going to work on this until we've rewritten everything."

[00:35:06] Matt Baker: And I think that's hard. I think it's hard to rewrite a system a little bit at a time, and it requires a certain set of, I think it requires some skills or a strategy or two, some of the things that are coming to mind. You need things like feature flags to enable maybe parallel implementations for a while. Anyway, we could talk about the techniques, but yes, I agree with you, Dave. And I also think that a team that sets out to rewrite something that they wrote originally will write it better the next time. But I just don't think it's going to deliver on what they think. You know, "when we rewrite it this time, all of the problems will be solved." And you're going to get some of them. The ones that are most painful, probably. You'll come up with a solution. And then that solution will have, you know, a new set of problems that you didn't expect. But the big risk there for me is just the length of the feedback loop. You know, if you set out to go rewrite a thing for a year, you're going to think it's great until you try to deliver it. And so, you know, in the presence of that ambiguity, I think you need to tighten up those feedback loops, which is something Allan's been saying.

[00:36:13] Allan Stewart: Yeah. I think there's a lot of discipline involved too, right? It's like, are you disciplined enough that you can write a small piece of code or change a small piece of code? And what is your discipline about writing code in general in the project? I mean, at some point you got to this mess. What have you changed about how you write code so you don't just go and make another big mess?

[00:36:42] Dave Adsit: Right, and one of the things that comes to mind for me is how do I even know where to start? What tells me which pieces are most important? Is it just my gut, or can I do something to find out what the first piece is and then the second piece that I should tackle?

[00:37:03] Matt Baker: Yeah, this, oh, I'm avoiding the soapbox on this, but I do want to comment on this, that you can get lost here, I think. You can get lost talking about all the different cool patterns you could apply. It's going to be microservices. We're going to use Kafka. We're going to, all these things that you're talking about, there are going to be improvements to the system. And I think that they are all siren songs pulling you away from what you really need to do. And what I think you really need to do is add the next feature to the project, and in the process of adding that feature, when you hit the pane that is the old system, try and just fix just that little piece. Then employ what Allan is talking about. You can't rewrite that whole module, but hey, if that variable name is messing with you, change that thing. No matter if that function could be extracted, do that. But do it in the pursuit of adding value to your user. I think you run a risk when you click out of that and into, we're in sort of make the system better mode, and you're not thinking about the user anymore. All I hear there is, okay, so your feedback loop is now boundless, and it ends when you decide to try and ship this thing. So I think pursuing user value, but acknowledging that along the way you're pulling weeds, and you're slowly refactoring. I think that's the approach I would advocate for without knowing the particulars of every situation.

[00:38:32] Allan Stewart: Absolutely. That's such a great method of starting to change things. It's like, you were going to be in this piece of code anyway, so change that. Don't go and find that one gnarly bit of code that you know is gross, but it turns out that it doesn't get run that often, and so improving it isn't going to help anything. Thing, which goes to what I think Dave was starting to hint at is you can start measuring. How do you know which part of the system should to change? Well, which part of the system is bad? And there's a number of ways you can start measuring it, right? It's like, oh, are you measuring performance? Are you measuring, um, cyclomatic complexity? Are you measuring, uh, you know, linting failures or something like that? There's, there's a lot of different ways that you can measure to find a place to change.

[00:39:21] Dave Adsit: I don't think that most failing projects are super worried about their lending failures right up front. Probably not. Wait, what? We're hoping to someday get to the point where our lending failures are causing us grief. But for now, maybe this 20-second response time on this critical path HTTP request might be. But I don't know, right? How many times have you gone in and somebody says, "I know what the problem is," and they start working on it, and they spend one to three to six to nine weeks on the thing. And they ship whatever it is, somehow through a miracle, and the problem persists as though they had done nothing.

[00:40:08] Matt Baker: Or just moves, right? Right. Like, when you go into these six or nine week cycles, you, you, your solution will have unintended consequences. And the longer you wait to discover those, you're just going to be right back in it.

[00:40:24] Dave Adsit: One of the things you, you have mentioned multiple times, Matt, is the feedback. And that's one of the things I'm looking for very quickly. As soon as I have CI and I've started people on the path of testing, I want to measure the active system as it's running. And so I start looking for tools like that. And sometimes that means I'm grabbing a performance analyzer that I'm running right there on a developer's workstation, in kind of a pseudo environment. A fake, you know, kind of a faked up environment that is working, not quite like production. But sometimes it means that I'm looking at things in my production system and there's a lot of ways we can get direct feedback from our production system. Just to kind of shotgun blast a bunch of them out there, if you're cloud-hosted, almost every provider is going to give you some kind of insight into what's happening on the machine. If you are running any kind of modern software toolkit, it's most likely got hooks that you can get performance information out of or it's got the capability of writing log files that you can analyze yourself after the fact. And failing that, or in addition to that, even better, you can start pushing out your own metrics to some kind of a metrics tracking tool so that you can observe things in real time. One of the things that I've done on a recent project is I went in, and one of the first things was was set up live performance tracking for every inbound HTTP request. Because people had ideas about how many requests were going on. They had ideas about how big the scope of the problem was. But nobody had any data. And one of the things that we're doing as an industry is we're moving a lot more towards data-driven decision-making. So after we had turned this on in our production environment and we started to see where the hotspots were, which requests are slow and called a lot. Well, let's focus effort on those because they'll make a big impact to a lot of people. This one's slower, but it gets executed four times a day. We can handle four slow requests a day. This one gets called more often, but it's also faster, so it's not a problem for as many people.

[00:42:56] Allan Stewart: It goes back to the law of bottlenecks. Right. Because if you go in and you say, "hey, we're just going to change a random thing." If you didn't actually hit the bottleneck, the thing that was causing it to be slow, then you won't really have changed the performance of your system. You won't really have changed. And whatever that is. Right. So that could be a performance bottleneck, or it could be a code bottleneck where it's like, "oh, well, the problem is that, you know, we've got six different styles of code. And whenever we hit this one particular style of code, it's like a mental speed bump. And we go and hit that speed bump every single day." So whatever it is that's the bottleneck, you got to find that and fix it, because fixing anything else won't improve the throughput.

[00:43:42] Dave Adsit: put. Yeah. Or the only developer who understands this algorithm just happens to be on a two week vacation, right, when one of our biggest customers needs us to make an improvement to the algorithm for some reason.

[00:43:57] Matt Baker: So a question I have kicking around in my mind as we're talking, I agree with everything that we're saying. Um, what would you say to someone who, um, is maybe entering their first, you know, big failing project that they're trying to turn around, and they're looking for that first step? We've talked about a lot of things, but I can empathize with that person, and maybe almost a paralysis that would set in. You just don't know where to start. "I can't measure without this. I can't test without that. I, you know, and everything seems to have all these prerequisites that I can't meet, and I want to do something because this house is on fire." What do you say to that person? How do you guide them through their first few steps?

[00:44:39] Dave Adsit: So my, my first response, my gut response is I would tell that person to find someone who is a mentor or someone who is in a community of professionals and ask them for help with the specific context, because there's, There's too much there for me to give a blanket answer. But if I have enough context, I can help you find that one loose thread that I can start to pull. And once we start pulling that thread, we can start to unravel this knot. But if that's not an option, then we're going to have to give some more general purpose types of solutions. And they're going to go back to first thing, start working on your team. Maybe your team doesn't have enough knowledge of testing or patterns or delivery or the programming language to solve the problem effectively. Or maybe you have all those things, but three of the members of the team hate each other because they were on competing teams in some kind of a sports type event, or, who knows, who knows why. I don't know. It could be that they were on rival, uh, guilds in a multiplayer player virtual world. I don't know. There's a lot of reasons why people might not get along. There's a lot of reasons why people might just slowly isolate on a team in an unhealthy culture. But like, if you don't know what else to fix, at least fix your team. Because if you have the right team, you can overcome any problem. And if you have the wrong team, every problem will defeat you.

[00:46:38] Allan Stewart: So I think the other thing, uh, to go back to your question, Matt, I was like, "oh, well, if we're starting to get this paralysis and feeling it, cause, oh, there's so much to do." I think that actually it doesn't have to be that hard, right? It's going to be more difficult if you haven't done it before. And so you need to look at that and, and find that mentor like Dave was talking about. But some of these things aren't that difficult to do, especially today, right? So, like, 15 years ago, or even 10 years ago, getting a CI tool up and running was kind of hard. But now a lot of them are baked into, like, a lot of the cloud provider stuff, or you can just, you know, download a Docker image that probably has one. And you can just get something going, right? So it's like, if you want to get CI going, going. It's like, "okay, can you build it? Great. Okay. Can you execute a test? Great." And then you just build on these things incrementally. And so if your problem is that code, then what is it you want to measure about the code? You know, like Dave, Dave was joking about the linting thing, maybe isn't your most important problem, but for some, some people, maybe that's it. It's like Like the code is so inconsistent that you can't, you just can't navigate it. And so if everybody gets consistent about the style of the code, maybe that helps. So you figure out what is that one thing that you're hurting, you know, and then you can get the tests running. You can get the CI running. Or maybe the thing that is hurting is that the system just falls over every so often. Well, maybe start measuring and, you know, pick a tool. Maybe it's a logging framework. Maybe it's a metrics thing like Influx or Graphite or something like that. You can put a dashboard on top of it like Grafana. And they're not actually too hard to get going these days. So getting it working the first time might take you a day or two to just, like, get something, just, like, that first heartbeat out onto that graph. But once you've done that, you've opened the door. You've unlocked the ability to start measuring things. And as soon as you're measuring things, then you have information, and you can start doing things. And CI is doing the same thing. It's just in a slightly different way. It's measuring something. Tests are measuring some form of reliability, or it's a measurement of whether the system has changed, whether it has broken. You know, linting or style rules, that they're a measure of how consistent your code, the style of your code is. And so you're starting to get feedback, and you're getting feedback, you're getting data that drives you to say, "okay, well, now that we have data, what do we do? Where is the hot spot, where is the thing that we should fix?" And then you, and then you have a way to start recovering, pulling out of the dive, the death spiral that your project has been in.

[00:49:50] Matt Baker: Yeah, it almost sounds like Um, maybe there's more than one right answer in that situation, and maybe more importantly than you know, figuring out the exact function that that is the hottest, or the exact, you know, uh, endpoint that that's getting the most beat up, maybe there's, there's something to be said for just take a step, just, just start trying. And I, I don't know which one of the steps would be valid, but I am picking up on this, you know, in our conversation, that, um, maybe there's not one right right answer. And the more important thing is to, to, to just do something. And the thing you do, keep it small so you can get the feedback quickly, and then, you know, maybe start adjusting from there. But, and I'm sure this doesn't apply to all projects, but maybe there are just some projects that just need some kind of love and give it something. And then over time, that'll pick up steam.

[00:50:51] Dave Adsit: Right. And you know, like you say, context really matters. And if you're trying to of work kind of in a context-free way, there are some things that have been shown to have a positive effect across a wide variety of projects and systems and teams. Nicole Forsgren and Gene Kim and Jez Humble put together a book called Accelerate, and it talks about healthy projects. And if you're on a sick project, what you'd like to do is become healthy. So some of the things that they talk about in there, they've got a set of key performance indicators that you can track or calculate without too much tooling, that will allow you to get a handle on some of the potential areas for improvement. And the four that they talk about as being the most indicative of success are the lead time to complete something new, the deployment frequency, like, is it hourly, daily, weekly, monthly, annually, heaven forbid, you might get to release once while you work there. What is the meantime between failures? Like, how long do you go in your production environment before you have what you would consider to be some kind of a failure? And when you do have a failure, how long does it take you to recover? So you've got those four things, the lead time, the deployment frequency, the meantime between failures, and the meantime to recovery. And if you can measure those, and then you can pick one of them to work on, you can make improvements that will have a drastic impact, positive impact in the health of your project overall.

[00:53:02] Allan Stewart: all. One of the things that I like that this kind of leads into is that we talked about a lot of technical things, you know, set up a CI pipeline, set up a, you know, a metrics dashboard, or log to your logging framework or whatever. But there's also the people aspect, right? And we touched on on that a little bit. It's like, you know, if you need tests that give you confidence that you can change code, well, your team has to be on board with that. The team has to, you know, to do that. If you need a consistent style, the team has to be supportive of that. If you're gonna change how you deploy, your team needs to be supportive of that. And so that leads into some other team-related and management-related concepts. One of the things that I have found, about five years ago, I got introduced to Lean software development. And some of the principles behind Lean just really help with some of these problems. You were talking earlier, Matt, about how sometimes the problem is that there's so much work pushed onto a team. But if you switch to a true Kanban board where the team is pulling work instead of having work pushed upon them, then that can make a huge difference. That can change the morale and the outlook of the team. If they come in and you've set work and process limits, and so you know that, "hey, we're not going to start on 17 things, but we're going to finish one or two," that can be a huge adjustment to a team and a huge benefit. If they come in and they see, "hey, our backlog of things that we need to do is not 10,000 items long, and we can't actually even enumerate them all," that helps. There are these human aspects aspects of the system that we can improve that relate into some of the Accelerate metrics that you were talking about, or relate to Lean and Agile principles that can really help turn a software project around as well.

[00:55:30] Matt Baker: Yeah. I really like that one, WIP, or work in process. This one in particular, there are contexts where it doesn't apply. Yes, but it seems to be consistently one that is violated in unhealthy projects. There's something about this one. There's something about starting up too much work. And then, you know, we, we could talk another time about all the, the effects, um, that come from that. But maybe one concrete thing that you could at least consider when you're stepping into a project like this, and it feels like you don't feel so big, is to, you know, look at your WIP. If you had a team of 10, 10 engineers working on 15 things, and you know you've got a problem, maybe bring that down to, to one or two, or three, and, and just try it out. You know, I, I just this, this particular point seems to come up again and again for me, that a lot of unhealthy teams are struggling from just too much work absolutely

[00:56:33] Allan Stewart: Absolutely. And people want to set their WIP limits based on the number of people in their team. And that's generally a good idea. But usually what they say is, "OK, well, how many people in the team plus some constant?" And what I recommend is, "well, how many people in your team divided by some constant?" So if you've got six people on your team, it's a good place to start. Don't set a WIP limit of eight so you have some buffer room. Instead, set it to three or two or even one, so that then all of a sudden you're saying, okay, we have to get a thing done before we can start the next thing. And that goes into some of these other concepts, right, that are traditionally considered technical concepts. But, like, continuous integration, if there are less things to integrate, great, you will be happier because it's, you have fewer merge conflicts, you have fewer places where somebody changed the behavior of the other class that you were depending on, because they needed to do something else for a different reason, because you were changing two things at the same time. And you start to corral those in, and then it forces you, you say, "oh well, I can't start on a new thing because of the WIP limit. So maybe I can go and work with that other person" and suddenly starting getting into these other people-related issues of, "oh, well, maybe you should be doing some pair programming." If you have six people and a WIP limit of three, well, then you can have three pairs. And they can all be working together and improving the quality of the system that they're building and trying to do less things all at the same time.

[00:58:22] Dave Adsit: Well, and I would add to it that one of the big reasons that teams end up starting too many things is because nobody has trust that they'll get it done. I mean, we talked about this earlier, but if I come up to a dev team and I demand that they start my thing, it's because I think somehow that by starting it, it will get finished, which isn't necessarily true. But you can start to correct that. You can start to repair that relationship by delivering something. Sure, it was small. The first one was probably small, but you delivered it. Now they've seen that the team can deliver it, and now you deliver another one, and now, oh, some things are starting to happen. You start delivering more and more value, you, by focusing on all of your efforts on one or two or three things at a time, you can start to build a relationship of trust where fewer and fewer people come over and demand that you start and finish this thing by this date. Or at least please start it, because I know if you start it, you'll eventually get it done, because I'm not thinking about the other 100 items that you you started this week.

[00:59:43] Matt Baker: Yeah. Yeah. I, I don't know the words to say, um, that really resonates with me, other than just saying that really resonates with me. Um, and that doesn't quite capture it. I, uh, there's a mentality there that you're commenting on. That's, that's, uh, you know, I'm, I'm reacting to pretty strongly, um, Allan, to something you said, teams adding more WIP to create a buffer. Isn't that, like, one of the big paradoxes here? You know, we say, "well, we, the way that I've seen this lately is, well, we have this priority swim lane across our boards for the high priority items. Or, like, we've extended our backlog just a little bit just so we don't forget everything we want to remember." And there's these reasons for coming up with, with more WIP equals more buffer, but I think the reverse is what is actually true. The WIP of three has more buffer than the WIP of five.

[01:00:43] Dave Adsit: The good thing about the important items in your backlog is that you will never forget to do them. So if you're struggling with team morale because of the overwhelming amount of stuff that someone somewhere in the organization committed you to do, delete the whole backlog, just every card, just trash them, because actual important ones will get remembered and will get done.

[01:01:15] Allan Stewart: And if it wasn't important, then you're just wasting time worrying about it, or, or doing a backlog grooming meeting where you talk about, you know, the t-shirt size of how much work is that going to be? Because if you're never going to do that work, who cares? Who cares? So that goes back to me for, that goes back to queuing theory, right? So we went to a conference one time, and one of my coworkers said that he was in a meeting, you know, one of the sessions, and the speaker says, "you know, nobody cares. Nobody gets fired up for queuing theory." And I thought about that a lot, because the more that I think about it, it's like, I think I get fired up for queuing theory. I hear you, man.

[01:02:02] Dave Adsit: Allan, can you tell us how many hours you have logged on Factorio? Oh, man.

[01:02:08] Allan Stewart: There are a lot of hours of moving plates and inserters and learning about queuing theory. I think, though, in Factorio, it's kind of opposite. Because in Factorio, I want all the buffers and I want to back up everything all the way back to the beginning of my factory. But in software development, I do not want to do that. And that's where your WIP limits are really small. It's like, I just want you to go in and get a thing done. What is the most important thing that you could do right now? Do that, ship it to production, do the next one. Don't do seven things, because queuing theory, when we look at the mathematics about it and when we model how people actually behave in software development, moment, we start to see the cues forming between each other, the cues of, "well, I was waiting for so-and-so to do that pull request, to give me feedback on my thing. And so I started yet another different thing. And so now I've got more stuff, and I'm busy doing that thing when they give me the feedback that I was looking for, or when they submit their pull request to get the feedback from me." And we start building up these cues. And when we model it out with queuing theory, we see that, you know, they're, I can't remember what the term is. It's, you know, they're non-deterministic Markovian chains or something like that. And reducing WIP, reducing WIP gives you better throughput, because that's what matters. It doesn't matter how many things you started today. How many things did you finish today? That's, that's what matters.

[01:03:49] Matt Baker: Yeah. It's the, to control throughput, you control input, you know, a full freeway doesn't move. I don't know how else to say it other than that.

[01:03:59] Allan Stewart: Oh, man, but the taxpayers, Matt, they're getting so much return on investment for that freeway because every bit of freeway is being used.

[01:04:08] Dave Adsit: There is a car parked on every square foot.

[01:04:13] Matt Baker: Success.

[01:04:15] Dave Adsit: One of the things that comes up often in my executive team meetings is, what is the one thing that we can do that would make everything else we're doing irrelevant or obsolete? And that really resonates with me because I want to be focusing on the one top priority item that will make a difference in the business. Half of the developers that I interview you mention first thing that what they want is the ability to make an impact in the business. I think we all have that desire to do something that really matters. And grooming a backlog never provided any customer with direct value.

[01:05:07] Allan Stewart: you. So related to WIP, work and process limits, and related to this idea of, you know, what is the thing that matters? There's a really great book out there called Making Work Visible. And so before, we talked about, "well, you know, what do you do, if you're a person who's in one of these failing projects and you're not sure what to do?" Before we answered that with kind of like these technical answers. But if you think it has something to do with this process of how your team executes work and more of the people things, I really recommend this book, Making Work Visible. And it's all about helping people see what is actually going on. And then you can start identifying the problems and fixing it. Because sometimes it becomes blindingly obvious as soon as you just make it visible. And you're like, "oh, well, how come nothing's getting done?" Well, maybe there's a bunch of things that are getting done and it's just in secret. It's like the shadow IT kind of stuff. It's like, well, things are happening, happening, just nobody knows that it's happening. Or, um, you know, some, you know, this project never gets off the ground. Um, and it's like, "oh, well, our key person is always busy." And it's like, "oh, what are they busy doing?" Well, you can find out by making that work visible. It's like, "oh, they're getting a bunch of requests from another part of the business, and we didn't realize it, or there's some other problem that is occurring, or, or maybe it's just that the team has started 16 different tasks, and, and you had no idea that everybody was so busy doing all this stuff." You make the work visible, and suddenly it becomes very obvious where the problems are. I guess the, the question is

[01:07:14] Dave Adsit: is how does all of this relate to crafting software anyway?

[01:07:24] Allan Stewart: For me, I think a lot of it goes back to the discussion of the Software Craftsmanship Manifesto. Are you steadily delivering value? If you have a CI-CD pipeline that lets you deploy at a regular interval, then you can measure that. You can say, "yeah, we delivered value today." If you've made work visible, you can tell, "well, how many things did we deploy to production? How long is it taking us to deploy things? How often do we have failures?" You start building that up. So it's about that delivering value. It's also a lot about about professionalism in my mind. It's like, do you know what the system is doing? Right? Writing a bunch of code and throwing it over the wall to QA to test, and then you throw it over the wall to operations to run in production, and I don't have to care, because I'm a developer and that's somebody else's problem? That's not professional. You don't know what your code is doing. If you don't have tests, you don't know whether you've written bugs or not. And so to me, all of these things that help you rescue the project, these different techniques, testing, measurement, people changes, collaborative coding, they're all ways to do better professional development. You're making better software, delivering better value.

[01:09:02] Matt Baker: Yeah, for me, I think it comes back to, I don't know how to restate what Allan just said, just, you know, being professional, looking out for the people that are paying you money. I can't help but, my mind is just drifting towards how you do that in this scenario. So I think, I guess, my answer to your question, Dave, crafting software in the context that we're talking about it means to be professional. And I believe that being professional means honoring the agreement you've, you've created with someone. They're giving you money, and you, you in exchange offer them a thing. And so then, you know, it's incumbent upon you to do the thing as best as you can. And when you get into these projects that are failing, I think it's really important to anchor your feedback loops, you know, to that agreement and to that customer, to that value add. So that's the thing that's really kind of coming up in my mind as we've been talking this past little bit here is, studying this stuff, this idea of crafting software, has taught me to really anchor my efforts to where I'm adding value and then let that guide me, especially through these failing projects.

[01:10:25] Dave Adsit: Yeah, and I would add that I have rarely seen a failing project that is actually adding value to the customers. And it's not to say that a well-crafted software project will never fail or that a poorly crafted project will never succeed. I think that it is possible to have both of those. I do believe that you are less likely to find yourself on a very sick project when you are paying attention to your profession and putting effort into crafting your software well. So if that is one of your goals, then it will motivate you to find solutions to some of the problems that are causing your project to fail. Because you recognize that it isn't somebody else's problem to solve. It's my problem to solve because this is my project. And as a professional, I must craft this project, craft this software effectively for my customers. Whatever that means, whatever I have to do to get there.

[01:11:45] Allan Stewart: And with that, I think we will wrap up and recommend all of our listeners go and join a community of professionals by attending a software crafters group or meetup in your area. You can find that person that will help give you some more feedback about how you can rescue your failing project. You can get some additional pointers into how do you implement some of these things that we've been talking about today. And if you happen to be in Utah, the Utah SC group at utahsc.org meets the first Wednesday of each month. In Draper, Utah. And we'd love to see you there.