Crafting Code Podcast

[00:00:00] Allan Stewart: Welcome to episode two of the Crafting Code podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately I've been thinking about how to make software developers resilient to distractions.

[00:00:16] Dave Adsit: I'm Dave Adsit, CTO, and recently I've been thinking a lot about the potential for using bottlenecks to create value in a system.

[00:00:25] Matt Baker: I'm Matt Baker, software architect. Lately I've been thinking about forming teams around data domains.

[00:00:33] Allan Stewart: And our topic for this episode is rescuing a failing project. So, to get things started, what is a failing project? How do you recognize one? How do you know if you're in a failing project?

[00:00:49] Matt Baker: Oh, bad answer, but I think it applies. You know. You just know. You know when you dread going into work and when you leave work defeated. You know when the release cycle is 27 hours or bugs galore. I feel like I could go on and on here, and I'm watching the grimacing faces of my co-hosts. It's kind of confirmation that I'm hitting some nerves.

[00:01:15] Allan Stewart: So this is the Justice Potter Stewart version of I know it when I see it?

[00:01:20] Dave Adsit: Yeah, it is. I think that it's... It's like being sick in that there are a lot of symptoms, but you feel it all the way through you. Some of the symptoms I look for are a slowdown in delivery of value. I look for increased demand for time and investment and giving to the company. I hesitate to... I hesitate to use the term death march because it's overused in a lot of contexts, but man, I've lived them. And when people are asking you to put in longer and longer days or when the manager more subtly just sits there working from 7 a.m. until 9 p.m.

[00:02:16] Matt Baker: Yeah, I've worked a 36-hour shift. I've slept on server room floors. So...

[00:02:22] Dave Adsit: Yeah. Yeah. You wrote really good code the last 24 hours of that shift, right?

[00:02:29] Matt Baker: That was great.

[00:02:29] Dave Adsit: Best code I've ever written. Best code. The parts where I was lucid, man, you wouldn't believe.

[00:02:37] Matt Baker: Another thing I think that occurs is deadlines. I believe that there's some relationship between a failing software project and the increased demand for deadlines. Tell me when it's going to be done. And I also believe that the death rattle is the give me two weeks. I just need two weeks. The perpetual. I just need two weeks.

[00:02:56] Dave Adsit: I think you're right about the deadlines for sure. People start asking for deadlines or start demanding code be done by a point in time because they don't believe that otherwise they're going to get it at all. Right? That's a pretty good sign that your project is failing. Oh, man. Allan, what do you think?

[00:03:19] Allan Stewart: Well, I think just the fun aspect of it. Right? If you're not enjoying going into work anymore, that may be related to the fact that the software project is failing. And so you're not tackling interesting problems. You're not feeling that delivery of value and saying, hey, I worked on something and customers are using it now. And whatever that is that's stopping that, whatever is lengthening out that deployment timeline, whatever it is that, you know, maybe it's really bad code that's slowing you down. Maybe it's the team. Maybe it's something else. You start to feel that in just the enjoyment that you're having with the project.

[00:04:09] Dave Adsit: I was just thinking that one of the ways I know a project that I'm working on is failing is when I go home and the people who are there waiting for me tell me that my project is failing. Because I bring that baggage with me and they get to experience it too. And man, that's worse. That feels worse even than just letting the project fail. Like bringing the baggage of failure home and having whoever it is that you meet at home. Know that things are not going well and haven't been for some period of time.

[00:04:50] Matt Baker: Yeah, I think there's this amazing thing that happens when we sleep. In this case, and it resets, maybe at least my temperament. So when I'm in a failing project, oftentimes I show up at the end of the day, I show up at home just defeated. I don't know what to do. I can't see a direction. I just feel like it, you know, all hope is lost, but I calm down. I relax. I spend some time with people that I love and then in the morning, I'm ready to go again. What I've noticed is that when I'm in a failing software project, that's a daily thing for me. I, in the evening, I'm defeated in the morning. I'm rejuvenated. I'm ready to go, but without fail, eight hours later, I'm back to that defeated state.

[00:05:31] Allan Stewart: So let's talk about some of the different aspects of parts of the system that might be failing. So one of them could be code, right? And I think especially in the software crafting community, we talk a lot about like the big ball of mud, you know, all this, this terrible code that gets written and it, and it's not a value judgment against the people who are writing it, but it's just this, these different approaches to writing software that are very additive tacking it on. Just, just get it working, band-aid it, slap it together. And before you know it, you have this, this code base that's just really hard to maintain. You know, it's buggy. It's hard to change. It's, what other aspects of code have you noticed? Like, like, how do you know? So Matt, if you went into a software project that, you know, you had never seen before a new code base, how would you know if that code was a mess or not?

[00:06:39] Matt Baker: Yeah, you know, I think we all develop little heuristics to, to answer this question. And some of those are going to be preferential. Let's see some of the things that I look for lack of tests. I look for that. I'm looking for size of individuals. I'm looking for individual files or size of functions. I'm looking for degrees of coupling. Maybe some, those are the ones that come to mind immediately.

[00:07:04] Dave Adsit: I start looking at a software system from the outside. I want to know how many bugs have been reported, but not resolved. Is the product slow? And what does that mean? Like, who knows what it means to be slow? Different systems have different slowness characteristics, right? But if you ask people, is it slow? They will have an answer. I want to know how hard is it to change the, for the last feature that you developed? How long did it take? How long does it take on average? I want to know how often do you have failures in production and how long do they last and who fixes them and how did they know what to do? And then, one of the big ones that's typically hard to measure or impossible on a failing project is, are your customers happy? you might have a hint at it, but you might not even have customers yet. You may be failing before you even get there.

[00:08:05] Allan Stewart: So it sounds like there's a lot of aspects to this. It's that it's not just the code, right? There are people involved with this. There are people that are writing the code. So, is it the team? Is the team the problem? What signs do you look for to see whether there is a problem with a team that is causing a failing project?

[00:08:27] Dave Adsit: So, in most management books that I've read, they always say that every problem is a people problem. And I think to some extent that's true. And to some extent that's the only hammer I have is people. So I'm going to pound on them as hard as I can. I mean, something about nails. But, there is some truth to the fact that, you have to have the right team. Does your team have skills? Have you given them the training to have the skills? Is the team aligned? Are they working together towards a purpose? Are they working against each other? Do they collaborate? Or does everybody sit facing the wall with their headphones on and they talk at lunch for two minutes? Maybe, if at all. Like, does the team actually get stuff done? One of the big questions, one of the easy ones to feel, but not necessarily to measure is the morale of a team.

[00:09:28] Matt Baker: Yeah, you know, what you said about being able to feel the morale really speaks to me. I was on a team one time where we decided to do a leap year kata. And if you've done that kata, you know that it's not a huge ask and you should be able to solve it relatively quickly. And we took an hour to try and do the kata. We didn't finish it. We didn't really get far at all. And in fact, we spent the entire time, arguing. We just had a lot of communication issues. We weren't able to move forward because no one could agree to anything. So when we were in that mob, we did it, you know, with mob programming, you definitely could feel the low morale.

[00:10:05] Allan Stewart: Well, one of the things that that makes me think about is that maybe what's going on with a team is not so much about the individuals. Because I've worked with a lot of teams over the years and almost everybody that I work with, I like. As far as as working with people, but not every team has had the same skills. Not every team has been aligned around the same practices. So it might not be the team like it might not be like the people. And so when we talk about teams, I think it's important to recognize it's like it's not a value judgment against any person. Occasionally, occasionally, there's one person that is just hard to get along with. You're a person. personalities, clash, you just can't stand being around a particular person. And in that case, yeah, okay, maybe it's maybe it's a person maybe it's the one person who is just a jerk and there's no two ways around it, but most of the time, that's not the case. Most of the time, there's something around the team, whether it's like practices or management or something else that is contributing to the failing software project, more than the individual team members and their software coding skills.

[00:11:24] Dave Adsit: Yeah, there's definitely a lot of things in the system, which is basically, like you say, that's everything around the team that's delivering the software. There's something in that system that is not working. And that could be so many different possibilities. It could be things like the team room. It could be the team's practices that they've chosen or been chosen for them. Previously, we talked about how the CTO at one of your companies, Allan, told everyone they were going to TDD, but not how to do that, right? That could be a problem. It could be, how are decisions made? Are decisions actually ever made? Are we in a constant state of flux and turmoil and we don't ever have a decision that we can work toward? There's so many things. Like, how do you interact with other parts of the company if you do it all? Again, a lot of these problems come from not having the communication pipelines. Basically, anything around a team that affects their motivation and influences their ability to be successful could be part of the problem for that team.

[00:12:41] Matt Baker: Yeah, this really speaks to me, and I want to add just some more concrete examples. You know? Oftentimes, you can find yourself in organizations where the team does not choose the rate of input of new work. So they're just sort of given new work, and if it's really bad, they're given new work without any consideration for current work. So, you know, if the team's already, let's call it they have this utilization. If the team's 100% utilized, meaning they're doing as much as they can do, and you add more work, it's not going to do what you think it's going to do. You're going to slow everything down further, but on this subject, you know, what we're talking about, which is how do system things affect team morale, that sucks for team morale. I mean, you're already at capacity. Now you're going to come work weekends. So I believe that that's it. And I also think that there's another concrete that really speaks to me lately is trust. If the team doesn't feel trusted to be able to go execute on the problem that they're trying to solve, morale is just going to be low. And I don't know all the ways that's going to manifest itself, but having been on teams that were not trusted, I know that it does. Well, I shouldn't say in my experience, it's definitely affected the morality.

[00:14:01] Dave Adsit: I worked on one team where we had so many items in the backlog that we finally decided to do the mega grooming or whatever we called it at that point, where we quickly saw, we analyzed every card in the backlog without much context beyond what it said in the title. And then somebody did some back of the envelope math and we came up with an approximate completion deadline or completion date for the items we had so far of about five years in the future. And we all looked around at each other and nobody felt like they were going to be there for five years. Morale got real bad real quick. And then we started going through and eliminating some items, right? We started trying to do some things to save that failing project. So I could talk about some of the things that I've done, but what about you guys? What are some things that you've done once you've identified a failing project? What do you do next? Like everybody's been on one, right? I'm sure people are like, yeah, yeah, I get it. Now, next. Now what? What are you telling me? What are you telling me? Oh, so you want some answers.

[00:15:21] Allan Stewart: At least you want to know how to rescue a failing project. Well, for me, one of the big things that I have found works wonders is getting continuous integration and or a continuous delivery pipeline set up. Just that ability to get code off of an individual developer's machine that is in your trunk or master branch and test it, run all the tests that you have, which maybe is just the one that you just added just now, but it runs the tests and you can deploy it. That's such a huge thing. And now it's to the point for me that if I'm starting a new project, that's one of the very first things I do. It's like, okay, we're gonna build a thing. Oh yeah, maybe it's gonna have a database. Maybe it's gonna have, you know, a database. Maybe it's gonna have an API or whatever. It's like, okay, great. There's gonna be a thing. I don't even care. What I'm gonna do is deploy Hello World with a test and make sure that it does continuous integration and we deploy it into our staging environment and that we have it set up so that we could deploy it into production. And sometimes I do now. I'll deploy it all the way into production because nobody's got that URL because this is a brand new thing. Just go ahead. Write on out to production. And then every other thing you do, you have that ability to get things out. Because with these projects that are dying, that are failing, I feel like there's no momentum. There's no way that you get stuff into production. And putting that change in place is huge.

[00:17:08] Matt Baker: Yeah, it sounds like feedback. You've recognized that a project needs feedback, and it almost seems like there's this dopamine hit that occurs when you experience that feedback loop. When I ship something to production, especially when I see someone use it, that's incredible.

[00:17:25] Dave Adsit: Right, even if it's not the thing that they really, really, really wanted, even if it's just a hint at that, something that they can see and say, oh, by the way, now that I see this, I have an idea for something that would get me some value quicker or easier. One of the things I was, going to make a snarky comment about what if it's really hard to deploy my system, then how do I do a CI pipeline? But like they say in software, if it hurts, do it more and automate it more, right? So, yeah.

[00:18:03] Allan Stewart: If it's hard to deploy, then you probably want it automated. Because more often than not, I find that projects that are suffering in the deployment phase, have the problem because they are trying to get code out to production and there's all these manual steps and it's like, oh, well, we forgot to run the one script. Oh, yeah, did you remember to reset the server processor or something that a human forgot to do?

[00:18:35] Dave Adsit: If you are in the team room or the war room, worse, and you are trying to do a release and it sounds the same as, when your team is sitting around with their phones playing space team, you have done it wrong. My teams currently deploy once or twice a day, no less frequently than that. And sometimes the deploys are just small, but we know that they did something that is potential value for someone.

[00:19:11] Matt Baker: Yeah. You mentioned war room. I want to offer a positive take on that for a minute. One of the things that I think can be very effective, depending on the situation, when you walk into a failing software project is to take all the people that are working on the project and go sit in a room and just kind of ask a simple question. What's the first thing we need this thing to do? Maybe it's allow users to sign in, or maybe it's allow someone to send an email, whatever it is. Then as the group, solve that problem. Solve the first one. And all the way, code it, ship it, test it, design it, whatever needs to happen, but do it as a group. And you do that a few times, and something I've noticed is that there's this reorientation that occurs where the team stops saying, well, it's Matt's solution versus Dave's solution. And then it's, it seems to change to it's Matt and Dave versus the problem. And we get into this mode of, all right, we need to get a user to sign in. We need to get a user to send an email. And you kind of go through the list, but there's, I don't know exactly what it is, but there's something kind of magical about war rooming that. Now, war rooms are typically in response to firefights, and that's not, I'm not necessarily saying that, but what I'm saying is get the necessary party, people, rather, go into a room and just start working on it. And I think you might see the morale improve.

[00:20:46] Allan Stewart: Yeah, the positive aspect of a war room, I think, is that collaboration, right? It's like, but the crazy thing is that we tend to do war room at the end when everything is on fire, everything's broken, everyone is sad. That's when we say, well, I guess maybe we should work together instead of working all individually.

[00:21:06] Dave Adsit: But there's no way you could ever work together all the time. That would be far too hard. That's far too effective.

[00:21:12] Allan Stewart: Oh, but Dave, don't get me started talking about that topic, because today we're talking about rescuing a failing project.

[00:21:20] Dave Adsit: And I think one of the things that is interesting about a failing project is that typically it's failing for one of those three reasons. It's the code, it's the team, or it's the system. If it's the system, it's gotta be pushed up to a management level. Managers have to recognize the problem and start working it. They should do so with input, right? If it's the code, we know what to do. There's no better way to improve the quality of your code than to start testing it with automated tests. And there's no better way to improve the quality of your team than to start collaborating regularly, by which I mean not pull requests, I mean pair programming or mob programming. Get people to work together on the same level. Same problem, not adjacent problems.

[00:22:14] Matt Baker: Yeah, and I think you can grow out of this. So maybe you start with this, you know, everyone piling in the same room and work for a little while. And maybe from there you graduate into a remote PR thing or something else, right, where the collaboration is a little bit less to accommodate some other variable in the situation. I think, you know, that can be fine. But I do think it's important to start together. And figure out your ways of working and then start to, you know, split up if that's what you intend to do. But I tend to agree with you, Dave. The further you get from that face-to-face, hands on the same keyboard collaboration, the harder it gets.

[00:22:58] Dave Adsit: It's too easy. I've read somewhere that the half-life of trust is something like six weeks. It's too easy to blame the other guy on the team if you're not working with him. If the two of you are working together, are sitting together at one computer, then you're working against the problem. And that does, it builds something that is essential to rescuing that failed project. If we're working together against the problem, then we have a chance. If we're working against each other, we have no chance.

[00:23:37] Allan Stewart: Yeah. And going back to continuous integration, I really like that as an enabler, right? Because continuous integration, especially when we think about CI, not as just a set of tools, right? I'm not just talking about Jenkins or Bamboo or whatever your favorite, you know, favorite CI tool is, but continuous integration, the concept is about, hey, get all of your code together more often. And when you're doing that, then it naturally leads you no matter how exactly you're collaborating. I'm a big proponent of pairing. I'm a big proponent of mobbing. And you're getting that. When you mob programs, you are creating continuous integration because now you don't need a server because everybody's writing the code at the same computer and it's all integrated and you can see whether it's working or not. So it enables you to do those things. And if code is the problem, then it enables you to have tests. Because if you have tests but not continuous integration, then most likely what happens is that only the one person runs the tests and everybody else breaks the tests.

[00:24:58] Dave Adsit: So you gotta get into that pattern. That's a good recipe for not running the tests anymore.

[00:25:03] Allan Stewart: Yeah, then you don't run the tests anymore and then you comment out all the code and then you just don't have tests anymore.

[00:25:09] Dave Adsit: I've worked on a project where, you know, I've done a lot of stuff, and it had all the tests at one point, and they were just excluded from the build file because they started failing. And that just made it so much easier to keep writing more and more code faster and faster until all of a sudden it made it impossible to continue writing any code.

[00:25:28] Matt Baker: Yeah, who wants blinky tests? So if you get a blinky test, just delete it.

[00:25:31] Dave Adsit: Just delete it. In fact, just delete the whole test suite. The hard part is that once you stop running your tests, you can never start again until you start over. Yeah, you basically have to start over. But the tests give you confidence. And they, once again, improve that feedback cycle. Yeah. That feedback loop is even shorter with tests than it is just with CI.

[00:25:53] Allan Stewart: And then all of a sudden you feel like, oh, I can change code. Because in a lot of projects that are headed downhill, in my experience, the developers are scared of changing the code. And so it's like, well, instead of changing that bit of code, I'm just going to do it. I'm just going to add a new different code over here. And so then it grows. It starts to spread almost like a cancer throughout the system. It's like, oh, we're just band-aiding all these different lumps of code together.

[00:26:23] Dave Adsit: But for what other purpose am I going to use switch statements?

[00:26:28] Allan Stewart: That's not what they meant by open-close principle, Dave. Oh.

[00:26:32] Dave Adsit: We'll have to have a conversation about that at some point.

[00:26:35] Matt Baker: No, I think there's truth to that, though. Most software projects, there's a module. There's a set of modules that are known to be kind of the don't touch that module. That one, so-and-so is the only person that knows how to work on that module. And to Allan's point, it's almost like that dirty module just perpetuates itself. Because there's some precedent that's set in that module to comment on the rest of the system. And people just continue that. I worked on one piece of software that someone had... I had written their own ORM, their own query language, the whole thing. And it was used in some core functionality. And if you coupled to it, you had to also use it. So, similar to that saying, given a barrel of water and a drop of urine, you have a barrel of urine. That's what it was. It just spreads and left, you know, without interference, it'll just, you know, your whole code base is now based on a custom ORM. And, you know, good luck. That sucks. When that thing gets to be a few hundred engineers and a million lines of code, then it's rather unwieldy.

[00:27:52] Dave Adsit: But it does keep people employed working on that custom ORM all the time. Imagine the value. Wait, that's not value.

[00:28:02] Matt Baker: I mean, it's a value. Right.

[00:28:05] Allan Stewart: The other thing that that confidence that the tests bring in for me, one of the big things, there is your ability to refactor code. And if nothing else, as soon as you can start changing the names of things, you have just just enough confidence now between your tooling, between your testing, that you can rename something. Then all of a sudden you can go in and say, okay, what even is this code? Like, I don't even, I can't even understand what's going on here. And so, you think about it for a little bit and you figure out what it is and then you change the name of it. You say, okay, now you have a name. Now this other thing has, this method has a name. Now I've extracted a method and it has a name. And all of a sudden you start building up or you find the thing. It's like, oh, that says user ID, but it's actually the administrator user ID. And now all of a sudden you have more context and you can understand, you can understand better. And you don't have to think about a hundred lines of code. You can just worry about a few lines of code. And suddenly your, your system starts to heal itself. It's like the, uh, you know, putting new coral in to a dead coral reef and it starts to grow again and, and heal.

[00:29:31] Dave Adsit: Yeah, it's pretty amazing what you can do once you've built some amount of confidence around changing your code and it starts to snowmobile. The more of your code base you put tests around. So kind of going, kind of going back to the idea of, of crafting, uh, my family has always done woodworking. My dad was a contractor and so we always did that stuff. And so one of the things that for me is important is when you're working on a piece of wood, you make a jig or you make some, or you use some clamps or you use something to hold it together. While you're making a change while you're attaching one piece to another or while you're cutting one piece off from another, you want it to stay stable. And that to me is one of the big advantages of starting to add those tests. You've added braces and clamps and support to the code so that you can change it because you know it needs to be changed. It's already, the code is a mess and now we would need to start fixing it. So adding in testing and there's a lot of techniques for that and I don't really, I don't know that we really, really have time to dig into all the possible techniques, but if you find yourself in this situation, there's a lot of things you should research like test driven development, effective unit tests, good unit tests, things like contract driven testing, acceptance testing, UI testing. Um, there's just so many different options. Think about how you could white box test it or black box test it, you can create some of those clamps and some of those jigs that are holding the, your precious project in place so that it will be safe while you are busy ripping some of its guts out and replacing them with much, much better guts.

[00:31:29] Matt Baker: Yeah, I'm now thinking about the refactor versus rewrite debate and I, it's interesting to me that I believe that what we've been talking about is a refactor, you know, um, refactoring is a refactor. And for me, one thing that's important when I go into a project like this is to remember that it's just going to take a lot of time. It's going to, it was a lot of little decisions that got you there and it's going to be a lot of little decisions that get you out of there and it gets further complicated by the fact that you have to try and balance typically anyway, the continual delivery of value with this refactoring. You can't just shut down the warehouse for a year and rebuild the thing and then, and then flip it back on and go back to work. You can't do it. So, it's hard and it's a slog and I, for me, it's important to remind myself to settle in and, and Allan, like you've been saying, get that one thing under test, do that one rename and then using your CICD pipeline, get it out, get it shipped and then just get into that flow and it's going to take a while, but you know, you'll start to see the progress and like Dave said, it does snowball, um, you know, a month into that kind of work, you'll start to, to, to really start to reap the benefits of it.

[00:32:41] Allan Stewart: Yeah. And I think you, you mentioned refactor versus rewrite and more and more. I find that with a lot of projects, you actually end up needing both. But the problem is when we say rewrite, what we usually mean is, well, let's throw all of this garbage away because I'm just frustrated with it. I can't figure out what to do about it. And so let's just throw away everything. And the problem is that you can, you can never catch up. From that, right? It's the, it's the whole tiger team question. It's like, oh, well, we're going to start building all the, the new, the new code from, from scratch and you just can't catch up to all the features that the other thing did. But if you cut it down small enough and you say, well, I'm just going to rewrite this one little piece, it's just this one method or just this one thing. Sometimes that's easier than refactoring. But refactoring requires a lot of work. It requires tests to do it properly. Otherwise you're just changing things. And so sometimes it's easier to figure out, Hey, what, what is this intended to do? Let's do that instead. Let's write the littles piece that does that thing and then we'll insert it into the project. And, and so ultimately I think you need both. Sometimes you refactor, sometimes you rewrite, but the key is that you're just rewriting really small things.

[00:34:05] Dave Adsit: I would completely agree with that. I, anytime I've been on a group that has discussed the rewrite, the question that always comes to mind for me is, but we're the same people that made the first mess. How do we know that we're going to do better this time around since we've forgotten all those little decisions that we've been making for the last two years? I'm always really hesitant about rewriting in the large because we are the same people. We may have learned a little bit, a little bit more, but we're likely to make the same mess again. But if we refactor, we give ourselves an opportunity to make safe, repeatable, reversible, incremental improvement in the system that we have, which enables us to learn and improve the system at the same time without risking the business by doing the pull the ripcord full stop. We're not going to work on this until we've rewritten everything.

[00:35:06] Matt Baker: And I think that's hard. You know, I think it's hard to rewrite a system a little bit at a time and it requires a certain set of, you know, I think it requires some skills or a strategy or two. Some of the things that are coming to mind, you need things like feature flags to enable like maybe parallel implementations for a while. we could talk about the techniques. But yes, I agree with you, David. And I also think that a team that sets out to rewrite something that they wrote originally will write it better. The next time, but I just don't think it's going to deliver on what they think. You know, when we rewrite it this time, all of the problems will be solved and you're going to get some of them. The ones that are most painful. Probably you'll come up with solution and then that solution will have, you know, a new set of problems that you didn't expect. But the big risk there for me is just the length of the feedback loop. You know, if you set out to go rewrite a thing for a year, you're going to think it's great until you try to deliver it. And so, you know, in the presence of that ambiguity, I think you need to tighten up those feedback loops, which is something Allan's been saying.

[00:36:13] Allan Stewart: Yeah. I think there's a lot of discipline involved too, right? It's like, are you disciplined enough that you can write a small piece of code or if you or change a small piece of code? And what is your discipline about writing code in general in the project? Like, I mean, at some point you got to this mess. What have you changed about how you write? Right?

[00:36:42] Dave Adsit: And one of the things, one of the things that comes to mind for me is how do I even know where to start? What? What tells me which pieces are most important? Is it just my gut or can I do something to find out what the first piece is? And then the second piece that I should tackle?

[00:37:03] Matt Baker: Yeah, this. Oh, oh, I, I'm avoiding the soapbox on this one. This, but I do want to comment on this that you can get lost here. I think you can get lost talking about all the different cool patterns. You could apply. It's going to be microservices. We're going to use Kafka. We're going to all these things that you're talking about. There are going to be improvements to the system. And I think that they are all siren songs pulling you away from what you really need to do. And what I think you really need to do is add the next feature to the project. And in the process of adding that feature, when you hit the pain, that is the old system, try and just fix just that little piece, then employee. What Allan is talking about, like you can't rewrite that whole module. But hey, if that variable name is messing with you, change that thing matter. If that function could be extracted, do that, but but do it in the pursuit of adding value to your user. I think you run a risk when you click out of that and into we're in sort of make the system better. Mode and you're not thinking about the user anymore. All I hear there is okay. So your feedback loop is now boundless and it ends when you decide to try and ship this thing. So I think pursuing user value, but acknowledging that along the way you're pulling weeds and you're, you know, you're slowly refactoring. I think that's the that's the approach I would advocate for without, you know, knowing the particulars of every situation.

[00:38:32] Allan Stewart: Absolutely. That's such a great method of starting to change. Things like you were going to be in this piece of code anyway. So change that don't go and find that one gnarly bit of code that you just, you know, is gross, but it turns out that it doesn't get run that often. And so improving it isn't going to help anything which goes to what I think Dave was starting to hint at is you can start measuring. How do you know which part of the system should to change? Well, which part of the system is bad and there's a number of ways. You can start measuring it, right? It's like, oh, are you measuring performance? Are you measuring cyclomatic complexity? Are you measuring, you know, linting failures or something like that? There's a lot of different ways that you can measure to find a place to change.

[00:39:21] Dave Adsit: I don't think that most failing projects are super worried about their lending failures right up front. Probably not.

[00:39:28] Matt Baker: Wait, what?

[00:39:32] Dave Adsit: We're hoping to someday get to the point where our lending failures are causing. Us grief. But for now, maybe this 22nd response time on this critical path, HTTP request might be, but I don't know, right? How many times have you gone in and somebody says, I know what the problem is and they start working on it and they spend one to three to six to nine weeks on the thing and they ship whatever it is somehow through a miracle and the problem persists. As though they had done nothing.

[00:40:08] Matt Baker: Or just moves, right? Like when you go into these six or nine weeks cycles, you, you, your solution will have unintended consequences. And the longer you wait to discover those, you're just going to be right back in it.

[00:40:24] Dave Adsit: One of the things you have mentioned multiple times, Matt, is the feedback. And that's one of the things I'm looking for very quickly. As soon as I have CI and I've started people in the path of testing, I want to measure the active system as it's running and start. So I start looking for tools like that. And sometimes that means I'm grabbing a performance analyzer that I'm running right there on a developer's workstation in kind of a pseudo environment, a fake, you know, kind of a faked up environment that is working, not, not quite like production. But sometimes it means that I'm looking at things in my production system. And there's a lot of ways we can get direct feedback from our production system. Just to kind of shotgun blast a bunch of them out there. There's if you're, if you're cloud hosted, almost every provider is going to give you some kind of insight into what's happening on the machine. If you are running any kind of modern software tool toolkit, it's most likely got hooks that you can get performance. Information out of, or it's got the capability of writing log files that you can analyze yourself after the fact. And failing that, or in addition to that, even better, you can start pushing out your own metrics to some kind of a metrics tracking tool so that you can observe things in real time. One of the things that I've done on a recent project is I went in and one of the first things was set up live performance, tracking for every inbound HTTP request. Because people had ideas about how many requests were going on. They had ideas about how big the scope of the problem was, but nobody had any data. And one of the things that we're doing is an industry is we're moving a lot more towards data-driven decision-making. So after we had turned this on in our production environment, we started to see where the hotspots were, which requests are, slow and called a lot. Well, let's focus effort on those because they'll make a big impact to a lot of people. This one's slower, but it gets executed four times a day. We can handle four slow requests a day. This one gets called more often, but it's also faster. So it's not a problem for as many people.

[00:42:56] Allan Stewart: It goes back to the law of bottlenecks, right? Because if you go in and you say, hey, we're just going to change a random thing. If you didn't actually hit the bottleneck, the thing that was causing it to be slow, then you won't really have changed the performance of your system. You won't really have changed. And whatever that is, right? So that could be a performance bottleneck, or it could be a code bottleneck where it's like, oh, well, the problem is that, you know, we've got six different styles of code. And whenever we hit this one particular style of code, it's like a mental speed bump. And we go and hit that, we hit that speed bump every single day. So whatever it is, that's the bottleneck. You got to find that and fix it because fixing anything else won't improve the throughput. Yeah,

[00:43:44] Dave Adsit: or the only developer who understands this algorithm just happens to be on a two-week vacation, right? When one of our biggest customers needs us to make an improvement to the algorithm for some reason.

[00:43:57] Matt Baker: So a question I have kicking around in my mind as we're talking, I agree with everything that we're saying. What would you say to someone who is maybe entering their first, you know, big failing project that they're trying to turn around and they're looking for that first step? We've talked about a lot of things, but I can empathize with that person and maybe almost a paralysis that would set in. You just don't know where to start. I can't measure without this. I can't test without that. I, you know, and everything seems to have all these prerequisites that I can't meet and I want to do something. Because this house is on fire. What do you say to that person? How do you guide them through their first few steps?

[00:44:39] Dave Adsit: So my first response, my gut response is I would tell that person to find someone who is a mentor or someone who is in a community of professionals and ask them for help with the specific context because there's there's there's too much there for me to get. I can't give a blanket answer. But if I have enough context, I can help you find that one loose thread that I can start to pull. And once we start pulling that thread, we can start to unravel this knot. But if if that's not an option, then we're going to have to give some more general purpose types of solutions. And they're going to go back to first thing, start working on your team. Maybe your team doesn't have enough knowledge of testing or patterns or delivery or the programming language to solve the problem effectively. Or maybe you have all those things, but three of the members of the team hate each other because they were on competing teams in some kind of a sports type event. Or who knows? Who knows why? I don't know. It could be that they were on rival guilds and multiplayer virtual world. I don't know. So a lot of reasons why people might not get along. There's a lot of reasons why people might just slowly isolate on a team in an unhealthy culture. But like if you don't know what else to fix, at least fix your team. Because if you have the right team, you can overcome any problem. And if you have the wrong team, every problem will defeat you.

[00:46:38] Allan Stewart: So I think the other thing to go back to your question, Matt, is like, oh, well, if we're starting getting this paralysis and feeling it because, oh, there's so much to do. I think that actually it doesn't have to be that hard, right? It's going to be more difficult if you haven't done it before. And so you need to look at that and find that mentor like Dave was talking about. But, but some of these things aren't that difficult to do, especially today, right? So like 15 years ago, even 10 years ago, getting a CI tool up and running was kind of hard. But now a lot of them are baked into like a lot of the cloud provider stuff, or you can just, you know, download a Docker image that probably has one and you can just get something going, right? So it's like if you want to get CI going, it's like, okay, can you build it? Great. Okay. Can you execute a test? Great. And then you just build on these things incrementally. And so if your problem is that code, then what is it you want to measure about the code? You know, like Dave, Dave was joking about the linting thing. Maybe isn't your most important problem, but for some, some people, maybe that's it. It's like the code is so inconsistent that you can't, you just can't navigate it. And so if everybody gets consistent about the style of the code, maybe that, So you figure out what, what is that one thing that you, you're hurting, you know, and then you can get the tests running, you can get the CI running, or maybe the thing that is hurting is that the, the system just falls over every so often. Well, maybe you start measuring and, you know, pick a tool. Maybe it's a logging framework. Maybe it's, maybe it's a metrics thing that like, like influx or graphite or something like that. You can put a dashboard on top of it, like Grafana and it, they're not actually too hard to get going these days. So getting it working the first time might take you a day or two to just like, get something just like that first heartbeat out onto that graph. But once you've done that, you've, you've opened the door, you've unlocked the ability to start measuring things. And as soon as you're measuring things, then you have information and you can start doing things. Right. And, and CI is doing the same thing. It's just in a slightly different way. It's, it's measuring something. Tests are measuring, you know, some form of reliability or it's, it's a measurement of whether the system has changed, whether it has broken, you know, linting or, or style rules that they're a measure of how consistent your code, the, the style of your code is. And so you're starting to get feedback and you're getting feedback. You're getting data that drives you to say, okay, well now that we have data, what do we do? Where is the hotspot? Where is the thing that we should fix? And then you, and then you have a way to start recovering, pulling out at the dive, the death spiral that your project has been in.

[00:49:51] Matt Baker: Yeah. It almost sounds like, um, maybe there's more than one right answer in that situation. And maybe more importantly than, you know, figuring out the exact function that, that is the hottest or the exact, you know, uh, endpoint that that's getting the most beat up. Maybe there's, there's something to be said for just take a step, just, just start trying. And I, I don't know which one of the steps would be valid, but I am picking up on this, you know, in our conversation that, um, maybe there's not one right answer. And, and the more important thing is to, to, to just do something. And, and the thing you do, keep it small so you can get the feedback quickly and then, you know, maybe start adjusting from there. But, and I'm sure this doesn't apply to all projects, but maybe there are just some projects that just need some kind of love and give it something. And then over time that'll pick up steam.

[00:50:51] Dave Adsit: Right. And, you know, like you say, context really matters. And if you're trying to work kind of in a context free way, there are some, some things that have been shown to have a positive effect across a wide variety of projects and systems and teams. Nicole Forsgren and Jean Kim and Jess Humble put together a book called Accelerate, and it talks about healthy projects. And if you're on a sick project, what you'd like to do is become healthy. So some of the things that they talk about in there, they've got a set of key performance indicators that you can track or calculate without too much tooling that will allow you to get a handle on some of the potential areas for improvement. And, that the, the four that they talk about as being the most indicative of, of success are the lead time to complete something new. The deployment frequency, like, is it hourly, daily, weekly, monthly, annually, heaven forbid, you might get to release once while you work there. What is the meantime between failures? Like how long do you go in your production environment before you have, what you would consider to be some kind of a failure? And when you do have a failure, how long does it take you to recover? So you've got those four things, the lead time, the deployment frequency, the meantime between failures and the meantime to recovery. And if you can measure those and then you can pick one of them to work on, you can make improvements that will have a drastic impact, positive impact, impact in the health of your project overall.

[00:53:08] Allan Stewart: One of the things that I like that this kind of leads into is that we talked about a lot of technical things, you know, set up a CI pipeline, set up a, you know, a metrics dashboard or log to your logging framework or whatever. But there's also the people aspect, right? And we touched on that a little bit. It's like, you know, if, if you need tests that give you confidence that you can change code, well, your, your team has to be on board with that. The team has to, you know, to do, to do that. If you need a consistent style, the team has to be supportive of that. If you're going to change how you deploy, your team needs to be supportive of that. And so that leads into some other team related and management related concepts. one of the things that I have found about five years ago, I got introduced to lean software development and some of the principles behind lean, which just really help with, with some of these, with these problems. You were talking earlier, Matt, about how, you know, sometimes the problem is that there's just like, there's so much work pushed onto a team. But if you switch to a true Kanban board, where the team is pulling work instead of having work pushed upon them, then that can make a huge difference that can change the morale and the outlook of the team. If, if they come in and you don't have, and you've set work in process limits. And so, you know that, Hey, we're not going to start on 17 things, but we're going to finish one or two. That can be a huge adjustment to a team and a huge benefit. If, if they come in and they see, Hey, our backlog of things that we need to do is not 10,000 items long, and we can't actually even enumerate them all. That helps that there are, there are these human aspects of the system that we can improve that relate into some of the accelerate metrics that you were talking about, or relate to lean and agile principles that can really help to improve the system. And turn a software project around as well.

[00:55:30] Matt Baker: Yeah, I really like that one whip work in process. This one in particular. There are contexts where it doesn't apply. Yes, but it seems to be consistently one that is violated in unhealthy projects. There's something about this one. There's something about starting up too much work. And then, you know, we, we, we could talk another time about all the, the effects that come from that, but maybe one concrete thing that you could at least consider when you're stepping into a project like this. And it feels like, you know, feel so big as to, you know, look at your whip. If you had a team of tangent 10 engineers working on 15 things, and you know, you've got a problem, maybe bring that down to two one or two or three and just try it out. You know, I, I just this, this particular point, seems to come up again and again for me that a lot of unhealthy teams are struggling from just too much work.

[00:56:32] Allan Stewart: Absolutely. And people want to set their whip limits based on the number of people in their team. And that's, that's generally a good idea. But usually what they say is, it's like, okay, well, how many people in the team plus some constant? And what I recommend is, well, how many people in your team divided by some constant? So if you've got six people, your team, it's a good place to start. Don't set a whip limit of eight. So you have some buffer room instead, set it to three or two or even one. So that then all of a sudden you're saying, okay, we have to get a thing done before we can start the next thing. And that goes into some of these other concepts, right? That are traditionally considered technical concepts, but like continuous integration. If there are less things to integrate, you will be happier because it's, you have fewer merge conflicts. You have fewer places where somebody changed the behavior of the other class that you were depending on because they needed to do something else for a different reason because you were changing two things at the same time. And you start to corral those in. And then it forces you. You say, oh, well, I can't start on a new thing because of the whip limit. So maybe I can go and work with that other person and suddenly starting getting into these other people related issues of, oh, well, maybe you should be doing some pair programming. If you have six people and whip limit of three, well, then you can have three pairs and they can all be working together and improving the quality of the system that they're, they're building and trying to do less things all at the same time.

[00:58:23] Dave Adsit: And I would add to it that one of the things that I've noticed, one of the big reasons that teams end up starting too many things is because nobody has trust that they'll get it done. I mean, we talked about this earlier, but if I come up to a dev team and I demand that they start my thing, it's because I think somehow that by starting it, it will get finished, which isn't necessarily true. Um, but you can start to, you can start to correct that. You can start to repair that relationship. Delivering something. Sure. It was small. The first one was probably small, but you delivered it. Now they've seen that the team can deliver it. And now you deliver another one. And now, Oh, some things are starting to happen. You start delivering more and more value by focusing on all of your efforts on one or two or three things at a time. You can start to build a relationship of trust where fewer and fewer people, people come over and demand that you start and finish this thing by this date. Or at least, please start it because I know if you start it, you'll eventually get it done because I'm not thinking about the other 100 items that you started this week. Yeah, yeah. I don't know the

[00:59:46] Matt Baker: words to say that really resonates with me other than just saying that really resonates with me. And that doesn't quite capture it. There's a mentality there that you're commenting on that I'm reacting to pretty strongly. Allan, to something you said, teams adding more whip to create a buffer. Isn't that like one of the big paradoxes here? You know, we say, well, the way that I've seen this lately is, well, we have this priority swim lane across our boards for the high priority items. Or like we've extended our backlog just a little bit just so we don't forget everything we want to remember. And there's these reasons for coming up with, you know, more whip equals more buffer. But I think the reverse is what is actually true. You know, the whip of three has more buffer than the whip of five.

[01:00:43] Dave Adsit: The good thing about the important items in your backlog is that you will never forget to do them. And so if you are really, if you're struggling with team morale because of the overwhelming amount of stuff that someone somewhere in the organization committed you to do, delete the whole backlog. Just every card. Just trash them. Because the actual important ones

[01:01:12] Allan Stewart: will get remembered and will get done. And if it wasn't important, then you're just wasting time or doing a backlog grooming meeting where you talk about, you know, the t-shirt size of how much work is that going to be? Because if you're never going to do that work, who cares? So that goes back to me for, to me, that goes back to queuing theory, right? So we went to a conference one time and one of my coworkers said that he was in a meeting, you know, one of the sessions and the speaker says, you know, nobody cares. Nobody gets fired up for queuing theory. And I thought about that a lot because the more that I think about it, it's like, I think I get fired up for queuing theory. I hear you, man.

[01:02:02] Dave Adsit: Allan, can you, can you tell us how many hours you have logged on Factorio?

[01:02:07] Allan Stewart: Oh man, there are a lot of hours of moving plates and inserters and learning about queuing theory. I think though in Factorio, it's kind of opposite. Because in Factorio, I want all the buffers and I want to back up everything all the way back to the beginning of my factory. But in software development, I do not want to do that. And that's where your whip limits are really small. It's like, I just want you to go in and get a thing done. What is the most important thing that you could do right now? Do that, ship it to production, do the next one. Don't do seven things because queuing theory, And when we look at the mathematics about it, and when we model how people actually behave in software development, we start to see the queues forming between each other. The queues of, well, I was waiting for so-and-so to do that pull request, to give me feedback on my thing. And so I started yet another different thing. And so now I've got more stuff and I'm busy doing that thing when they give me the feedback that I was looking for, or when they submit. And now I'm doing their pull request to get the feedback from me. And we start building up these queues. And when we model it out with queuing theory, we see that they're, I can't remember what the term is. They're non-deterministic Markovian chains or something like that. And reducing whip gives you better throughput. Because that's what matters. It doesn't matter how many things you started today. How many things did you finish today? That's what matters.

[01:03:49] Matt Baker: Yeah. It's the, to control throughput, you control input. You know, a full freeway doesn't move. I don't know how else to say it other than that.

[01:03:59] Allan Stewart: Oh man. But the taxpayers, Matt, they're getting so much return on investment for that freeway because every bit of freeway is being used.

[01:04:08] Dave Adsit: There is a car parked on every square foot.

[01:04:14] Matt Baker: Success. Success.

[01:04:15] Dave Adsit: One of the things that comes up, often in my executive team meetings is what is the one thing that we can do that would make everything else we're doing irrelevant or obsolete. And that, you know, that really resonates me with me because I want to be focusing on the one top priority item that will make a difference in the business. Half of the developers that I interview mention, the first thing that what they want is the ability to make an impact in the business. I think we all have that desire to do something that really matters. And grooming a backlog never provided any customer with direct value.

[01:05:11] Allan Stewart: So related to whip, uh, work in process limits, um, and related to this idea of, you know, what is the thing that matters? Um, there's a really great book out there called making work visible. And so before we talked about, well, you know, what do you, what do you do if you're a person who's in one of these failing projects and you're not sure what to do? Um, before we answered that with kind of like these technical answers, but if, if you think it has something to do with this process of how your team executes, work and, um, more of the people things, I really recommend this book making work visible. And it's all about helping people see what is actually going on. And then you can start identifying the problems and fixing it because sometimes it becomes blindingly obvious as soon as you just make it visible and you're like, oh, well how come nothing's getting done? Well, so, well maybe there's a bunch of things that are getting done and that's just in secret. Like it's, you know, it's like the shadow it kind of stuff is like, well things are happening, just nobody knows that it's happening or um, you know, some, you know, this project never gets off the ground. Um, and it's like, oh, well our key person is always busy and it's like, oh, what are they busy doing? Well you can find out by making that work visible. It's like, oh, they're getting a bunch of requests from another part of the business and we didn't realize it. Or there's some other problem that is occurring or, or maybe it's just that the team has started 16 different tasks and, uh, and you had no idea that everybody was so busy doing all this stuff. You make the work visible and suddenly it becomes very obvious where the problems are.

[01:07:12] Dave Adsit: I guess the, the question is, how does, how does all of this relate to crafting software anyway?

[01:07:25] Allan Stewart: For me, I think a lot of it goes back to the discussion of the software craftsmanship manifesto, right? Are you steadily delivering value? If you have a CI CD pipeline that lets you deploy at a regular interval, then you can measure that. You can say, yeah, we delivered value today. If you've made work visible, you can tell, well, how many things did we deploy to production? Um, how long is it taking us to deploy things? How often do we have failures? Um, you start, you start building that up. So it, it's about that delivery value. Uh, it's also a lot about, uh, professionalism in my mind is like, do you know what the system is doing? Right. Um, writing a bunch of code and throwing it over the wall to QA to test, and then you throw it over the wall to operations to run in production. And I don't have to care because I'm a developer and that's somebody else's problem. That's not professional. You don't know what your code is doing. If you don't have tests, you don't know whether you've written bugs or not. And so to me, all of these things that help you, you rescue the project, these different techniques, testing and measurement, you know, people changes, collaborative coding. They're all ways to do better professional development. You're making better software, um, delivering better value.

[01:09:02] Matt Baker: Yeah. For me, I, um, I think it comes back to, I don't know how to restate what Allan just said. Just, you know, being professional, looking out for the people that are, you know, paying you money. Um, I can't help, but my mind is just drifting towards how you do that in this scenario. So I think, I guess my answer to your question, Dave, crafting software in the context that we're talking about, it means to be professional. And I believe that being professional means honoring the agreement you've, you've created with someone. They, they're giving you money and you, you and exchange offer them a thing. And so then, you know, it's incumbent upon you to do the thing as best as you can. Uh, and when you get into these projects that are failing, I think it's really important to anchor your feedback loops, you know, to that agreement or to that customer, um, to that value add. So, uh, that, that's the thing that's really kind of coming out of coming up in my mind, you know, as we've been talking this past, uh, little bit here is, uh, studying this stuff, this, this idea of crafting software, uh, has taught me to really anchor my efforts to, you know, where I'm adding value and then let that guide me, especially through these failing projects.

[01:10:25] Dave Adsit: Yeah. And I would, I would add that I have rarely seen a failing project that is actually adding value to the customers. And it's not to say that a well-crafted software project will never fail or that a poorly crafted project will never succeed. I think that it is possible to have both of those. I do believe that you are less likely to find yourself on a very sick project when you are paying attention to your profession and putting effort into crafting your software well. So, if that is one of your goals, then it will motivate you to find solutions to some of the problems that are causing your project to fail because you recognize that it isn't somebody else's problem to solve. It's my problem to solve because this is my project. And as a professional, I must craft this project, craft this software effectively for my customers. Whatever, whatever, whatever that means, whatever I have to do to get there.

[01:11:45] Allan Stewart: And with that, I think we will wrap up and recommend all of our listeners go and join a community of professionals by attending a software crafters group or meet up in your area. You can find that person that will help give you some more feedback about how you can rescue your failing project. You can get some additional pointers into how do you implement some of these things that we've been talking about today? And if you happen to be in Utah, the Utah SC group at Utahsc.org meets the first Wednesday of each month in Draper, Utah, and we'd love to see you there.