Crafting Code Podcast

[00:00:16] Allan Stewart: Welcome to the Crafting Code Podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately I've been thinking about politics. When you change what you say based on how you want others to react rather than what you really

[00:00:33] Dave Adsit: think. I'm Dave Adsit, a VP of engineering, and recently I've been thinking about the evolution of a system architecture due to changes in the team and the growth plan for the team.

[00:00:45] Allan Stewart: This is the first episode of what we're calling version two of the podcast following our long hiatus. We're back. And our episode topic for today is technical debt. Technical debt is something that has been in the news recently off and on, but it's not new. There were some headlines earlier this year in the Wall Street Journal about tech debt being the invisible $1.52 trillion problem. A couple of years ago, there was a big outage at Southwest Airlines. There was also some discussion recently about LinkedIn and some of the issues that they had modernizing their legacy code base because of the technical debt that has grown up in their system. So a lot of people are talking about this now, not just software engineers. It's in the Wall Street Journal. So what exactly is this? What is technical debt? What are we talking about here,

[00:01:49] Dave Adsit: Dave? Well, technical debt is an interesting concept, right? It's a concept that was originally coined by Ward Cunningham. And he said that basically when you take on technical debt, it's an intentional choice to ship software that may not be perfect. Also, I've heard it described as the change or the delta between what the software does and your current understanding of the domain model that it operates in. If the... So technical debt is... A lot of things get thrown into the bucket of technical debt, sometimes up to and including just code I don't like. I prefer to exclude that from actual technical debt personally, but it often gets included.

[00:02:44] Allan Stewart: Yeah, for sure. It can be a catch-all. Anything that we don't like about the system, it's technical debt. Any decision that was made by some other developer before we got here, well, that must be technical debt or legacy code, right? Some of these terms kind of get thrown around. And it's difficult because really it is a metaphor, right? There's not an actual technical debt as far as a real financial... You can't swipe your card and pay it off.

[00:03:16] Dave Adsit: Sure. Yeah, exactly. Right? You can't just go withdraw from your savings account and say, okay, we're paying off all the technical debt. Now the debt is gone. And now we are back at zero. And we're going to keep going forward. Exactly. And so like any good metaphor,

[00:03:30] Allan Stewart: you can stretch it too far. You can break the metaphor. It doesn't extend indefinitely, but there are useful aspects about it, right? It's a useful way of thinking about things. So one of the problems that I've seen with the term technical debt is that businesses often use debt as a tool. And so it can be difficult sometimes to get everybody aligned around seeing that debt is a problem. If you're in like a VC funded startup, well, that's your life. Your entire runway is built on debt. And so it can be hard to get people to understand that this is a lot more like consumer debt. This is the kind of thing that causes you trouble and

[00:04:14] Dave Adsit: you get into problems. Yeah. I mean, I've worked at several companies where we took on debt. We borrowed money against the future in the hopes that we could spend that money well, now to make more than enough to pay it back later. And with technical debt, I've heard people say similar things that, oh, well, we took a shortcut here and we want to come back later and fix this. And we'll come back later and we'll write it the right way. But right now, we're just going to take the shortcut. I feel like that is... It doesn't ever work out as well as you hope it's going to.

[00:04:59] Allan Stewart: Paying back the debt is always more difficult than incurring it. That's for sure. Yeah. And that's one of those places where I feel like it's easier to compare it to consumer debt. Right. So if you're in a startup and you're looking for funding, that can be a lot of work to convince somebody to fund you. But technical debt is more like the credit card that shows up in your mailbox. And they say, yeah, please sign up, start using this right away. You're pre-approved. And before you know it, if you just keep on swiping that card, you can find yourself in a lot of trouble. And that's where I think a lot of technical debt, I think the metaphor can hold in cases that aren't as intentional. So not as broad as just any code that we don't like, but I like to think about technical debt in the terms of there is some technical issue. There's some part of our code. There's some reason that is causing us to spend extra time and money. Right. It's kind of like a tax. Every time that we go into this bit of code, we have to pay some additional tax or interest on this technical debt because of something that happened. And the investment wasn't necessarily, or when you took on that debt, it wasn't exactly because you intentionally said, oh, hey, I'm swiping this technical. Debt card now, but more like you started doing software at all in the first place. That's the investment that you're doing. You're investing in writing code and code is a liability. Right. The benefit of code is what it does for you. But all of these other technical issues that kind of get lumped into this broad tech debt concept are really an issue of you were doing software development in the first place.

[00:06:58] Dave Adsit: Yeah. Yeah. I really like that phrase that functionality is an asset and code is a liability. So when I think about it, that's typically how I think about the debt that we have is due strictly to the fact that we wrote custom code. We wrote custom software. So one of the things I think about a lot is that you're going to incur technical, no matter how careful you are, because the act of shipping software and giving it to customers reveals what the software should have been. Right. And so it's when you have real users first engage with the system that you begin to learn what the system ought to have been all along, but you didn't know because there's no way to learn without shipping and giving it to people and letting them try it and see what they do. So we're going to incur technical debt no matter what, because our understanding of the domain model, the domain, the problems domain, the solution domain, the whole domain model that we use to solve it, it's going to evolve as we build software, ship software, and have users engage with our software. I don't know. I would say that waterfall is a myth, the myth of waterfall, at one company where we did annual releases and that sure felt like real live waterfall. But the myth of waterfall, as we imagine it, is that you'll go and do all the analysis and figure everything out. And then all you have to do is go code it. Right. If that were true, then in those situations, you wouldn't have technical debt. But my experience is that you are always learning as you ship software and as people use it. And therefore the model that you used for the That's in production is going to have a difference between there's going to be a difference. There's going to be a problem. There's going to be a debt between what we shipped and what we wish we had shipped if we had known then what we know now. And so it's not that we seek to avoid technical debt. It's just that it's kind of, you kind of have to accept that it's a part of your life as a software developer, but you have to do things to manage it. You can't just live in it because it

[00:09:25] Allan Stewart: Yeah, absolutely. I like that, that concept that it doesn't go away. You can't, you can't ignore it. You also can't avoid it unless you get out of software completely because it's, it always is there. And so that part of the metaphor, you're going kind of closing the loop on this whole metaphor discussion. How do you take on the debt? That's still kind of questionable. What count says this is technical debt versus this is just a mess versus just anything. We don't like, you have to watch out that people don't always use the metaphor in the same way. I really like Martin Fowler's tech debt quadrant, right? He talks about like, there's different flavors of it. There's reckless versus prudent and deliberate versus inadvertent. And so there's, there are places to play in that metaphor, but I like what you were saying. What we're saying here is that there's going to be tech debt. One way or another, there's going to be that thing that you have to pay your time and money on that you wish you didn't have to, because it's not directly connected to the features, to the business, to the thing that you're delivering, but it's some technical decision or, or even just that the world has changed. And the things that you were depending on before now have security vulnerabilities or end of life of that version of the software or something like that. So what do we do about it? So we've talked about the metaphor and there are these people and articles talking about tech debt. What do we need to do to manage it?

[00:11:05] Dave Adsit: Well, I think that, you know, the first thing that you need to do is you need to identify where you, where it exists in the system. So I would say it's kind of the same as, as all of the other things, right? We have to identify where the problem is. We have to make a plan for addressing the addressing and tracking the problem. And then we have to execute on the plan to resolve it. And so, you know, first thing is identify, I have, I have technical debt in this part of the system. I built the system with, with the assumption of X. And it turns out that that assumption is incorrect. I didn't think any of my users would ever want to use two of the same thing. And it turns out that 10% of my users use two or more of this thing all the time. And so now I have to update my system to take account the fact that they may be using the software in a different way than I expected. Right? So, okay. So that's the technical debt. I've identified it. Now I'm going to create a plan of action for it. And the plan of action could be a whole bunch of different things. It could be, I'm going to accept it and I'm not going to address it. And I'm going to leave the system in its current state. And I'm just going to work around that problem for a long time. Um, I chose to create a full name field in my database. And it turns out that all of my users need to be sorted by first name or last name or whatever. And, but I'm just going to write a more complicated SQL query. So I'm paying the cost of that choice that I made, but, um, I I'm not changing the software to make it easier to work with in the future. I'm just paying the cost now and paying the cost every time I work with that name field. And maybe I don't do that very often, and maybe that's the right choice. Just pay the cost in an ongoing way. Just basically like paying the interest on your consumer debt, right? You're not paying it down, but the interest is low enough that you just

[00:13:06] Allan Stewart: accept it and keep going forward. Yeah. I've definitely seen places where that has happened. And I think the problem is if that's all you ever do, then it just compounds because you're, you're adding more, you're always getting more debt, more technical debt is always happening. And so at some point, you have to do something about it, but I agree. Sometimes the right thing to do is you, you ignore it, or you, you need to be strategic about what things are most important. You know, if this, if this class, you're only going to come in and edit some things three times a year, maybe it's not that important, but if it's something that impacts you on a daily basis, then it's good to, to get rid of that because you're constantly paying that, that interest payment.

[00:13:55] Dave Adsit: Right. And so, I mean, the other choice then is to pay down that principle, to find that piece of that place where the system does not behave the way that you wish that it did and, and change it. So we can, we can do a refactoring, which is to change the, change the way the code is implemented or executed, but without changing the functionality. And that's typically where we're going to start, right? If we're talking about, Mm-hmm. that name field, maybe I decide I need to have a first name field and a last name field. And so then I have to decide where, where in the code is the right place to make that change? Where do I split those? Do I just create two, two new columns in my database and then copy all the data over and then change all of my data interaction layer, my repositories or whatever, so that they write to both of those fields. And then maybe they read from the two fields and combine them because most of the code doesn't care that it's, it's just one full name. Right. So I need to start paying that down. And that's typically, like I say, going to be first through a change in how the code is implemented that does not change the functionality because most of the functionality is what we already want, but maybe that enables us to then add additional functionality in a way that we couldn't before we paid down that technical debt before we did that refactoring.

[00:15:19] Allan Stewart: Yeah. I like that. The other aspect of managing technical debt that I think about a lot is, is just making it a regular part of your working ethos. When, when technical debt is this thing that you are always pushing off, it's like, Oh yeah, we know about this, but we're not doing anything about it. Now, sometimes that happens because you're picking your battles. You, if you spread yourself too thin, that definitely causes way more problems because you're doing a little bit of fixing, but you're not getting much of the benefit and so you're, it takes a really long time to get any one piece of benefit. So I like to focus. I like to be strategic about what do you pick to work on? But to me, that important aspect is that you're doing it on the regular, right? Refactoring is a regular part of how you work. It's not just making the code work, but making it work well. Right. And then if you're continually doing that over time, things get better and better. You have to have buy-in with your team. You have to, be working together and you have to have alignment on what does better even mean so that everybody's rowing in the same direction as it were. But, but once you do that, what I found is that it takes a, it takes more effort than people want to believe, especially people who are not developers. I really like Marty Kagan. He's in the, he's a luminary in the product space. And he once working together. So 20% of your capacity should be the standard for an engineering team. That that's where you should be all the time. So as a team, you shouldn't spend more than a week without having at least 20% of that time being paying, paying down your technical debt. And I like that continual process of it, because if you save it up and you're like, oh, well, we're going to spend 20% of our year on technical debt, all in one big bang. That's exhausting. And nobody is happy with that outcome. But if you do it a little bit at a time, you keep things clean. And sometimes you need more than that 30%. If you're already well into your debt, then you're going to have to spend the time to get yourself back out of the pit that you've been digging for yourself.

[00:17:45] Dave Adsit: Definitely true. That's one of the things that I like to think about as well is like, you know, there's a lot of ways to structure consumer loans. Um, but typically you're going to be paying back something every few weeks, every whatever, every few weeks, every month, you're going to have a bill. Uh, when it comes to technical debt, anytime I've seen people try to do a cleanup sprint or a big refactoring project or whatever, I, I've seen that those tend to grow beyond the expected time box and usually leave you in a bigger mess. Than you were when you started. And so one of the things I tell my teams is that they should do four product cards and then one engineering card at a minimum. And even that, which turns into for most of my teams, that might be, you know, two or three days working on product and then a half a day to a full day working on an engineering thing, usually related to cleaning up some technical debt of some sort. Mm-hmm. That feels so expensive. Yeah. I mean, I'm not saying that it's expensive to the people who are not engineers on the team. They just act like that 20% is just the biggest burden that they've ever had to deal with. Uh, and they say, can't we just push all this off and do it all later? And I'm like, if it's, if it's a burden at a half a day, what are you going to say when I tell you that you can't have any new functionality, any new features, any time from the team at all for two weeks, if we stored all of this up and did it once a quarter. Yeah. Do you really think that you would like that better? Like the pain gets bigger if we try to pay too much at once.

[00:19:27] Allan Stewart: Yeah. And, and it's harder to get stuff done. Uh, I've definitely worked in code bases where it started as a mess and we, we went into it with this methodology of we're just going to continually be cleaning things up a little bit at a time, a little bit at a time. And after a while we could ship features much more quickly. Right. So, so even, even your ability to deliver improves dramatically by handling it this way. But if you wait, it just, it, the mess gets in your way. Um, as long as we're talking about metaphors, I'll throw in one more, uh, another one that Martin Fowler, uh, brought up in an article once and he compared it to, uh, the kitchen. If you are writing code, it's like cooking in the kitchen. And when you cook, you make a mess. It's just, it's inevitable. So something happens, there's a mess and now there's dirty plates or there's stuff splattered on the counter or, or whatever. And then you have a decision to make. What do you do about that mess? If you're in a hurry, you might just leave the mess. You leave some stuff in the sink and you move on and you, you go ahead and cook lunch. And if you don't clean that up, then by dinnertime, it's more difficult, it's increasingly difficult to. Yeah. You're not going to be able to do what you need to do, right. Deliver the feature of cooking a meal because you've got all the debt of your, um, dirty dishes. And now we've thoroughly mixed our metaphors, which is one of my personal favorite things to do. Although it doesn't always necessarily aid in understanding, but hopefully the idea there is clear that, that like the kitchen technical debt, it builds up and it gets in your way right away. Like it doesn't take any time at all. Yeah.

[00:21:47] Dave Adsit: You may have created debt that you don't know about until much later, even in choosing your tech stack. There's a lot of tech stacks that are currently in fashion and in use across our industry. But if you chose one, and I mean, say you chose, I'm not going to name a specific programming language, but you choose a programming language that's currently fashionable. And you know there's a lot of people that are excited about this programming language because it's the current fashionable programming language. And then a year later, you're trying to hire and grow your team, and you realize that most of the people that were excited about the fashionable language have moved on to the next fashionable language. The new fad has replaced the old fad. And not that many people learned how to use this programming language anyway. And so even if the system itself is, you know, well, well factored well tested. You know, it's got a solid hexagonal architecture and it's deployed with CI CD and you've used infrastructure as code from top to bottom and everything is magically delicious. You may see, you may find that you don't have the ability to hire developers that you need or if you hire a developer you're going to have to spend a lot of time training them on the uncommon stack that you picked. And so, even that. Could be considered a form of technical debt. You may say hey, look, we used some language, and now we need to rewrite in a language that more people know or use, like, you may find that that is a cost that you will have to take on and it may be an expensive one to bear. That doesn't necessarily mean you made the wrong choice at the time given the information that you had it just means that our understanding of the system and of the problem space. Expands the more we work with it. I think I've heard. What's his name. Krijevsky Joshua Krijevsky has said that you have to rewrite the system five times before you can write it well. And how many of us actually do that I mean it is so much work to start over and write the system again in fact, if any of my developers say I would like to start over and rewrite this system and build system to I tell them, please go read about the system to effect and come talk to me about it at our next workshop.

[00:24:38] Allan Stewart: not trying to do the entire thing all at once. Right. Yeah. That's the other thing is that you,

[00:24:44] Dave Adsit: you just cannot pay all your debt down. You can't pay it all down. You can't pay it out all at once. You can't pay it down over time. You have to, you pay it down, but you cannot pay it off. I should say. Yeah. Yeah. It's always there. As you are paying it down, you are creating new debt that

[00:25:00] Allan Stewart: you are unaware of until later. Yeah. And I like what you were saying before. I was just thinking it's, it's Conway's law. Just, it crops up everywhere. You, you make these technical choices and it has an influence on your people, like what you were saying with hiring. And now you've got a people issue that is also part of your technical issue in this big socio-technical mess that, that we create every single time. And that's, I think that's part of the reason why, why it's that way. We can say that our code is going to sit, you know, It's not bit rot. Like the, the bits aren't going to rot away, but people keep changing the world around you. The people are mutable and they are always changing. Variation is always happening to you. And so you just keep ending up in debt. Well, and that's, that's an interesting

[00:25:54] Dave Adsit: concept right there as well, is that the idea that software doesn't rot. What's the problem? Software doesn't rot. We don't have to work on this because we did it and now it's done. And it's forever good, right? From the perspective of the actual bits on disk, they're not changing and getting worse. They're not decomposing. Bacteria are not consuming them. You know, barring anything like an EMP, they're just going to be the same as they were until you change them. But to your point, the entire world changes around you. You released this on, I don't know, .NET 4.7.1, but now everybody's moved on to .NET Core. You built this on Node 10, but Node 10 is no longer LTS. Now it's, you know, we're closer to Node 20. You, if you are not constantly paying for the maintenance and the upkeep of your software systems, they will inevitably fall into debt. And so, you know, that's another thing, another area to think about is like, what are your dependencies and what, what are, what are the parts of the system that are actually outside of your control, but you still rely on them? Yeah. And, and that is an area of debt that you're also going to accrue whether you like it or not. It turns out malicious actors on the internet are always trying to find better ways to exploit systems and the security experts on the internet, the white hats are trying to harden the systems against those attackers. And so we, as you know, your typical application developers need to be updating our systems continuously in order to, um, stay ahead of our predators. So I guess the last thing I would say is just make sure you are paying down your technical debt on a regular cadence, decide what works for your team, whether that's, you know, we're going to do one day a week, or we're going to do one card in five or one card in three, because we have a lot of technical debt in our system. Or, you know, if you work in a system where you do a lot of detailed estimates and you actually got good at um, decide what percentage of your team's capacity you're going to expend on keeping the system clean so that you can keep adding functionality to it. So you can keep increasing that value to customers, to your business, to any of your users, um, without continually slowing down further and further and further as more and more of your effort goes to maintaining just the interest on the technical debt that you've already accrued.

[00:28:32] Allan Stewart: Totally agree. You have to change how you work. It's like going to the gym. If you want to be stronger, more healthy, whatever, you got to change your diet and exercise. And you can't just do that in January and hope that by November that it'll still be beneficial for you, right? It's a, it's a technical lifestyle change that, that you have to make if you want to keep ahead of your technical debt.