Crafting Code Podcast

[00:00:16] Allan Stewart: Welcome to the Crafting Code Podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately I've been thinking about the intersection of job enjoyment and fulfillment.

[00:00:31] Dave Adsit: I'm Dave Adsit, VP of engineering, and recently I've been thinking a lot about building effective teams and engineering cultures for in-office, remote, and remote hybrid environments.

[00:00:44] Allan Stewart: This episode topic is technical health. I think it was a couple of years ago that I started reframing the conversation about technical things. I started reframing the conversation about technical things away from this idea of debt and started talking about it in terms of health. Apparently, other people have been having a similar revelation because the ThoughtWorks tech radar started tracking a blip for tracking health over debt. I think it's an interesting mindset shift to start thinking about it away from a negative frame. Right? Health is a more positive framing, although inevitably when it comes to technical health, we're telling you about how it's not good instead of excellent. But at least it gives you that positive frame of mind that you're thinking about.

[00:01:44] Dave Adsit: Yeah. As we started talking about this originally, it got me thinking a lot about the concepts behind positive psychology. Yeah. Yeah. Yeah. Yeah. The founder of positive psychology, Martin Seligman, and how he talks about, he's got that book, Learned Optimism, actually, is the name of the book. And basically, the concept there is that you can teach someone through consistent negative reinforcement, you can teach someone learned helplessness. They could potentially improve their situation, but because you continually beat them down every time they try, they eventually give up and accept the bad state that they're in. Yeah. Yeah. Yeah. Yeah. You can learn to think in a much more optimistic way around, hey, I can improve this situation no matter what it is. And he talks about how the most successful salespeople are those who can make 99 calls and have, or make a hundred calls and have 99 people hang up on them and then have one person schedule a meeting and then go to a hundred meetings and have 99 people say no and only one says yes, and they still consider that a win. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. put you into a negative mindset, which will decrease the possibilities of you succeeding in the space. And so what we want to do is give ourselves, I mean, it kind of sounds silly because I remember back to the Saturday Night Live sketch around daily affirmations or whatever, right? But actually, putting ourselves into a positive mindset can actually impact our ability to have the endurance to go the distance to have the success we need to have. And so I think there's something really positive and powerful about switching the metaphor from debt to health. And as we discussed on a previous episode,

[00:04:01] Allan Stewart: you will always have technical debt, right? If you're thinking about it in terms, because it's not actual debt, it's a metaphor, but this concept of in your system, there's always going to be something that could stand improvement. And so if you are taking that Then you will always be in debt and it's not recoverable, right? You can't get out of it. You can pay down debt, you can take care of certain problems, but you will always have more debt. And so I can definitely see how that would feed into that downward spiral of just learned helplessness or just acceptance of, well, I don't know that we need to really worry ourselves with tech debt, because no matter what we do, always there's more, there's always tech debt. So like, why do we even

[00:04:57] Dave Adsit: try? Yeah, this system is terrible and it will always be terrible no matter how much work we put into it. So we might as well take shortcuts now because it's not like we're making it worse, right? Even though we know that we actually are probably making the system worse by taking the

[00:05:12] Allan Stewart: next shortcut. Exactly. And the debt metaphor doesn't fit in some cases too, because with debt, it's all dollars. Right. Or some kind of money and you can exchange one kind of money for a different kind of money. And so like, it should all just be equivalent, right? But we know that in software, it's not, there's different kinds of technical issues, different kinds of complexities, and some of them are more expensive. Some of them are stickier and harder to deal with than others.

[00:05:44] Dave Adsit: Yeah. I remember long ago, you and I worked in a system that still had one little, we had just, we can mostly decompress. We decomposed our monolith into microservices and we had this one little part left and we always called it the monolith tax whenever you had to work in that code that nobody knew very well anymore. It was like, things that should have been easy were just a little bit harder than expected and took a little bit longer than expected. And that was just the state of it, right? But as we moved more and more functionality into the other architecture that we were moving towards, things would get easier over time. And we would be able to move more quickly. And so the overall health of the system was increasing, even though it was very hard to work in the part where that we were considering to be our, our legacy debt.

[00:06:32] Allan Stewart: Yeah. I think that piece of the system, or at least one of those monolith pieces of the system that lasted a long time, actually outlasted the team that technically owned it because it didn't require a lot of maintenance. It generally was working. It wasn't an area of focus and Nobody actually knew how to run that.

[00:06:59] Dave Adsit: It's not that they didn't know they were physically incapable because everyone on the team had chosen to use MacBook pros and the legacy system was.net framework. And they couldn't even work on it anymore, even if they had known how they still would not have been capable of doing the work. Right.

[00:07:22] Allan Stewart: So coming back to this idea of health over debt, is it just, is it just a better metaphor? I mean, yes, we're, we're still talking in metaphors, but I think it's, it's more than just a rebranding, right? So when we, when we take something like this and we're trying to change our paradigm, it's not that we're just substituting in one word for a different word. So it's, it's not that we're just saying health and debt are interchangeable, but in fact, we're, we're reframing it in a way that gives us new insight into how we can measure the system, what we can do, like what opportunities we have for working on it. With the debt metaphor, it's always about looking for problems to pay off. But with a health metaphor, well, now you can think about increasing your health. There's the possibility of kind of a, almost a fitness aspect of it, where you're saying, Hey, in order to be able to accomplish something, are we fit enough? Right. We might be generally healthy, but is that, does that mean that we're ready to run a marathon that we might want like an elevated level of fitness, or if we want to be really good at a sport, there might be certain exercises that we need to do. Right. And so carrying that metaphor over into the technical space, there may be areas of your code where you're not as healthy as you want to be. And it's not because you're necessarily in bad health, but you've not prepared yourself for something really important that you know is going to happen that, that relates to your system technically. Right. So for example, can you scale to meet the kinds of load that your particular system lives in? Right. So maybe you've got periodic seasonal events, maybe it's around black Fridays, or maybe it's around the start of the school year or some other event that causes a big disruption. And are you healthy enough to be able to perform for, for that, that event, just like a, you know, just like a athlete, they're going to have to prepare themselves, practice, get themselves fit so that they're ready for a particular event.

[00:09:42] Dave Adsit: Right. And I like to think about it just to reemphasize that point. There's different types of health based on different goals that you have. You can do strength training because you want to be very physically strong. You want to be able to lift a large amount of weight, or you might be doing strength training to enable you to increase endurance by doing more cardio training based on having more strength. Right. And so different types of health, I mean, you would consider both a bodybuilder and a marathon runner to be healthy, maybe you've built your system in one way. And if you have really high spikes, you might build it in a different way. So if you have a consistent daily load, you might say, hey, our system is very healthy by having a fixed number of servers in place at all times. And if the number of servers drops, we know that we're out of our health band and we want to add one back or whatever. And if you have consistently low level of users with huge spikes, then you're going to have to build a system in a different way. You're going to have to build things like autoscale. You say, hey, this system has to have a really, really fast response time on scaling up new instances, new nodes to handle traffic when we start to see a traffic spike because we don't have time, we don't have a long time to respond. And so both of those systems could be healthy in their context, but it is a different context and we need to be aware of that. Agreed. Also, both of those systems, I mean, regardless, regardless of what type of health you're looking at, there is this idea of built-in continuous preventative maintenance and effort. No one says, hey, like I'm healthy because once I exercised and now I sit on my couch every day, right? If you want to be a bodybuilder or a marathon runner, you are going to be lifting weights and running almost every day. I mean, I don't know how to train for a marathon. What I've seen is running almost every day, but different distances every time. That's the extent of my knowledge because I've never trained for that. But I do know that going to the gym and lifting weights every day is not going to prepare you to run a marathon.

[00:12:16] Allan Stewart: Right. It's not in stasis, right? In the debt metaphor, we think about it in terms of, oh, well, we've got this much debt and like, why would we keep accumulating more debt? But in a health metaphor, it's more easily understood that, well, there are some things that just naturally decline. Mm-hmm. And if you don't keep up, you don't do that preventative maintenance, you're not going to be able to do it. And you can't just pay it off all at once either, right? I think that's the other thing that the debt metaphor is perhaps guilty of having business people think about, well, can't we just pay that off? Like, can't we just invest in getting rid of it? And well, yes, you can for a thing, but then they get frustrated later because there's still more and more and more. Right. is you can't just go to the gym, as we learn every year, you can't go to the gym for January along with your new year's resolution and then expect to still be fit, you know, come October, November. It's more lifestyle change kind of concepts. I've liked to talk recently about technical lifestyle changes where you're really changing how you go about owning and maintaining your code And getting away from those ideas of, okay, now we're going to do a project, whether it's a feature project or whether it's a technical debt paying off project, right? You have to change your mindset of what you do regularly in order to do that. And I think that the health metaphor gets people prepared to hear that and understand that concept much better than the debt metaphor does.

[00:14:00] Dave Adsit: Well, I like to think that if you're building a healthy system, you are probably investing every week in cleaning up bugs or issues that customers have found, users have found, as opposed to we're going to let all that stuff pile up and then once a quarter, we're going to do a big bug bash where we try to resolve them all in one or two days. The same thing happens. And I would say that we as engineers are more guilty of this than the rest. We say, hey, after this sprint, we need to take six weeks to rewrite this, this huge component that we made a giant mess on. And we're going to give ourselves a deadline so that by the time we get close to the end of the six week period, and we've only done half the work, we can make sure that we rush and rush and rush and rush and finish the rest of it super quickly, filling it with bugs so that we are guaranteed to need to do this again later this year. We never pitch it to the business that way, but that's what is inevitably going to happen if you try to address all of your technical debt all at once. It's the kind of the grand rewrite of the business. We were like, hey, we made such a mess with this first system, but we learned all the problems we could make. So we're going to make a new system to replace it. And now you have two systems to maintain, and now they're trying to keep up with each other. The new one is trying to outpace the old one. And eventually you end up with just a much bigger mess than you ever predicted. So if we have to change our technical lifestyle, what do we need to do to make that happen?

[00:15:33] Allan Stewart: One of the big things, in my mind, is just really around, how do you make this visible? How do you start measuring the health of the system? Once you start, once you figure out a way that you can quantify it, then you can start moving forward and set some goals, right? We see that, you know, in the health space, right? Physical, human health, we get these same kinds of things, right? We're setting goals for being able to do certain things and certain activities, or we're setting ourselves up with a particular diet because we know that we should eat the fruits and vegetables all the time and not only when we have scurvy.

[00:16:15] Dave Adsit: Right. Turns out I've never had scurvy. I guess I've eaten enough fruits and vegetables throughout my life to avoid it. Thank goodness, it doesn't sound pleasant.

[00:16:23] Allan Stewart: No. So the first way of making this visible that I have used that I think is interesting is creating your own tech radar. Mm-hmm. There are multiple ways that you can do a tech radar, but if you just take kind of the basic concept from ThoughtWorks, well, then you end up with these concentric circles and you have this idea of here are things that we like, that we adopt, that we want to do, and there's other things that are on hold, right? They're the stuff that we're trying to get away from. And it's pretty easy, in my experience, for a group of developers, right, to get away from, to look over their system, and they will very quickly label the things correctly. These are the things that we like and want to continue doing more of. These are the things that we dislike. And from that, you can kind of start getting an immediate gauge of where are you? Are there a lot of things that are on hold that exist in your system still, but you haven't gotten rid of it yet? It also helps with the directionality. Right. You can start to understand, oh, we're trying to get rid of this particular ORM, or we're trying to adopt this new pattern in our code. And having something like a tech radar, or there are some other things like architecture decision records and things like that that can help with that directionality. But the tech radar is also simple. You can throw one together on a whiteboard or a digital whiteboard, like Miro or something like that pretty quickly.

[00:18:08] Dave Adsit: Yeah, I really like the tech radar. I've set up several over different companies I've worked at, and they are very helpful in clarifying what is the core that we're doing? What are things that we're investigating? What are things that we are moving away from? That's super critical. Every long-lived system that I've ever coded in has had at least three different architectural styles that people put in place over time. Mm-hmm. Right? At one point, we were just connecting controllers directly to the database. And then we read Domain-Driven Design, and now we're putting in place a service layer and a rich domain model. And then we got a little bit more in-depth, and now we've decided to have workflows or the script, the decision script or whatever, that says what it is that needs to happen when this endpoint is called. And in this part of the code, we're trying to expose all the entities directly to the web, and then we'll compose them in the front end using GraphQL or whatever. But in this other part of the code, we're doing backends for frontends where every page has a specific endpoint that it gets and a specific endpoint that it posts. And which one of those is the current strategy, and which one are we moving away from? And that's always a discussion that you have to have ongoing, right? And if you can have a way of recording that, then that's going to be a lot of fun. And if you're recording it, that everybody on the team can go look, it allows everybody to go a little bit faster and be a little bit better aligned. And if somebody disagrees, they can go argue with the tech radar and have a discussion at whatever your high-level architecture meetings are or whatever, rather than just going off and doing their own thing and introducing the third, fourth, or fifth way of doing code in the system.

[00:19:56] Allan Stewart: Yeah, and I think as you were talking, I was thinking about how you should probably want to continue to add more. Yeah. More ways of doing things, because you are improving, you are understanding your system better, the world is changing around you. So it probably makes sense that you at least want something that you're working towards and the old thing that you're moving away from. I mean, not for everything always, but that sense of continual evolution, that we're getting better at the software, it's getting better fit for our company, or business, or whatever we're doing, and we're getting better as our craft of writing code. And so you want to have that change, but you just want to limit, ideally, you want to limit how many of those are in play at any given time. Because even if you know the directionality of it, knowing that there are seven old ones that you're trying to get off of to move to the new one, well, that tells you right off the bat something pretty significant about the health of your system. Exactly.

[00:21:03] Dave Adsit: Yeah, I would say it's not a very healthy system if there are so many architectural patterns that nobody knows what the right one is to use at any given point. So having this tech radar, making those things visible, actually helps increase the health without even actually having changed the system yet. But there's a bunch of other things we can do to actually start measuring and fixing and improving our system. One of the really popular ones for the last few years are the DORA metrics, which come out of the DevOps Research Association. Is that right? And those are metrics around deployment frequency, frequency of failure rate, the mean time to recovery and the lead time for changes from when code is committed until it is in production. I think we've probably all worked on systems where you commit code and you grade, you create a back end. You create a batch of code that's ready to go out. And then by the time you've said, okay, this is dev done, whatever that means, it could be a week, it could be two weeks before it's actually deployed into production. I worked on one system where we intentionally chose to have annual releases. That's a long time between when a request for change is made and it's actually seen by the users. And so these DORA metrics are a way for us to understand system health. And they're actually rigorously researched and they correlate very strongly with team success, product success, and business success. And so I would say if it's taking you a year to make a change in your production environment, that's probably not a very healthy system. I guess I can imagine a couple of cases where that might not be true, but I think that in general, if it takes a long time to deploy a team, it's probably not a very healthy system. And so if you make a change, you are not operating in a very healthy system, or you are low in responsiveness health, reactivity health, right?

[00:23:12] Allan Stewart: Yeah. Yeah. And it's nice that those DORA metrics are all fairly easy to understand, right? Sometimes there's questions about, oh, well, where do we measure this, right? What is the cycle time versus the tack time versus how do you put these measurements into play? Yeah. But there's also a lot of resources to help you work through that and be able to measure it. And one of the things that I've liked about it too, is that even if you don't go all the way to fully automated metrics, some of them are fairly easy to understand. Like you were just talking about deployment frequency. You can get a sense of that. Even if you don't have like an actual measurement, you can know it's like, well, we usually deploy every day. And if you know that, Versus well, we always deploy monthly or at the end of every sprint, which is probably two weeks, maybe a week. You immediately start having an understanding of, oh, well, this is where we're at. Even if you haven't gone and actually tracked. These are all of the deployment markers. And what is the mean time between them. Another way that you can think about technical health. That I. that I just barely thought of remembered. The idea of fitness functions from evolutionary architecture is an interesting one where you can go in. The idea of a fitness function is basically you're deciding something that you want to measure and a threshold for how good it is. And then the fitness function is ideally something automated, but definitely something that you can measure regularly to let you know where you're at. So measuring things like code coverage on your tests or cyclomatic complexity are examples of something that you might care about. Or there might be some architectural boundaries that you define, like certain pieces of code are allowed to talk to other pieces of code, but we care about the directionality and the boundary between those layers. And if those boundaries are circumvented or broken, then you can get an alert and you know, oh, this is something that has regressed. It is less good than it was before. And on some of them, you can also set it up in kind of a ratcheting fashion. So right now, maybe we have really low code coverage and we want higher code coverage. And so we find out where are we at? And if we go below our current, then that's, that's a problem. But as long as it keeps going up, then we're okay. And we used to be at 20%. And now we're at 30% and we ratchet it up. So now if we drop back down to 20, it'll alert us rather than letting us kind of slip back down into mediocrity.

[00:26:18] Dave Adsit: Yeah, I really like to think about fitness functions as taking your, your abstract concepts, like your illities, right? The scalability, reliability, recoverability, whatever, and putting concrete numbers around them, right? So instead of saying, we want a highly available website, the fitness function will say something like, we want a website that is available 99% of the time between, or 24, 7, 365. And so now I can measure that for the system. And if I fall below my target threshold, now I know that my health is dropping. I need to put in work to improve the health of my reliability. usually if you say we're going to target 99, most people aren't going to be super happy with that because that's a lot of downtime per year, right? People are going to want to target things like 99.5, 99.9. I mean, I don't think that most of us need to target 99 or five nines, but I would sure be unhappy if AWS decided that S3 buckets only needed two or three nines of reliability. Yeah. Right. And so the, I like to think about the fitness functions as taking an abstract concept, anything abstract like that, like reliability, availability, scalability, and making it concrete and putting a test around it. So we're looking at it on a regular basis. So we know how well we're doing against that target.

[00:27:44] Allan Stewart: And the connection to the health is right there in the name. It's a fitness function.

[00:27:49] Dave Adsit: That's right. I know these servers need to lift more weights because they are not handling requests well enough.

[00:27:58] Allan Stewart: I like that too, because it's, it's, it's a fitness like your health, but also fit for purpose, right? Fit for purpose. Are you actually healthy enough in the way that you need to be healthy for your purpose? Right.

[00:28:12] Dave Adsit: Yeah. It's a huge waste of money to make a system that's an order of magnitude more reliable than necessary. When we talk about increasing the number of nines of reliability, every additional nine is basically, I mean, I'm going to say 10X cost. I don't know exactly how much it's going to cost in every, every instance, but it is a substantial cost to increase the number of nines of reliability you have. And beyond a certain point, beyond a really, a reasonable point for your user base, it's unnecessary. I'm reminded that people like to use the term real time. We need real time reporting. And when we, as engineers start thinking about real time, we're like, oh man, hard real time. The system has to crash if it's not hard real time. How am I going to build a hard real time system? And then you go back to the user and you ask, what do you mean by real time? And they're like, well, it has to be accurate within about a 24 hour period. It has to be up to date as of end of day yesterday. Like, aha, I can build that system. I don't need to go into the hardware to make that system happen.

[00:29:15] Allan Stewart: Yep.

[00:29:16] Dave Adsit: Right. So understanding and having a good understanding of what people need across the system is really important. And so it would be a waste of time and money to build a system that is hard real time. When, what, somebody really needs to do. What somebody really needs is reports that are accurate as of end of business yesterday. So one of the other concepts that we've talked about a lot and found a lot of value in is scorecards. We've used scorecards for security, for general system health and a variety of things. And I actually really love the health scorecard you've put together for the company you're working at, Allan. I'd love to hear more about it.

[00:29:55] Allan Stewart: Yeah. So a while back, I discovered that, uh, having some kind of simple system helped me communicate better with the non-technical folks. Um, and I also found that it's a lot of work to set up some of these metrics, right? So, um, they're, they're very useful. So like we talked about the Dora metrics, they're very useful to, to get you a sense of where you're at and you can measure them, but it can be difficult. Fitness functions are a really cool idea that you have to implement. Right? There's, there's work to be done there. And depending on the nature of your company, that can be difficult. I've worked at a couple of smaller startups where it's really hard to justify. You know, when a lot of the system is currently on fire and there are problems that are affecting the day to day, it's, it's harder to justify, Hey, we're going to go around setting up some automation for these fitness functions. so yeah,

[00:31:11] Dave Adsit: yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah,

[00:31:24] Allan Stewart: down. Exactly. So in the scorecard that I'm using at my current place of employment, I broke it up into some categories that would be recognizable and hopefully somewhat understandable for people outside of the engineering department. So I've got a whole column of just features and I broke it. I even broke that up into these are the key features that are important for our business. These are the supporting features that help our business along. Some of those are just even like the checkbox kind of features that, well, we have to have this because our competitor has it, even though it's not our distinguishing feature. So I've got a column for those. I've got a column for different components. These are the deployable things. And this is a little bit less obvious for people outside of engineering, but we're still a pretty high level. So here's our server API. This is the state of our mobile app. This is the state of the web app. People understand those things pretty well. And so that's my second column. And then I've got a third column for non-functional requirements. So things like accessibility, data integrity, maintainability, security, scalability, testability. And in each of those columns, I've listed out the various things. And I've broken it down. So like some of the components, right, for our mobile app, our deployment process, and the SDK that we're using, automated tests and code quality are examples of things that fit under the mobile app. And then we're just using a really simple kind of traffic light style system to share that with people. So yes, we could probably come up with a more quantitative way that would maybe be even, closer to reality. But I found that that's not really necessary for it to be useful. Some things are kind of hard to measure, like latency. And some measurements are hard to decide, like at what, how much test coverage do we need before we call this green versus yellow versus red. But what I found is that you can come to a consensus pretty quickly with a group of developers, especially on found that if you have a bunch of projects where you don't have as much time and money to invest in automating the metrics, it's often small enough that you can get to consensus pretty, pretty quickly. And the traffic light is easy to understand. If it's green, that's good. If it's red, that's bad, right? Red is mean stop in traffic, right? So you're not getting where you want to go. There's a problem. You're stuck on the freeway. And it's bumper to bumper traffic, right? Stop is generally not good. And then yellow is kind of the in-between. It's a cautionary state where you're telling, it's like, hey, this is probably affecting us negatively, but not so bad that we couldn't ignore it for a little while.

[00:34:36] Dave Adsit: Right. Yeah. When you have reds, then yellows probably get pushed to the background, you know, to the back of the to-do list. But if you manage to get to a point where you've addressed all of the things that are actually on fire, maybe now all the yellows are the most important thing to focus on to increase your overall system health. And I can honestly see a case for, hey, this is a very infrequently used component. And the fact that it's yellow and not super healthy is kind of okay, because we don't really care that much about it. It's not used very much. It doesn't make very much revenue. It's not a priority for the business or for the users at this point. So let's just not address it. Let's, instead, work in an area where we can add more functionality and hopefully do it in a way that's healthy and overall increases the amount of green we see on our scorecard by adding new, good, well-maintained functionality of whatever sort. I really like the fact that you've reduced your scorecard to a small number of statuses versus having a more granular, like 100-point scale. What percentage healthy is that? Well, I mean, we could spend all day arguing about, is it 56% or 57%? And it really doesn't matter or help at all to know that. And then we also argue, at what threshold of percentage does it change color? Yeah, right. I would say maybe an exception is if you're, okay, so I have used a security scorecard in the past, and a lot of the parts of the security scorecard would be well-served by... By having a red, yellow, green, right? But I do remember the security person I worked with said, hey, I have to report this to the board every month or every quarter. And so this one little metric right here, it's percentile, or it's percentage, percentage healthy, whatever that means. And so every quarter, I just tick that up one. So it looks like we're making a little bit of progress. None of the developers complain if it went from 56 to 57 to 58 to 50. Because they all know it's kind of okay, but not fantastic. But he's like, but I can show progress on it. And that makes certain people feel good about the investment they're making in security. Yeah. A security is one of those things that is a huge space that you also have a lot of potential for tracking health in different ways, right? What's our password security? Well, if our passwords are stored in plain text, it's probably red. Or double red. Do you have that as a concept? Black hole? Could be. Yeah. You might need to add black for it. It will kill us. Yeah. But if we're using like, I don't know, an older password algorithm, that might be orange or yellow, right? Like it's fine. It's not hardened against GPUs, but we don't really have time to invest in implementing bcrypt. So we're just going to leave it because nobody's going to get our database and try to crack it with GPU. They certainly already have rainbow tables for the low value target that we are. But I could see, you know, if you're working your way towards higher and higher levels of security, having a security scorecard that talks about all those, the various things like, what are we doing around security training? What are we doing around penetration testing? What are we doing around code security and code analysis? All of these types of things become important to your overall security profile.

[00:38:23] Allan Stewart: Yeah. And like you're saying, it's easy to add other states, right? So, you know, do you need a double red or a black or an orange? It doesn't have to be red, yellow, green.

[00:38:34] Dave Adsit: Like we have so many reds, we have to create a distinction between red, red and orange red. It's not as bad as the red reds, right?

[00:38:44] Allan Stewart: Right. And, and it goes back to this concept of understanding your audience and, and what makes it useful and what makes it easy, right? So you could just do a numeric scale, you know, maybe it's just like a simple, like one to five, kind of like the Def Con style. You just have to decide at what level are you willing to deal with it? Because you don't want to get into that place where it's like, well, it's a hundred point scale. And now we're arguing about percentage points or like decimal. Yeah. You don't want an unnecessary precision. This is just, like, especially in this, this state where you're not automating the measurements, but you're just getting kind of the feel of it. It's nice to be able to just quickly go through and say, yeah, this is good. This is bad, but in a way that makes everybody could just understand, right? Especially the people who aren't engineers, they don't write code. If they can look at the score sheet and say, oh, there's a lot of red and yellow in there. And that makes me uncomfortable. That's where we want to get. Yeah. Right. Or if they look at it and they say, oh, hey, look at the progress that we've made over the course of a year, we had a bunch of things that were red and yellow before, and now they're green. They feel good. And hopefully there are consequences that go along with that. As you make the technical health of the system better, if you're doing it right, it should do things like improve your feature velocity. It should do things like improve the scalability and the reliability of your system. And so people will see that. And if they can draw that correlation from it was more red and yellow before, and now it's more green. And also I feel like my customers are happier, then they're going to trust in that system more and you can continue working on it.

[00:40:35] Dave Adsit: Well, and one of the things I think about there is the reason why debt was such an appealing metaphor is because business people understand debt, right? You take on debt. Okay. That's bad. Now our revenue is reduced by our debt, right? We didn't make profit because we still have all this debt. Okay. So that's well understood. And debt is a way to basically manipulate the emotions of the rest of the business because they don't understand what we're doing. It's very complicated and very technical and you require specialized training, just like a lot of us don't understand accounting. So if we're using debt to incite fear, that may not be the best way to collaborate with other members of our company, other people. And so, but I do think that there is a benefit when we start talking about scorecards and people can actually see it and then they can come to their own conclusions around, hey, it says right here that card processing is red. Why is card processing red? What's going on there? Can we have a discussion about this? And then maybe I want to invest in making that green or at least yellow, because as opposed to just saying, hey, we have to do this big thing and you don't get it. We're not going to get to have a decision on it. And we're mandating that you let us take time to fix X, Y, and Z. And then at the end, we might have actually broken X, Y, and Z a little bit. We fixed one thing and broke two others. Kind of what happens when you try to do these big refactor rewrite pushes. We forget functionality that people actually needed. But by making things visible with a scorecard, we can actually bring the interested non-engineering non-developer parties into the conversation so that they can help us make priority decisions. They might say, hey, look, I know this one app, you've flagged it as red. I don't care. No one uses it. Not going to invest in it. Or people use it. We've intentionally decided to strategically move away from it. And so leave it red. We're going to replace it with something else or we're going to deprecate that part of our business or whatever. This part over here that's yellow, super critical to our success. We need to make it green. How do we change this yellow to a What do we need to buy? How much time do you need? I feel like you can bring people into the conversation better by making things visible than just by being the expert in the cloud or on the top of the tower, yelling down orders at the people below. Yeah. And it comes to mind that

[00:43:05] Allan Stewart: you can also improve your scale beyond green, right? So I don't know what that is. It's double green or it's blue or light green and dark green. I don't know. But you can come up with something. And that might also be a way to strategically look at your system and say, hey, we want this part of the system to be really good, right? Not just passable good, but like great, maybe even world class or like, because this is our differentiating feature. And we want to be able to iterate on it so quickly. We want it to, you know, work flawlessly. We want it, whatever it is. That makes a difference for your business. Well, that could also be a strategic choice because, you know, getting into that, that concept, I think most, we talked earlier about like, are you just generally healthy or are you healthy for a purpose? And so that would be one way to start looking at that. And so they could say, okay, things that are yellow and red, well, we're going to invest in those because we don't want to be in poor health. But then beyond that, we also want to invest in, in certain strategic areas that we're going to say, hey, it's, it's not good enough to be okay, or just healthy. We want to be strong in a particular area that fits the need of the business.

[00:44:32] Dave Adsit: Yeah. So I guess all things said and done, I really like the idea of talking about health instead of debt. And maybe it's in addition to debt, but I do think that you can have a more positive conversation. You can make more progress. You can basically, turn around the perception that engineers are constantly pessimistic and start being more optimistic and working with your business partners to deliver on the needs of customers, users, et cetera. And I like the idea of using health a lot. I think we've both seen success by using this at different companies we've been at. And I, I would just encourage people to give it a try and see if it doesn't change the way conversations go when you're talking to people across your company.