Crafting Code Podcast

[00:00:16] Allan Stewart: Welcome to the Crafting Code Podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately I've been thinking about how the problem of juggling priorities has intensified as AI creates things faster.

[00:00:33] Dave Adsit: I'm Dave Adsit, an engineering leader, and recently I've been thinking about the challenge versus content volume trade-offs in retro games versus modern games.

[00:00:42] Allan Stewart: Our topic for this episode is mistakes and experimentation. As I think about this topic, the big thing that immediately hits me is that the consequences of failures are very different when you're doing an experiment versus if you just made a mistake. Outside of the context of an experiment.

[00:01:01] Dave Adsit: Yeah, that's definitely true. And the, I guess part of the problem could be just how prepared are you for something to go wrong? If you expect everything to go well and it doesn't, you're probably not as well set up to handle that as if you are anticipating the possibility of something going wrong. So we've talked about this before and we, there is the obvious question comes up. It's like, what is the ideal amount of failure to maximize learning?

[00:01:33] Allan Stewart: The answer is 50% because that's at the point where you've maximized. You don't know, right? At 50, 50, it could go either way. If it's anything other than 50, 50, you, you already have some, this additional knowledge that is affecting like the statistical likelihood of, of an outcome. And so you're less likely to learn because it's just going to confirm. A bias, right?

[00:02:00] Dave Adsit: You've either done too much prep and you are not actually experimenting at the you're experimenting too late, or you haven't done enough prep and you're more likely to fail than succeed. Right? So the optimal time to run the experiment is when you have a 50% chance of failure, a 50% chance of success. Right.

[00:02:25] Allan Stewart: If what you want to do is. Yeah. Learn. That's not the goal that you're always after is the learning. Sometimes you are having a goal of meeting operational demands for your customers and making sure that the system is online and, and scaling. And recently with so many providers having a lot of downtime, it's a pain that a lot of us have felt when, when something isn't online that we were depending on and just expected to be there. Right. And this is specifically, if you want.

[00:02:58] Dave Adsit: To learn as fast as possible. If you are in an early state of a new project or feature, you're trying to discover something. The fastest possible way to learn it is when you put it out to customers, when there's a 50% chance that it's not going to work, they're going to reject it. They don't want it. It's not going to do what it's supposed to do. So, you know, given that we have this, if, if we have a goal of learning and we are using experimentation to learn, what about failure? What does it? Mean? What does failure mean? There's this idea of a celebration grid, right? That you could either screw up and get lucky and which case you don't learn, you could get unlucky and something goes wrong when it probably shouldn't have, and you're not going to learn much. But if you are in the middle of that space, where it's like you could fail and learn or succeed and learn either of those is learning, right? So what then becomes a. What is a failure? How do we measure failure?

[00:04:00] Allan Stewart: Yeah, it's a, it's a good question. And I think it, it kind of depends on that mode that you're in. Uh, right. If, if you're experimenting and the experiment comes off in a way that you weren't expecting, yeah, you could, you might classify it as a failure. It didn't work, right? Like what, what we tried was unsuccessful, but if the goal was learning, then that's the experiment was successful at least. Right. And so. I think for measuring failure. It really comes down to the question of, okay, when you're outside of that experimentation world, things go wrong and you have to address it and you have to understand like, okay, well, what was the degree that it went wrong? Did a couple of customers get a weird experience for a couple minutes, but they were able to move on was an entire site outage. It was all of a customer's data deleted and it's gone, right? Like there's, there's different. Magnets. Magnitudes of the impact of that, of that failure and, and what happened, what is the outcome of this unanticipated event?

[00:05:07] Dave Adsit: There's a concept in product development of building to learn where we want to maximize learning and ship early. And we want to have that, that 50% chance that this new feature isn't adopted and then build to earn. So we've built something, we've learned what the market wants, and then we build something. So that we can. We can deliver at scale. And those are different goals, different strategies that apply at different times. And failure is a different thing in those two, right? If we're doing build to learn and we don't learn anything, that's our failure. If we're doing builds to earn and we have a target, an SLO of like nine, nine, nine, five, and it goes down for an hour a day, that's a failure, right? It. It. It. So it depends where we're at. We are in the stage in the phase, right? And so we want to think about, be aware of that first of all. And, you know, I think in this case, we're mostly talking about what we're doing in a build to learn phase of software development. If we're talking about mistakes versus experimentation, you don't want to experiment that much on your four, nine system, right? That's not a place to run experiments.

[00:06:24] Allan Stewart: Right. So if we. If we dig into that for a moment and just talk about failures in general and how we address them, because this is a real thing that happens all the time in software development and it is, it is fiendishly difficult to get rid of, right? We talk about some of the advantages of mean time to recovery as an acknowledgement that failures happen. Bugs will be shipped to production. Things aren't going to be perfect. And this is a mechanism for. Redressing that as quickly as possible, as opposed to the meantime between failures, which is also important, but very difficult to get rid of. And so if we take it as a given that there will be failures, I think there's a big question around how do you approach the failure? Are you going to, and I've seen lots of different ways of this being done. Are you going to blame others? Are you going to shrug it off? Like it's no big deal. Are you? And say, well, this is, this is just part and parcel with software development. Are you going to freak out or perhaps more importantly, are your leaders going to freak out and come looking for, you know, the blood consequences, severe consequences that they, that they want to take out on you?

[00:07:50] Dave Adsit: Well, I mean, the extreme comes from one of uncle Bob's books. It's called the clean coder. I believe he talks about, Hey, if you cause a mistake and it costs the company money, you better whip out your checkbook. Right. We don't do that.

[00:08:04] Allan Stewart: No. I like the idea of it. Like, as far as in the concept of crafting code that we would do a really good job and we tested and that we believe that it's working. And yet I also like my company acting as the form of insurance for my personal mistakes. Because there's a lot of things going on and it can be, if I had to whip out my checkbook, I would behave differently. And I don't know if it would be a hundred percent better. Right. Like in some ways I think it would be better. Like I would work very hard to ensure that I always do the practices that I don't cut corners. And yet, especially for early phase startups or a place like this. Places where you're trying to figure out product fits, or I want to introduce a new feature onto our existing platform where there are a lot of things that can go wrong. There's also a need to go quickly. And I think it's because we're dipping into that learning space, which we'll talk about a little bit more in a minute. But it makes it very difficult for me to say, yeah, I'm going to, I'm going to whip out my checkbook and pay for this outage because I forgot to do a thing. Because of all the pressure that I have.

[00:09:39] Dave Adsit: percent chat session close heavy floors planning shade Build choosing Bubble Bubble instead of getting, you know, like, and, and that happens when you take on the responsibility of founding your own startup and running your own startup. It's a big reason why a lot of us choose not to do that and instead become employees. We don't see the full upside of the business going to the moon, but we also don't take on the same risk. Yeah. So setting that aside, we're not going to whip out our checkbook and write the company a check or the customers a check when the software goes down for a few minutes. What are we going to do?

[00:10:26] Allan Stewart: I think it's important to not go too far on the other end of the spectrum as well, because if we just, if we're continually blaming somebody else, like that, that doesn't get us anywhere. It doesn't help. I've worked with some people who just shrug it off and say, I was like, Oh, this is just software. It happens. And there's some truth to that, but I feel like there's also this excuse, right? Like that. I'm just, putting in my time. All I care about is getting my paycheck or I couldn't be bothered to think something through. I couldn't be bothered to like examine the potential consequences of a change. And so I've, I've seen cases where things happen and sometimes it's, sometimes it's purposeful. I've seen a case where we increased our infrastructure spend significantly for no reason, just because it was, it was, a mistake. It was a misunderstanding on somebody's part and it never got rectified. But then it also happens on accident, right? Like just today I found we had a storage bucket in our product that was accumulating a bunch of stuff that was meant to be emptied and we were paying for the storage. And so it was just a mistake that needed to get cleaned up and it wasn't malicious. Nobody, it wasn't that nobody thought about it. It was just an accident.

[00:11:50] Dave Adsit: Yeah. So I want to work in organizations that are constantly improving. And you mentioned both meantime to recovery and meantime between failures. I want to be improving on both of those metrics at all times. You know, anytime we have an incident, that's an opportunity for us to pause and look at the root cause and put in place mitigations to prevent that thing from happening again. I don't want to be having the same. I don't want to be having the same mistakes over and over and over. Yeah. I want each of our mistakes to be new and interesting, not boring and repetitive, right? So things that we've used many times in the past include blameless incident reviews. You get everybody around a table. I like to do it between 36 and 72 hours after the incident. I know some people like to do it the day of with and have a, a report within 24 hours. I like to intentionally take a pause, let the emotion go out of the incident and then get everybody back together while it's still fresh enough that we can attack the problem logically. I don't know personal preference on that one, but I want to get everybody together and I want to do a deep analysis of what caused this problem. What were the root causes? Let's go five wise, 11 white, go back as deep as we need to until we what it is that we could put in place that would prevent this problem from happening again.

[00:13:26] Allan Stewart: So related to that idea of guardrails, there are a number of practices that I think are really important. And it's the reason that I do them. And it's part of the reason that I think it becomes part of the discipline, right? Both that we are disciplined enough to, to do the right thing, but also the kind of industry discipline. So Yeah. testing an exhaustiveacak nadIST tuning scene setting session sessionах tests crash command trace Where To Dist τον involves like a research or implementation spike to test something out, to try something out. And I think these are all great practices to help us get those guardrails into place. Because if we have good observability, we can see how things are going. We can react better. We can understand why a failure is happening, when it got introduced, what are the likely reasons behind it. If we have feature flags, we can see a failure and then turn it off and say, oops, let's figure that out. Kind of along with your blameless incident review, right? It was like, if we turn it off, we can cool down for a minute and say, okay, everybody calm down. The problem's not happening this minute. Let's think about this and get outside of that. Oh no, panic, immediate action must be taken where you're more likely to fumble and make a mistake and tone that down and say, okay, I know what I need to do. I know what test I need to write. I know. Where to go and look in our logs or graphs to see what went wrong. And then having those guardrails in place makes it less likely that you're going to run into a problem or that the problems that you run into will be less severe.

[00:15:33] Dave Adsit: Yeah. I definitely like having all of those practices in place as both a prevention and a mitigation strategy for the types of issues that we tend to see in software systems. So, you know, one of the things that I, that came to mind as we were talking about this is that sometimes we make mistakes that don't rise to the level of a software incident. And that's where tools or practices like those found in radical candor come into play, pulling somebody aside immediately and saying, Hey, this and this and this, this happened. You said this, this person, you know, do kind of that situation behavior impact analysis with someone. This was a situation, the meeting we were just in. You said this. It hurt so-and-so's feelings. I need you to change your behavior going forward and also go apologize. You know, those are some of the things that we do on a person-to-person level that don't quite rise to the level of a software system incident and certainly don't rise to the level of an experiment. I don't need you to go into a meeting and see what happens if you're a jerk to your coworkers today and not a jerk to your coworkers tomorrow.

[00:16:43] Allan Stewart: Not an experiment that we need. To do. No, no. Yeah. For those interpersonal things or even just the structure of how we work. Right. So I've noticed that, you know, you are far more likely than your peers to push a commit that fails the build pipeline. What can we do to address that? What can we do? What behavior can we change on your local machine so that that's less likely, right? Like those kinds of discussions. It hasn't raised up. To an emergency, but it is impacting other people or it is impacting the software.

[00:17:21] Dave Adsit: Yeah. And I would definitely categorize those as mistakes or failures, right? Yeah. In our concept of talking about mistakes versus experimentation, like there's interpersonal mistakes. Yeah. So let's move on and talk a little bit about experimentation. What is the difference between an experiment and a mistake? We talked a little bit about build to learn versus build to earn, but what makes something an experiment? Yeah. Yeah. Yeah. There may be more, but these are the three that come to mind for me. You need to have a prediction. If we make this change to the software, it will affect churn rates like this. You need to have a measurement. Churn rates are currently 3% and we want them to go down to 2% per month. And we need to have reversibility or we're not really experimenting. We're just changing stuff. If I make a change and I can't go back, that's not an experiment. That's just a, you know, I'm now on a new course.

[00:18:13] Allan Stewart: I like these three. During carryover effects your 물ALK nodes across methods step request Necessarily. If you've taken the time to actually do these things that you have a prediction and ideally you have communicated in some way, this is what we're doing, what we're going to try, what we think is going to happen, how we're going to tell whether it happened or not. And can we back it out? Then you can actually say, yes, we were really in an experimental phase and we were not just deceiving ourselves or trying to deceive others.

[00:19:13] Dave Adsit: Right. Yeah. That's the trick, isn't it?

[00:19:15] Allan Stewart: The prediction part I think is important too often. We just assume it's like, oh, this is the next feature. And so, you know, whatever it is I'm doing, like, it must be right. We don't always stop to check ourselves and be like, oh, well, let's, let's predict. What was going to happen. And even less often, unfortunately, in my experience, do we ask ourselves, how are we going to tell? It's easy to see if the thing is working from a technical standpoint. It's like, yes, the API did return a 200 with a JSON payload. And so therefore it was successful, but we can probably do better in those measurements of what were you really trying to achieve? And did you, did you actually do it?

[00:19:57] Dave Adsit: Yeah. And I think a lot of experimentation. Is around the product life cycle is like around the product outputs or sorry, the product outcomes. Like we want to change you some kind of user behavior in some way. Sometimes you're doing experiments on, can we create an API that can handle a hundred thousand concurrent connections? Like, okay, well, that is definitely a very important technical challenge, but you're probably not going to do that unless there's a business justification for it. That would be one of those areas where you are unnecessarily overburdened. Building or over-engineering something. If there's no chance that the users are going to get to that, the usage is going to get that high.

[00:20:36] Allan Stewart: Yeah. And as long as we're talking about these criterion with the reversibility one, I think that is also so important that going into an experiment and what helps us differentiate that. Yeah, this was a real experiment versus a mistake or a failure that happened is that we had a, we already had a plan of what we're going to do about it. And if we're. Writing at that 50% could fail, could succeed. That maximizes our learning. Then we got to put ourselves into the mindset of when it goes wrong, not if, but when it goes wrong, because odds are it will, if you're going to do more than one experiment, you're probably going to get a failure. Half the time. How are you going to address it? What's the, what does the reversibility look like? Is that as simple as a feature flag that you turn off? Is it tearing down some code and saying, Nope, this, this feature really isn't working. And we've, we've got a plan for how, how we take it away from the users who have been trying it. That forethought that we put into, how are we going to deal with it? Elevates it from mistake to no, this is part of the plan. And instead of it being a panic moment, well, we know what to do.

[00:21:52] Dave Adsit: Yeah. Well, and I, one of the things I like to think about is that this is not new. This is, these are not new concepts. This is not a new way of. Thinking in software or business development. There's been things like the, the plan, do check act or plan, do study act from the, the Deming trainings, the Deming school of thought that have been around since the, the 1960s. Right. These are the planning, what you're going to do, doing, doing it, studying the results, and then taking action based on those results. That is very much design and experiment builds to learn. So you can measure the outcomes. Now build to earn based on what you learned. Right. It's the same concept and it's been around for a long time. There's the, the Heilmeyer catechism from DARPA, which is very much about the same kind of thing is like, is this risk worth taking? And what are we going to do if everything goes wrong? And so these are, these are concepts that are well known in industry and should be considered in software as well. One of the. Things that I think has sharpened my thinking around this is the idea of thinking in bets, which is, you know, a book about someone who decided to go work or go replace their real job with a working by working on like the, the world poker tour, I think is where she ended up. Anyway. So thinking in bets, how do you make smart decisions when you don't have all the facts? And the thing is, is that you're going to have to learn to make bets of the real. Right size. And you're going to have to learn or plan ahead for what you're going to do when things don't happen the way you want them to.

[00:23:38] Allan Stewart: I think there's an important aspect of avoiding hubris there that we don't know. It more readily resembles everyday life. Honestly, we don't know what's going to happen next. One of the things that I think we enjoy about software and writing code sometimes is that it's easy to get into a space. Where we think, or we, we maybe even deceiving ourselves a little bit that no, we have control of this and we can write the code to make whatever thing happened that we want to have happen. We can make the button appear on the screen. We can change the colors. We can call an API. We can do all of these things. And, and it gives us this sense of control and oftentimes a sense of I'm doing this right? Like this, this is me. I've got control. I don't have to interact with anybody else. I didn't get into software to talk to people. But the more I learned about software development, the more I realized that you do have to have interactions. And so if you can add a dose of humility and say, Hey, I know I can't control everything. Definitely can't control what the customers do. I can't, I can't even control that the network is always going to be up. Right. We get away from the, uh, what is it? The fallacies of distribution. Of course. We say, yeah, things are going to fail, but it's okay. I recognize it. I'm planning for it. We've got an idea of what, what we're going to do. And then on the personal side, I know that I'm going to make some mistakes and I've I'm leading with that in my interactions with people that it's like, Hey, I've got enough humility here to own up to my mistakes and say, yes. I did screw up. and this is what we're going to do about it, rather than kind of passing the buck and saying, oh, well, no, somebody else, it's someone else's fault for the thing that happened instead of saying, well, what can we do better as a system?

[00:25:41] Dave Adsit: That's definitely the key for me. I want to work in and build systems where everyone feels comfortable experimenting and feels comfortable admitting and addressing and correcting mistakes that have been made so that we can improve the system even further and move towards our ultimate goals.

[00:26:03] Allan Stewart: And then the reaction is so important too, right? From everybody who didn't make the mistake, how do we react to that? There are the kind of horror story ones where developer accidentally deletes database and is immediately fired. Well, why did they delete the database? What did they do that was wrong? And if you fire that person, did you just fire the person who best understands what not to do, who had the best chance of setting up a guardrail against that same problem happening again? And now you're just introducing some other poor bloke there who's going to take the... You're next to be fired because systematically we don't take well to failure despite failure being the status quo. of software development.

[00:26:55] Dave Adsit: Yeah. Why did we create a system that allows that person to make that mistake if it's so catastrophic? Right. So in terms of mistakes and experiments, our goal in product development, software product development especially, needs to be to create systems and environments that enable experimentation and encourage experimentation so that we can find the right product to build towards our goals. and increase our product market fit. And we also need to build systems that accept, acknowledge, and address mistakes. We need to have socio-technical systems, people systems, that acknowledge that mistakes will happen and bring those forward, create the psychological safety to admit mistakes, discuss mistakes, make plans for mitigating and avoiding future incidents, of those same mistakes. And all in all, enable us to learn faster, go faster, and have fewer unplanned outages.