Crafting Code Podcast

[00:00:15] Allan Stewart: Welcome to the Crafting Code Podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately I've been thinking about how if I start playing Factorio Space Age, I may never stop.

[00:00:31] Dave Adsit: I'm Dave Adsit, a VP of Engineering, and I have been thinking about whether or not adding constraints to a game can make it more fun. In this case, the game is base Factorio, and the constraint is Ribbon World.

[00:00:45] Allan Stewart: Our topic for this episode is asking questions. Dave, why do we want to bring this up as a topic for our episode when we consider all these other things that we could be talking about? A continuous integration, or dependency injection, or socio-technical systems, and all these things. Why is asking questions important?

[00:01:08] Dave Adsit: Asking questions is just a very important skill for us to develop as software engineers. It's one of the soft skills or social skills that gets overlooked when we are developing engineers and when we're thinking about being better engineers. Right? We're talking extensively about context, the importance of context, and asking questions is a great way to get context. It's a great way to increase the awareness of the system that we're building across the whole team, right? Not just, hey, I need to ask a question because I don't understand this thing. For example, this morning I had to ask multiple times about a certain DNS configuration that I just, I didn't understand why it was built the way it was, and I needed to know because it affects the way we're going to do a deployment. But it also can affect other people, right? When we are honestly seeking information, it can make people more open to sharing that information with us. I don't know, when we worked together a while ago, I don't know if you actually ever noticed this, but I made a note, actually a physical post-it note on my desk to remind myself to every week in front of the entire dev team, say, I'm going to do this. I don't know. And it was always honest. There's a lot of things I don't know, but I would always say it out loud as a way to create some more psychological safety for people to ask questions, expose ignorance, et cetera, and talk about the things that they need to understand in order to do the job better.

[00:02:46] Allan Stewart: Yeah, absolutely. I feel like asking questions is a fundamental part of collaborative coding. So if you're going to pair with somebody, if you're going to do mob programming, ensemble programming, you need to be good at asking questions because it's, that's what kind of makes it, well, honestly, just more collaborative because otherwise you only have a navigator and you're just taking turns, everybody saying what to do and where to go. And as soon as you start asking questions, yeah, you're exposing your ignorance, but you're also opening up a lot of room for learning because oftentimes people don't know. Yeah. And, and I like what you're saying about how that affects people. When you ask a question, if you, if you say you don't know, I think that that's, that's good. And when you ask a question that exposes kind of that similar unknowingness or, or that you're just, you're wanting to examine or even challenge an assumption.

[00:03:48] Dave Adsit: Well, and if you think about the, the fact, the question exists, right? The question exists, whether you ask it or not. So asking it is the only way to resolve that question. If I sit quietly, I, I don't know why the DNS has to be the way it is. People assume that I do, and we don't talk about it and we move on and I never learn. It's the act of asking that actually opens up possibilities for us as a team, for collaboration, for building a better system, for writing better code, for just general learning.

[00:04:23] Allan Stewart: Yeah. And I feel like, like you were saying, people are, we're more open to an honest question. We do have to be careful about how we ask questions, just like how, if you are very accusatory in the way that you make a statement, it might raise some hackles. Similarly, we need to be able to ask good questions, right? So saying that I don't understand, you know, can you help me understand this is, is probably a more productive question than how could this possibly work? Yeah. You're thinking through thinking through thinking through thinking through thinking through thinking through thinking through thinking through thinking through thinking through thinking thinking through thinking through thinking through thinking through thinking through thinking around stating facts or opinions or opinions masquerading as facts, which can oftentimes shut people down.

[00:05:33] Dave Adsit: Yeah. I think that one of the things we talked about in prep for this is that asking questions is at the core of the scientific method. I have recently been re-listening to the fan fiction, Harry Potter and the Methods of Rationality, which is set loosely in the Harry Potter universe. It uses a lot of the same characters, but very, very different ways. And I was just listening to the part where Harry teaches Draco about the importance of asking a question and trusting the answer that you get and really introducing the scientific method. And I just can't help but think about how it was a very challenging thing to happen in the book as the plot was developing, right? It was very difficult for... Draco to accept the outcome as the answers came into his questions. And I think that sometimes we get into that mindset as well. So it's important to ask and ask with an open mind. But even if you can't have an open mind, you should still at least ask so that someone still has the opportunity to explain something or teach you or whatever.

[00:06:40] Allan Stewart: Yeah. Yeah. And I really liked the idea that we should develop a mindset of inquiry. Mm-hmm. Pretty much unilaterally. The people who get described as know-it-alls don't have a good reputation, right? The people who go around spouting, it's like, oh, yeah, they already understand everything, know-it-all. You know, sometimes they're respected in more of a, I guess, kind of a guru or a wise old hermit, perhaps. But going around, having that mindset of, hey, I want to understand this better. Why are we doing what we're doing? Why does this feature work the way that it does? Why are we doing, I don't know, whatever it is that we're doing puts you in a nice position where you are opening yourself up and you have that mindset so that when a new problem comes along, you don't just jump on it and say, oh, well, let's just solve it in the way that we've always solved it. But you actually step back and ask. So let's talk about some of the ways that we can put our questions. Into action. The first area that we talked about is just general learning. If you want to gain knowledge, then you're going to have to go look for things. And I don't know about you. It's very hard for me to learn things that are just like thrown at me.

[00:08:11] Dave Adsit: Yeah. Right.

[00:08:12] Allan Stewart: But if I arrive and I don't have any, I don't know, arrive at a classroom or I'm going to go. Just open up a random book or somebody gives me a book and says, here, you should read this. I might not be very engaged in it, but once I have questions and I'm thinking about the topic and I'm, and I'm wondering, then all of a sudden I learn a lot more.

[00:08:36] Dave Adsit: For sure. One of the things I was thinking about when it comes to gaining knowledge is that if you have a question, almost inevitably, another member of the group has the same question and. Someone has to ask it for it to get answered. There's a book that I found very valuable in my career and I recommended it to a lot of younger developers is the book apprenticeship patterns. And one of the, one of the patterns that you can follow as an apprentice or as a more novice software engineer, it probably applies to other fields as well, but especially in software engineering. One of the patterns is expose your ignorance, which is basically an opportunity to expose your ignorance. It's an opportunity for someone to help you, which can be beneficial to all parties. Right. I have a question. I ask it. Now you can help me by providing an answer. You can also, you know, guide me towards doing, having deeper knowledge and, and better skills. Right. I don't know how to do mocking. I don't know how to do a constructor injected dependency injection. Right. I don't know how to set up this library. Uh, how can you help me with this? Can you show me this skill? Can you basically help me develop the knowledge I need? And now that is beneficial to both of us because I get new knowledge and you get the opportunity to help me acquire that knowledge, which, you know, there's, it is one of the more rewarding things we get to do as a software leaders is we get to make software and we get to train software engineers. Yeah. And we train them through action and through, you know, doing the work.

[00:10:20] Allan Stewart: And it's inviting somebody who knows to act in the role of a teacher rather than, um, I don't know, a billboard, a tyrant, a tyrant, uh, because if you're asking about something, then it almost just naturally leads into questions of like, well, why or how, or which one, but if you're just told what to do, you might not. You might just get, you know, kind of follow along and they know why they're telling you to do it in a certain way, but that's not coming out in the discussion. It's just, well, this is just how I've always done it. Or there is a, or maybe there's sometimes there's like an important gotcha, like, um, going to use a particular framework and you need to know that you avoid a certain pitfall. If they're just broadcasting, just, you know, directing the work, then it doesn't, it. Often won't come out, but as soon as you've made it into a question, then it almost more naturally gets into those, those fundamentals of, well, why is it going to be this particular way? Or how do we do it instead of just being told, do this, do that, do that, do that, right? You're understanding the principles behind it.

[00:11:41] Dave Adsit: One of the members of my engineering leadership round table was telling a story about that recently, where one of the engineers that he had on his team had been around. Forever and knew all of the reason knew the system inside and out, knew how everything was done in the system and never really took the opportunity to explain why to anybody. He just told people what to do. And so, I mean, that's easier than doing the hard work of learning doing is easier than learning. And so a lot of the people on the team just kind of did what he said. And, and then he, uh, got a new job opportunity and left. And all of a sudden that whole team had to step up and they realized. They no longer had someone to ask anymore. They had to go dig into the code and research. And, and honestly, that's the most important time to ask questions is when no one on the team knows the answer. And you're just all assuming that everybody else does, but you won't know that that's the case unless you have the courage to ask. If no one asks, then no one will know that no one knows.

[00:12:42] Allan Stewart: I've basically experienced that story because just my own natural tendencies lend me towards wanting to just kind of take control. You know, strengths finder calls it, um, responsibility and deliberation, but I know that it, what that really means is control freak. And so I'll get in there and I'll write a bunch of code. I'll, I'll learn a bunch of things and understand, but I'm not always as good as I should be about explaining because I have to slow way down and be like, okay, well, hold on. Why is it the way? Let me, let me explain something to you that I've already figured. Out that I, or that I've already intuited. Um, and so, you know, this is a good, it's, it's a good reminder for myself because there's things that I can do. Yeah. Yeah. It feels like extra work, but there's reasons for it.

[00:13:33] Dave Adsit: Yeah. So along with learning, we've really talked about this already to some extent, is it just seeking more context around a thing? I would, I was thinking initially around, I want context around the domain. Like I can learn the skills of software development. But I also need to know the context of this domain. How do we handle this, this case? Is this a common case? How do we, is it an edge case? How do we handle it? If it's an edge case, how many customers will be impacted? And so having that kind of deeper context around what we're doing, how things work can be super valuable to us on our learning journey as software engineers.

[00:14:09] Allan Stewart: Yeah, absolutely. And I love the phrase seeking context, um, in, in our show notes, uh, it was written down. Originally as gain context. Ooh, but I kind of feel like gain context. That's like, that's like requirements gathering. Yes. Like it doesn't, it's they're the wrong words. I had a, I had a college professor who, uh, who used to say that it's, it should be requirements elicitation because it's a lot more like a dentist pulling teeth than it is just going around like a gatherer and, oh, look at this fruit that's fallen on the ground. Oh, I've just gained some context. Oh, I got a requirement here. No, you have to go. Go in and drill. And so I like seeking context. Right. Because it kind of gives it again, it's a mindset thing. It puts you in that mindset of, oh, I'm actively looking for something. It's not just like, oh, Hey, I happen to gain $5. No, it's like, no, I'm, I'm after this context and I want to understand why things are the way that they are because let's face it. There's just so many ways when, when we get down to code and these technical things, there's so many ways that you can do a thing, right? There's so many flavors of database and framework and languages and so on that there, there are plenty of ways that a thing can be done. And context is probably the most important one to help you decide which one of those ways should we choose? That's actually going to fit the, the situation that we're in.

[00:15:41] Dave Adsit: Right. I do like that as well. I think that's a huge improvement because seeking is a much more active. Right. Verb than, you know, the passive gaining context doesn't come to you without effort. And that's, that's what we've been talking about the whole time, right? You have to take initiative. You have to ask in order to do achieve any of these benefits that we've been talking about. Definitely. So another area where asking questions is critical is in retrospective. A retrospective is in my perspective, a critical. Tool of every software team. I tell my teams to have a retrospective every week. I feel like there is a huge benefit that they have to getting together each week and looking back over the last week while it's still fresh and trying to figure out what, what happened. You know, there's the basic questions that you'll find in the guide to retrospective. Like what did we do? Well, what didn't go well? What should we try? You know, these are some of the most. Basic ways to approach looking back over the last week and then looking forward to how we can improve as a team. But, um, we've had many much better retrospectives than that often led by you, Allan, when you were on my teams that you would put a lot more thought into just asking the three questions and seeing what comes up.

[00:17:08] Allan Stewart: Well, and, uh, I appreciate that, but a lot of the reason that I will shake things up in a retrospective and not just. Sit down with the standard three questions is that they become monotonous. Right. And you stop, you stop thinking, right. Uh, it's kind of like the, the daily sand standup syndrome where I've got the three questions or the three things that I'm supposed to say, and I've stopped thinking about it or, uh, and especially the meta context of, well, why are, why are we even saying this every time that we come? And so by shaking it up with different. Activities or different, different ways. Um, it kind of puts a different spin on the retrospective as you do different kinds of activities and you're not just saying the same old three questions every time. And that can oftentimes get people to think, um, like there's, there's different things that come out of it because they're thinking, they're thinking about those, those questions.

[00:18:07] Dave Adsit: One of the things that I've really enjoyed in retrospect, if you've done in the past has been when you bring in other senses, besides. Of. Verbal, like when we draw the last week as a comic strip, or when we look at the last week as a weather pattern. And obviously we're not going to talk extensively about retrospectives today, but retrospective is a great place for us to ask questions. For the same reason that we want to ask questions when we're learning, we need deeper context on how others are experiencing the work that we're doing together. So I've, I have found in the past that there are times when I think we've had a great week because I've been making all the decisions and doing all of the fun work and shipping all of the code. And then I find out that the rest of the team has not been having a great week because I have been. A bull in the China shop getting in the way of everything they've been trying to accomplish.

[00:19:05] Allan Stewart: It's so critical that we include questions in the retrospective and, and those questions really need to involve. Or. Or revolve around what is going to change as an outcome of this meeting. What is going to be different in the future because we had this meeting? Because otherwise it might just end up being a venting session or, you know, a brainstorming that goes nowhere. So, I mean, I do believe that there is a certain healthy amount of venting that can go on. Work can be frustrating, right? We call it work. We don't call it fun. So there's going to be things that we don't like about it. And being able to have a little bit of a emotional release from that can be valuable. But if that's all that we're doing, or we're just getting into brainstorming a bunch of ideas about things that we're not, they're not actionable. We're not going to do anything. It's not going to change anything. Then was it worth the time that you spent on that retrospective? Was it worth the money that you spent on that retrospective? Right? If you think about everybody's hourly wage and, you know, you get four or five, six more people into the same room, having a retrospective, that can get pretty expensive quickly.

[00:20:22] Dave Adsit: Yeah. And it can definitely be worth more substantially more than the cost, but it can also be worth almost nothing. If the outcome of the retrospective is that nothing is improving and everyone is just more frustrated and sadder about the way the work went or what didn't go well over the last week. Right? The venting sessions tend to not to be very productive. And they tend to be the types of sessions where the retrospective is full of a lot of assertions about the work and the coworkers and other teams that aren't pulling their own weight and how that's impacting us, blah, blah, blah. Right? Retrospective should be a team focused activity where we're asking questions about how we can improve within our team. Right? And sometimes that means we're going to go have a conversation with another group or the leader of another team. Or someone else in the organization. But we need to be focused on what can we try to improve our work? And it's a good place for experimentation. You know, we go back to that scientific method. I have an idea that this will improve our team. Let's try it for a week and let's see what happens. And then we can come back and say, well, I have a question. How did that experiment go? Sometimes it went really well and sometimes it went really, really poorly. And that's an opportunity for us to learn and grow and develop our, ourselves and our practice.

[00:21:49] Allan Stewart: Yeah. I think we've talked before about the best experiments have a 50, 50 chance of success versus failure. Yeah, that's the, that is, if you, yeah, it's like, it's like the optimum, right?

[00:22:01] Dave Adsit: Mm-hmm. It's the optimum opportunity for learning. This is, if this has a hundred percent chance of success, I'm not learning anything. If it has a 0% chance of success, I'm not learning anything. If it has a hundred percent chance. Okay. or if it has a 50% chance of success, it's a coin flip for me. I don't have enough information to know whether or not it's going to work. Right. And if you had a hundred percent

[00:22:21] Allan Stewart: that it was going to be successful, like in the, in the context of a retrospective, well, why aren't you already doing that? And maybe that's okay because you're, you're, you're, you know, you're getting together and you guys say, okay, Hey, the CTO keeps saying you should be doing unit testing. And we've decided that now we're going to actually do it. Right. So that, that might be a case where it, where it's useful, but especially as you get into experimenting, there's going to be a lot of stuff that you don't know. There's going to be a lot of times where people get into retrospectives and they have differing opinions on how we could make things better and trying out some of those ideas as experiments where you don't know, like this might be better. It might be worse. And even if you feel like there's not, you know, you might personally feel like there's not a great chance of it being better. Give it an, honest try. You might be surprised and it does make things better, or maybe it doesn't really impact you, but your teammate is just so grateful. It's like, it's made their life a lot better or they'll realize, Oh yeah, I guess that didn't work out so well. And then they'll stop bringing

[00:23:28] Dave Adsit: it up every time. And honestly, sometimes in software, things don't go that well. Right. And so then we have the opportunity for a blameless post-mortem, a blameless incident review. Where the key question that we have to ask is what happened? What, I mean, did the system go offline? Did the database explode? Did we have a network failure? Like what actually happened? How many customers were affected? And then in the, in this post-mortem, one of the critical tools that we want to use is the question, why, why did the site go offline? Why did we start, stop processing requests? Why, why, why, why? And one of the tools that we've used extensively and successfully is the five whys. It's basically a mechanism for seeking deeper context. I think that in general, our desire is to get out of the incident review or the post-mortem as quickly as possible and say things like, why was the website really slow yesterday? Oh, we were experiencing a DDoS attack. Oh, okay. We're done. See y'all next time. Well, no, like, but why didn't we have mitigation in place for that? And why, you know, we can go deeper and five whys is a technique for going five layers deep in context with the goal being that we're going to find the, the, um, the root cause, if you will. And obviously you can go back to, well, the big bang. The reason that the website went offline is because of the big bang and we're not going to go that deep. Right. But, um, often we can get deep enough that we can find a root cause that when resolved will prevent this type of incident from coming back again later. You know, you can, you can ask seven whys, you can ask 10 whys. There's no, nothing specifically magical about the number five, but you should ask at least several times in order to get to a deep enough level of context that you

[00:25:41] Allan Stewart: Right. I think the point there is that one why is not enough. You got to go deeper than that. And two or three, you might get there, but five is a, is a good number where if you, if you're really honestly thinking through the problem, now you're pretty deep and you're really not just scratching the surface or throwing out ideas, which I think is important because in a post-mortem or like, you know, post incident situation, it's usually not a fun time, right? Um, people can oftentimes get in there and if you're not asking questions, well, what else is there to do? But point fingers, start assigning blame and say, okay, well, this is so-and-so's fault. Like, oh, well, is it? Uh, I think back to, there was a discussion. I want to say some company had, some databases and there was an engineer or operations person, somebody who was trying to maintain them and ran a command that deleted an entire or multiple entire databases. And so that person was fired. Problem solved. Problem solved. Right. Except for ever do that again, except for they might, right? Because we don't know why they did it. We don't know how it happened. Yeah. I'm sure that the person didn't say to themselves that smart that morning, you know what? I'm going to do today. I'm going to just delete some databases so I can get fired. But they were, they were trying to do something else. And there was some kind of safety mechanism that was missing. And if you don't, if you just go in, you know, guns a blazing with your accusations, you're not going to get to that very easily. And you can almost every thing that could, you could say is somebody's fault. If you're willing to give them the benefit of the doubt, there's probably some systemic. Reasons behind it that you can look at and say, oh, well, you know, there are some mitigating circumstances because usually employees or, or contributors to a project or whatever it is, generally, they're not acting maliciously.

[00:27:56] Dave Adsit: Every engineer I've ever interviewed or ever had a one-on-one with has had a desire to have a positive impact on the organization. I don't think I've ever worked with anybody malicious. I would say if you want a. Very concrete example of that kind of a very drastic impact on an organization. At go to Chicago a few years ago, Brian Cantrell gave a keynote talk about the time one of their junior operators accidentally rebooted the data center. And it's available on YouTube, and I highly recommend it. It's very entertaining. It will give you a lot of, a lot of feels for the people who are in a system in crisis. But yeah, we've all done things like that. We've all fat finger to delete. We've all, I don't know. I released some code that had a hard coded setting once. And when it got to the staging environment, we couldn't even test it because I had hard coded it to use my local environment for something. And, you know, we learn lessons over time. And so having the post mortem, I think having it be critically a blameless post mortem is. Is creates an opportunity for us to learn and to ask questions. Not least of all, like, why did we set up a system that would allow this person to do that catastrophic thing? Like, why did, if you go back to Deming, he talks a lot about how the system has the greatest impact on the outcomes of the individuals participating in it. And, and then you have to ask yourself, well, why did the management, why did the leadership set up a system that allows a junior developer. The ability to delete. Delete. that is more resilient to mistakes. And so that to me is why it is critical for us to keep asking questions in the post-mortem until we actually have something that is actionable that we can do to improve this system and just make it more resilient, make it better for the people who have to inhabit it.

[00:30:17] Allan Stewart: Absolutely. It reminds me of a time where the team I was working on, we had made some code changes and we thought we had tested everything. We had done a lot of work to verify this new feature. And when it went out, it was using a memory database like Memcache or Redis or something. And we were using a shared database and it turned out that we accidentally had a issue with the number of open connections that it would make. And so it wasn't something that we had really considered because everything worked fine when we were just testing it. And we were testing it locally and in our staging environment, there's not that much traffic. So we never had a problem. But when it went down, when we overloaded the number of connections to the database, the other part of the system that was also using it stopped working too. And so we didn't just break our thing, but we broke another part of the system. And just thinking back to that, I don't know that we would have found that out in another way unless we really, really made sure that we're doing, I guess a lot of load testing as part of our changes. But it wasn't the kind of thing that we imagined because it's already taking load. It's taking way more load than we're putting on it from this other part of the system. And so it just makes me think that when you're in there asking questions, you want your postmortems to be full of the questions about, hey, well, why is this happening? Let's get to an understanding rather than, oh crap, am I going to be in trouble? Because like, I mean, I remember feeling that heightened sense of anxiety is like, oh crap. Oh no. Okay. What do we do? How do we fix this as quickly as possible? What do we roll back? What are we doing? And that's bad enough when you're in those situations. And so knowing that you can go in and you're asking those productive questions really makes a huge difference.

[00:32:16] Dave Adsit: Well, I actually have a question about retrospectives. Obviously you don't want to do your retrospective while the incident is still ongoing. Like you should not have paused to ask, why did we take the system down before you brought the system back up? Before you figured out that you were, before you did whatever to resolve the system outage. But then you start asking the questions about why did this happen, right? But when in your experience is the best time to do a retrospective? Is it later the same day? Is it within a week?

[00:32:50] Allan Stewart: You mean like a postmortem? A postmortem. Not necessarily. Not a retrospective. Not a retrospective.

[00:32:54] Dave Adsit: A postmortem, a blameless instant review type of a thing. When would you do a postmortem on an incident that you have already resolved?

[00:33:04] Allan Stewart: I like to do it soon, personally, because that's when I can, you know, because you do learn a lot. And in the vein of asking questions, I guess we could say that there's kind of an in-between point where you're in crisis mode, crisis resolution. There's a lot of, you know, a lot of important questions you should be asking there too. Ideally blameless there as well. Let's just get to the bottom of this and fix it. And so I find that I learn a lot in that process. So you take something, you learn, you apply a solution that is sufficient to, you know, get out of the crisis. It might not be your final solution. It might just be a band-aid. But then I like to do those postmortems soon while that information is still fresh. But... But... But with at least a little bit of a gap, maybe a couple hours, maybe the next day, so that people have a chance to kind of breathe.

[00:34:03] Dave Adsit: Let that adrenaline drain out of their system.

[00:34:06] Allan Stewart: Exactly.

[00:34:06] Dave Adsit: Yeah, I would not... I don't... I would never tell my team to do a postmortem the same day. I generally give the advice, do it between 36 and 72 hours after. So that's not even the next day. And I'm... I'm trying to figure out, like, what are people doing around the industry? What are people doing? What have you done in your teams? What are other people doing? Because I've talked to other leaders who say, I want the results of the postmortem within 24 hours. We had an outage. We resolved it. I want to know within 24 hours all the reasons why and what we're going to change in the underlying system to prevent it. And I've pushed people back a little bit further than that because I want us to do it soon enough. That we still have all the context of everything that happened. But I want to wait long enough that most of the emotion has drained out of it so that we're engaging that system two instead of system one. Yeah. I don't want anybody involved in the postmortem to still be in fight or flight.

[00:35:08] Allan Stewart: Yeah.

[00:35:08] Dave Adsit: I want them to be, you know, thinking deeply and slowly about the ways to improve the system. So I don't know. There may not be one right answer in this case. Context may still matter.

[00:35:22] Allan Stewart: Yeah, probably not. But I suspect that the magnitude of the incident is going to matter a lot. The more impactful it was, the more draining it is for people, the more time I would want to give before you do it. But if it's... That makes sense. You know, if you've developed that healthy culture within a company where it's just, well, it's known, hey, mistakes are going to happen. We're not going to beat you up about it. Let's just try and make the system better so that we have... Oh, that's a really good point. ...fewer mistakes in the future, or at least different mistakes in the future, then I think that can also mean that people, like they don't, their highs aren't as high. Yeah. You know, during the incident, they can arrive at that cooled off phase a little bit sooner.

[00:36:19] Dave Adsit: Yeah, I like that. And I was thinking about it the other way. I was thinking, if I'm in a system where most of the engineers do not feel psychological safety when things happen, when bad things happen, how would I correct that culture? And I think the first thing I would do is exclude certain people from the postmortem. I always call them retrospectives. It's not just in this podcast, but in general. I would exclude certain people who are known to be agitators from my postmortems. Just, you don't get to come. You don't bring the right postmortem energy. You're not helping. You're not helping us solve the problem. You're just being a dick and you're just casting blame on everyone in the room. And I'm not, and I'm not okay with that. But the, I would also want to like reinforce the concept that this is, first of all, blameless. The goal is to improve the system, fix the problem, not fix the blame, right? I borrowed that from somebody else. It's much better than any quip I would have come up with, but we want to fix the problem, not fix the blame. Right. Right. So we're going to treat everybody as though they're professionals with the best intention. And then we're going to ask a lot of questions. And some of those questions might feel very pointed, right? They might feel like they're attacking you, but that is never actually the intent in a postmortem or it shouldn't be. And if you're doing, if, if the intent is to hurt people emotionally or otherwise, then you're doing your postmortems wrong.

[00:37:47] Allan Stewart: Yeah. You got to avoid those ad hominem.

[00:37:50] Dave Adsit: Yeah.

[00:37:51] Allan Stewart: Questions.

[00:37:53] Dave Adsit: Well, so we've talked about five wise as a technique for postmortems, but it's also a valuable technique in other parts of the system and other areas that we discuss things, right? Like we, we would use five wise in a retrospective. We would use five wise. If we're trying to understand the context of a system, if we're, if we're actually seeking context, Oh, why do we do this? Oh, okay. We do that. Okay. Well, why do we do that? Oh, and what led to that decision? And sometimes the answer is just, it is because it is. It was the old. Ultimately the answer may be that's the only way I knew how to do it when I did it. And so that's what we've got. And now we all know better and we can fix it.

[00:38:32] Allan Stewart: I like to use five wise in the context of feature development, because it's very common to have ideas thrown into developers laps. Uh, usually in the form of solutions, like, Hey, we should build this, um, you know, it might be a CEO idea of the way. It might be, or the day or the hour, depending on, uh, depending on your CTO, um, or, or it might, there's a bunch of different reasons, but they often come in. Here's new work in the form of a solution and using five wise can often help us get back away from the solution space and into understanding the problem space so that we can come up with a better solution. Or even if it is a great solution. All right.

[00:39:52] Dave Adsit: and then you brainstorm the problems in that space. And then you come up with one to solve and then you brainstorm solutions and you pick one solution to implement and you try it. And you're right. Most of the time when we're, most of the time as engineers doing product development, we're brought a solution fully formed, but we don't know any of the context behind it. And so asking the five whys is a great tool there as well. Because sometimes, maybe even often, there might be a more straightforward or easier way to learn or to execute that feature or to solve that problem than the one that was brought to you. And sometimes the problem, the solution that you get is the best one, but it's better to know that when you start coding than it is to just assume it at any given time. And I think that's one of the things that as software engineers, as leaders in charge of building systems, we often say we want engineers involved earlier in the process. And part of that is because we want to help bring the context of what is the problem we're even trying to solve and what solutions are even possible. Sometimes you get a really great solution and you're like, wow, that would be a fantastic solution. All we need is a generalized artificial intelligence or artificial general intelligence and time travel. And then we can have this to market any time. Right? But other times people bring you a solution and you're like, you've made this more complex than it needs to be. You've asked me to install a scanner next to the printer so that you can scan the printed documents to communicate between the two software systems. Meanwhile, we have a network and I know HTTP and JSON. And if you give me a day, I can connect an API on one side to the other and you won't ever have to worry about the printer or the scanner again. Yeah. That is actually a real example from a developer I used to work with. It was asked for a scanner to set next to a printer so that they could OCR the document that they printed between two systems for which they had all of the source code. That's excellent.

[00:42:05] Allan Stewart: Another way that we can apply questions in our work is making use of the Socratic method. This is something that I remember learning from you when we first worked together. And it definitely made an impression on me because the Socratic method is kind of a teaching tool where you're asking questions to get somebody to think about things. And I mean, I guess if you do it maliciously, you're kind of leading them down the garden path. But ideally, you're leading them into understanding something without having to tell them, right? Because oftentimes people don't want to be told. But if you can turn it into a question, you can get them to arrive at it in a way that helps their understanding, that makes them more engaged with it. And maybe along the way, you're learning about why it is that they, like sometimes somebody just keeps saying this one thing and you finally find out why, then you can address that underlying concern and then move on.

[00:43:14] Dave Adsit: I worry that the version of the Socratic method you learned from me might have been Like a baseball bat, where Socrates would have used it like a fencing foil. But yeah, I agree. It's a very, very useful tool for helping someone arrive at an understanding on their own. Like we can't transfer knowledge from one brain to another directly. So we use tools to help us teach people. And this is one that I really like because this is one where I potentially, as the educator, I have the opportunity to learn as well. Because I'm going to ask you a bunch of really probing questions and sometimes leading questions that are going to take you towards deeper understanding of a system. Like, oh, have you considered this aspect of a distributed system? Or have you considered that no other tool in our, no other product in our entire system is using that particular RPC method or toolkit? And most of the rest of the system is using, you know, JSON over HTTP or, you know, enqueuing messages into this system, whatever. Like asking questions allows you to potentially guide someone to a deeper understanding of why you have said a thing to begin with. And I have found that if you are an architect in a, with more influence than authority in a system, the Socratic method is going to be one of the very powerful tools you have for conveying the reasons why you're doing something. design the system the way you are.

[00:44:52] Allan Stewart: Yeah. And I just remember it being pretty effective way of delivering communication because again, it gets us out of that declarative mode. Yeah. And, and even if it is more of a baseball bat than a FOIA, if it's effective, if it's working, maybe that's okay.

[00:45:13] Dave Adsit: I like to hope so. I do think that, you know, it's a, it's an opportunity. Like I said, it's an opportunity for, it's an opportunity for learning on both sides, but it's also an opportunity for engaging the brain of the person who doesn't have the knowledge yet. And usually you're working with someone who is smart and professional and wants to understand and wants to do the right thing, but they have a different understanding of the right thing. Maybe they've never worked in a system at the scale that you're working at, like whether it's much smaller than they've ever worked at before or much larger, they've never worked on an embedded microsystem. Right. And so you need to ask them questions that will help them understand the constraints that you're operating under. Oh, did you know that we only have 16K of RAM on the deployed environment? And so loading the JVM is not going to be very effective for us. Did you know that we process 10,000 requests per minute during peak times? And if we do something that slows each of them down a hundred milliseconds, we will have to spin up X number of additional servers. Or what do you think the impact would be given that we process 10,000 requests per minute during peak load if we slow down each request by 10%? And then that person can start thinking about, oh, what would the impact be? It's kind of, you can use the Socratic method almost like a guided pre-mortem, which pre-mortem is another technique that we could be discussing, but we haven't. It's basically, in a nutshell, the pre-mortem is before, when you're starting a project, you think about what is the worst possible outcome. Like in six months, we won't even meet each other's eyes. We won't talk anymore because the project went so bad. What happened? And now we can work backwards from that presumed future awful state, guessing at the things that could happen and then trying to put in mitigations ahead of time so that those bad things never happen, right? So the Socratic method is a way for communicating brain to brain. Like here is knowledge that I think I have and I would like you to have as you are considering the problem that we are facing together.

[00:47:27] Allan Stewart: When I think about how many times, especially in a technical sphere, somebody who has experience will see some fundamental issues with a proposed solution and it's hard to explain. And so the first impulse is just to say, that won't work. I love it. That won't work. And it just, it happens all the time, but it shuts down conversation, right? It doesn't allow for learning. And so using the Socratic method in those kinds of situations where you might have an idea of why this won't work. It's like, oh, well, I know from prior experience that if you do this with this messaging system, well, you're going to turn around and publish a new message that's going to be received. And now we're going to create an infinite chain of messages going in a loop that magnify each other until you have an outage in your message broker. And I know that because I've done it before. You haven't experienced that yet. And so if I just say, no, that doesn't work. Well, now I don't know. It's not helpful. But if I start asking the questions and say, okay, well, how will we handle this next message that gets published? If it could, if it turns around and publishes in a loop and you get them thinking about it, it's like, oh yeah, I hadn't, hadn't considered that. And, and then, then instead of it being, let me tell you why this doesn't work. Hey, let's talk through a problem together and come up with a solution together.

[00:49:06] Dave Adsit: Yeah. Well, the Socratic method reminds me of another technique that's been used by some of the coaches I know. And I am very likely poorly understanding this and butchering the exercise. The implementation of it. But the technique is teaching from the back of the class, which is basically, as I understand it, you want to create a learning environment, a space for learning where people have all the information at their available for them to learn. And then you want to create the psychological safety necessary for someone to be vulnerable enough to learn. And then you want to provide gentle nudges towards the answer that will allow them to, basically you guide them towards learning the concept on their own, if you will. Right? I've created a context where we've talked about it as the pit of success. I've created a context which you can't help but fall into the pit of success and learn this thing that is important for us to go forward. And I will say that one of the big drawbacks of using teaching from the back of the class or the Socratic method or any of this is it is slow. If I know the answer and I'm the chief architect and I can do it, I can just say that won't work. We're not doing it. I can then walk away as safely assuming that no one will ever do the thing that I just told them they can't. Though as a person who has on more than one occasion said that won't work, you can't do it. I can tell you sometimes they go try anyway and waste a lot of time. And so if you think you're saving time by being the tyrant who has all the answers or the wise wizard, none of us ever thinks that we're the tyrant who has all the answers. Obviously, we're the wise wizard who brought the tablets. We brought the, we brought things down to the, to the masses so that they can produce better code. If you think that people always listen, then you have not entered in. You haven't really worked with too many people, I would guess. Because, you know, people need to learn on their own.

[00:51:02] Allan Stewart: Yeah. Yeah. This is an area where I'd like to improve. I'd like to learn more about the philosophies around teaching from the back of the class. But I can definitely appreciate those times where I have experienced something similar.

[00:51:16] Dave Adsit: Right.

[00:51:17] Allan Stewart: And, and oftentimes it kind of feels like you're guided along. Things are going good. Okay, I understand. Great. And then it's just that gentle nudge, which feels like a monkey wrench that just got thrown into the mix. Oh, wait. And it doesn't even have to explain it. It's like, it's not, let me tell you why this, this won't work. It's just, well, what about when this happens? Yeah. And then if you really sit back and think about it, oh, well, crap. Well, if that happens, well, then this, none of this is going to work. So it's, it's something that I would definitely like to spend a little bit more time in.

[00:51:53] Dave Adsit: And I would say that we all appreciate it when someone takes the time to slow down and teach us something. It's one of the things that is very valuable to me in a pair programming, ensemble programming, any kind of a collaborative programming environment is when somebody slows down and takes the time to teach me something. I really appreciate it. I remember those people for a long time. And I think taking the time to slow down and explain the context, explain something to someone who doesn't understand it and using the Socratic method so that they can actually truly understand it and follow along the reasoning and understand it themselves. I mean, that is actually going to save you time in the long run versus, and build a better relationship versus just telling somebody what to do. And they may go do the other thing anyway, which costs a bunch of time and you know is going to fail, but you didn't explain why it's going to fail. Right. And you didn't explain it in a way that they truly understood it, but also kind of, it can start to break down relationships. If you are the person who always just tells other people what to do, it's not a very healthy relationship that you have with them. It's very one-sided. No one likes the tyrant. People in what I've read about animal troops is, you know, in a troop of great apes, the strongest is, the strongest ape is typically, the leader. But if the leader is a tyrant and abusive to the other apes in the troop, the second and third biggest and strongest apes will get together and depose the tyrant, usually to death. You have been deposed to death. And now there's no longer a tyrant. And so what did you accomplish? If you can't work collaboratively with others, you're not creating the kind of healthy environment that we all want to work in and, and build self-software in. Yeah.

[00:53:48] Allan Stewart: Which brings us to the last category that we wanted to discuss, which is you can use questions to challenge. I think it's one of the more effective ways to challenge things. And I'll start with challenging assumptions. And it's easy to kind of look at that and say, oh, okay, awesome. I'm going to gear up. Let me get some data. I'm going to, I'm going to wait until my boss says something and then I'm going to throw that data at them to challenge them. Okay. But I think it's most effective when we turn around and we're challenging ourselves, challenging our own assumptions about how things are going to work, leaving ourselves open, just being in that mindset where we're asking questions all the time and we stop assuming that we know anything and just go and look, find out.

[00:54:36] Dave Adsit: Yeah. I'm reminded of the meme. You sure about that? Which is very accusatory type of a meme that gets thrown at people when, when you definitely don't agree with them and when you're kind of setting them up, right? So if somebody says, hey, I believe X and you say, ah, you sure about that? That is not a healthy line of questioning to start with. But there are ways to challenge those assumptions in a reasonable and healthy way. Like, oh, what do you, what leads you to believe that that's the case? What information have you seen that I haven't that would help me understand why you think that, you know, X is true. So it can be very, very useful when we have assumptions about, I mean, even if we all believe the assumptions, we should still verify with data as often as possible. So many startups fail because they fail to understand the customer. I mean, this is one of the things from the product management space is that a company or a product will succeed if you fall in love with the buyer versus, your solution. Because your solution may not fit their needs. But if you want to solve their problem, then you will keep trying until you figure out a way to solve that problem for them. So if we've fallen in love with our solution to their problem, well, they may not want it. They may not even need it. It may not be the right solution for anybody in our target market. So yeah, we should definitely be asking questions to challenge assumptions on a regular basis, especially our own. I know I make a lot of assumptions about what makes an effective team and how people are going to work together. And I need to test those on a regular basis to make sure that they are still true in the context of this team and these people.

[00:56:27] Allan Stewart: Yeah. Yeah, we can challenge product features. And you alluded to that and we talked about it a few minutes ago as far as coming to the problems instead of starting with solutions. But I think also we can challenge our assumptions or challenge a product, product feature through questions. So there have been some times where I will look at usage of code or features. It's like, oh, this API endpoint hardly gets any traffic. It's got complicated code. Couldn't we just get rid of it? And it turns out that it doesn't get a lot of traffic because it's part of maybe a monthly report or it's something that is very important for people to do sometimes. They don't do it very often. In our current system, we've got a feature that is quite complex that allows you to copy an entire project. People don't use it every day. But when they do use it, it's incredibly valuable. And so asking those questions and finding out the purpose behind this thing, even if you went and did you do diligence and grabbed all the data and you say, yep, I've got it. It's proof right here. This feature doesn't get used. Well, hold on a second. Let's ask a few. Two more questions to understand. Does this product feature, is it really needed? Because I've also seen the reverse where a feature is being used and it's got daily usage, but it's not very helpful or it's counterproductive to the strategic initiative of the company because we're trying to move on to a new business model or something. And it has to be gotten rid of. And so you've got to challenge that assumption. You've got to challenge that feature. Say, hey, should you really be there? Even if everybody says they like you, could we get rid of it and make the system better?

[00:58:21] Dave Adsit: Yeah, I've got a lot of experiences with product features in both of those spaces. One that came to mind immediately is the sales tax report. I mean, used at its maximum, you're only going to be calling that sales tax report endpoint once a quarter per customer, maybe only once a year. And if you say, hey, this doesn't get very much usage. It can't be very important to anybody. You would be...

[00:58:46] Allan Stewart: Hasn't been used in the last 20 days.

[00:58:48] Dave Adsit: Yeah, nobody's even used this in 60 days. You may find that that report is actually substantially more important to your customers than you ever expected. And when you have an integration with a partner and you're like, well, the relationship went sour and we're no longer partnering with that company anymore. Even if people are using it, we want to be pushing them to use our new partner or the thing we built in-house instead.

[00:59:12] Allan Stewart: So I guess to wrap... questions are really powerful. When we get into that mindset of asking questions and we start seeking context and we are leaving ourselves open to having a more positive impact on people through these questions, through kind of a inquiry mindset, it can really help us out. And like you said at the beginning, this is something that software developers need. It's one of those soft skills. that can be used pretty much every single day to great effect.