Crafting Code Podcast

[00:00:15] Allan Stewart: Welcome to the Crafting Code Podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately I've been thinking about what I can learn about the cost of delay from Factorio.

[00:00:31] Dave Adsit: I'm Dave Adsit, a VP of engineering, and recently I've been thinking a lot about code freezes and the concept of natural consequences in parenting and leadership.

[00:00:41] Allan Stewart: Our topic for this episode is work is work. And we're taking this title from the title of a blog post and an article that was written by Coda Hale. And we will have the link to this article. In the show notes so that you can follow along if you want or read it after the fact. But this is an article that came to my attention some time ago, maybe over a year ago, and it's just kind of stuck in my memory. And we both found it interesting. And so we thought that we would talk about it a little bit at a high level, just to kind of summarize the article for anyone who hasn't read it or paused us so that they could go read it first. The article examines. Concepts which Coda calls a priori truths that impact organizational performance, regardless of which organization you work at, because there's a lot of times where we think about or we discuss organizational performance and what we learn only really applies to a certain company for, you know, for certain reasons. But he tries to take a higher level of approach and look at things like work capacity and how it scales. Linearly, linearly contention costs and how they grow super linearly as you add people and that communication costs can grow quadratically. And then he talks about different principles, which he writes from beyond space and time, which promise to help any organization. And these include some things like parallel work, local resources, local decisions, force multipliers, independent or autonomous work product. And some things like that. So before we dive in, did you have any overall thoughts you wanted to share, Dave?

[00:02:34] Dave Adsit: Yeah, I think one of the important things to call out is that the very first line of the article, which looks like it was added later, is every time I've written or spoken about organizational design, I've regretted it. And I think the reason that that stood out so much to me is because so much of the time when we talk about organizational design. We are trying to draw generalities from a sample size of one in this company at this time, we did this and therefore this works for every company at every time and every scale of regardless of context. And I think that's a very tempting and easy trap for us to fall into as engineers, engineering leaders, whatever. Definitely. But I like that he goes beyond that. And talks about how there are in fact lessons that can be learned. And it's important for us to consider the constraints around organizational design, even if there are. So basically, he doesn't go into this in super great detail, but there are preferences and there are principles. And understanding the principles is critical. And then we can accept or reject preferences based on context or what it is we're trying to work on.

[00:03:55] Allan Stewart: Yeah. Yeah, I like that. Near the beginning of the article, he talks about how if you look at the success stories from any given organization and these abound, right? So you pick an organization, you pick Apple or Google or Netflix or Amazon or any well known house name tech company, you look at it and say, oh, well, why are they successful? And you look at the narrative of why they say they're successful. And from the outside, it becomes readily apparent that that doesn't work outside of their context. Right? You can't just cargo cult those changes and expect to be able to get the same results. So I like that. He's calling that out from the beginning and saying, okay, well, let's kind of go back to first principles and reason this out.

[00:04:50] Dave Adsit: Yeah, I agree. He says. Yeah. I get the idea. I get the idea. I get the idea. I get the idea. I get the idea. I get the idea. I get the idea. I get the idea. I get the idea. I get the idea. came to be or how things came to be successful. And so I like that he breaks it down and says, well, what can we know? Are there things we actually can know? And he's like, yeah, there actually are things we can know. And here are things that we need to understand when we are structuring our system, structuring our people. And we know, as engineers, we know that the systems we build mirror the organizations we work in. I mean, that is Conway's law, right? And I've yet to see it violated. There's a whole bunch of things you can do to take advantage of that fact. But if you are unaware of that fact, it will, in fact, eat you in the end. Your organization will influence the way you build your system.

[00:06:04] Allan Stewart: So let's go one by one. The first of these kind of generalized principles that hold true across any organization, he titles it, the ceiling is low. Right. And the concept here is basically that you can't scale any better than linear, right? So if I have five people, the best that I can do as far as having work capacity is having all five people always busy on separate things, right? Like that's like... Not just separate, but fully independent. Right. Yeah. They have to be fully independent because otherwise they block on each other, right? And this is the classic... This is the classic things we see with like merge conflicts or trying to think of another example, but these instances where somebody does something and it's dependent on another person or one team member or one team has to complete a task in order for another person to be able to complete another task, right? And so because of these interactions that we have between the people and they're inevitable, right? Like assuming that you're a team working on the same product, You have to interact. You don't have five completely separate products that are all completely separate. Unrelated. Right. And completely isolated and have no coupling whatsoever. As soon as you have even the smallest bit of coupling, then you no longer can be fully linear. Now you're sublinear and you have to deal with the fact that, oh yeah, there was a pull request. Oh, there was a merge conflict. Oh, we do need to discuss the contract. I get the idea. I get the idea. I get the idea.

[00:08:05] Dave Adsit: I get the idea. I get the idea. I get the idea. I get the idea. I get the idea. I get the idea. I get the idea. I get the idea. I get the idea. I get the idea.

[00:08:16] Allan Stewart: I get the idea. I get the idea. truth is that you're going to be somewhere below that. And that is the contention can be introduced

[00:08:25] Dave Adsit: anywhere, right? So the coordination points can be anywhere. It's like, I have to ask you a question. Now we are both blocking on you answering that question and me getting back to work or me getting out of your way so that you can get back to work. And if we start having staff meetings or status meetings or project updates, or we share a CI system and there's a, I'm waiting for a build because your build is first and I'm in queue, or we have to apply database schema updates and we're going to do them sequentially. Any of these things can become interactions that cause us to go below the capacity of each of us added up, right? So linear is the max we could hope for, and we aren't going to get it, which is very consistent with a lot of the things that I have come to be familiar with from other sources, right? When we talk about lean, we talk about linear. The lean system is both maximally flow efficient and maximally resource efficient. And we just don't ever get there because of things like inconsistency, like any kind of variation in the system will cause us to miss our target of perfect flow efficiency and perfect resource efficiency. And basically anytime humans have to interact, that will cause us to miss our ceiling of N1 plus N2 plus N3 plus N4 equals, you know, in total. And that's the capacity that we can have as a system.

[00:09:48] Allan Stewart: Sure. And this happens to systems regularly, right? Like even if you take those messy humans out of the equation, there's still so many things that happen in a system. Like even if it's a programmatic one or like an automated system, there's something that some bit of variation comes along inevitably, you know, the power went out on a particular server rack or the cosmic rays flipped a bit or, you know, something happens eventually. But, that changes things.

[00:10:20] Dave Adsit: Yeah. I was just reading that a control system has to have a sampling rate of at least twice the delivery rate of the system being controlled in order to be effective at all. And, and so you think about, you think about that in terms of, you may not even know there's a problem for a minute. Right. So, so yeah, rule one is the ceiling is low. The capacity of your system will, of course, be less than the capacity of all of the people in the system added up.

[00:10:53] Allan Stewart: Right. And so one of the things he points out in this section is that you need to look for force multipliers or another way. In other words, how do you increase the capacity of the people? And thinking about that reminded me of our discussion some episodes ago about 10X developers, or rather those developers who create 10X systems because of the capacity of the system. they're the kind of people who are engaging in this force multiplier effect where we say, okay, yes, we could do everything manually, but we automate this, or we add unit tests here, or we improve our CI CD pipeline. And then our capacity to deliver has increased because we're in, we're updating the system rather than just, rather than just telling everybody to just work harder or put in extra hours or whatever it is, you're actually making everyone more efficient.

[00:11:50] Dave Adsit: Well, and it goes all the way down to the one tool that we've identified as a context-free good, which is source control. Like I spent part of my afternoon today copying code, copying SQL scripts out of a database that was built manually into source control so that we could rebuild that database at some point in the future. And did I need to do that? No, because we have the definition of those, tables and views somewhere else. And I could just, at some point in the future, recreate them manually for a lot of work effort. You know, it would take a long time and a lot of work to do it. It took a long time and a lot of work to do it the first time. Unfortunately, we never bothered to capture that in source control. And now we are capturing it in source control so that when we make changes in the future, if we do it wrong, reverting will be easy and fast as opposed to re-imagining what the previous state was and working our way back towards it. So the ceiling is low. We've established that we can't get above linear unless we create force multipliers. But the good news is that we'll always operate really close to that ceiling, right?

[00:13:00] Allan Stewart: Yeah. So even then, I don't think that the force multipliers make us super linear. They just increase the capacity. Yeah. You're moving that line up, but it's not exponential. It's still linear. But it's still good. To move that up. Because as you were suggesting, the next two principles kind of drag us down.

[00:13:24] Dave Adsit: Right. The next principle is that the floor is not linear. The floor is lava, right? So as contention grows, so contention costs grow super linearly as we add new members to the team. So if you're familiar with, with your queuing theory or any of that, there's a pretty basic concept, Kingman's formula, which basically describes how a queue will fill based on arrival time, arrival inconsistency and arrival time, and inconsistency and processing time. And so um, as we add new members to a team, the contention costs grow super linearly. Basically that means that, I mean, not to spoil it, but at a certain point you can add so many members that you have your contention costs exceeds your new, your net capacity from adding a new person. We've heard of that before in books that were published as recently as the eighties.

[00:14:34] Allan Stewart: Yeah. Yeah. I love, uh, I love it when queuing theory is trotted out because there there's something nice about taking this mathematical, uh, mathematical approach to something that we tend to do without any thought at all, or not much thought in, in too many situations, right? Like we have a sense of, well, this is how things are done, but we don't really examine the impact or here, take this naive approach. And so I really, I like what we can learn from the mathematical models that you get out of, uh, queuing theory.

[00:15:08] Dave Adsit: Well, and one of the benefits to that is that they help constrain the reality. If, if under perfect conditions, you could not reach the capacity or the delivery that you desire, will, you know, you're not going to get perfect condition. Yeah. So it helps us reset our expectations around what is actually going to happen in the system that we are building.

[00:15:32] Allan Stewart: And give us context to what, what are the ramifications of making a particular change, right? So like in this case, what changes, as you, Uh, add additional people with the recognition that additional people add additional contention, right? So it's that same thing we were talking about before that at our very best, we can be linear, but we know we're not going to be linear because we need to interact with each other. And we're going to have a discussion. And now you bring in a new person and now you have to have more discussions. And so now there are, you know, things, things drop off. And if you add too many more new people, then all, you know, your expert developer spends all their day babysitting other people, trying to explain things to them and get them going instead of doing any work themselves, which might be a good long-term investment, but there's, there is at least some period of time that, that, that is going to be inefficient.

[00:16:32] Dave Adsit: Right. Well, and, and the thing about, the thing about efficiency is that I really just don't even like the word very much. Because of how it's typically used. I think that we often use efficiency when we would be better off if we were looking for efficacy, but that's kind of a tangent for another time. I was thinking more about like what we tend to do when things start to get behind. One of the first things that we always reach for is more people. Oh, we're going, we're not, we're not going to hit, you know, we set this roadmap. It has us finishing this feature in this feature by the end of Q2 and the end, then this feature by the end of Q3. Okay, great. Well, it looks like we're not going to hit it. So what are we going to do about that? Let's add more people to this system. And we've all read that adding people to a late project always makes it later, but here's some mathematical evidence that that is true, right? So our, as we add more people, our contention costs go up and our achieved capacity from each person versus their potential capacity goes down because, you know, people who are that? who are doing training, performing, training activities, training others, they're going slower than they would if they were just writing code. People who are being trained are going slower than if they knew the system and could execute on it. Great. So what should we do then? Well, if, if adding, if adding individual or adding staff to our company, individual contributors to our teams didn't help, what we should probably do is grab some external experts, right? So this is one of the quotes that I pulled out was, and this is a quote A commonly applied, but rarely successful strategy is using external resources, e.g. consultants, agencies, staff augmentation as an in run around contention on internal resources. Great. Often why it appears that this is going to work is because we bring these people in and we don't give them all the same constraints that we give our staff. And so they look really, really effective until we realize that everything they produced is garbage. Or at the very least, everything they produce is garbage. Everything they produced is in conflict with everything our team produced following our rules. And you usually eat up more time in the merge process, whatever that merge process is, then you saved by hiring people and allowing them to operate without the same constraints and same contention as you have within your system.

[00:19:00] Allan Stewart: Right. The only times that I've seen some of these external resource strategies work is either when they were doing something completely independent. And so there was no need for the integration or that they are trained to do things in the same way as the team. Right. You're you're you have a high bandwidth communication with those external people, contractors or whatever, at which point it's you're still suffering the same contention costs that you would if you had hired a new person because somebody is spending their time explaining what's going on. Exactly. And I think it's important to note that that doesn't mean that these are bad ideas. Hiring another person because you want to increase your your overall capacity is not wrong, nor is it necessarily wrong to use an external resource. But it's not a solution to get the thing done faster. Right. Especially if you're you're if you're late on a project and and you're looking for ways to get it done sooner. It's not a solution to that problem. In fact, it's a anti solution, anti pattern for solving that problem. But it might be the right solution for a different problem, such as next year. We want to have better capacity than we have this year. And so we're going to hire a few people and train them, recognizing that it's an investment in later, not an investment

[00:20:34] Dave Adsit: in now. Well, and I will say in the worst case scenario new code that was poorly tested, was functioning poorly and full of bugs. And so every day, my more effective team, my team with higher capacity to do net positive work, spent all their time correcting the work from the team that had the lower capacity for doing net positive work. And we actually increased the velocity of the team overall. We increased the ability to deliver value value by cutting the team in half and eliminating the offshore team. It was a very challenging decision to make. And it was hard to sell up and down the chain of command because people were like, you can't possibly go faster with half the staff. I'm like, well, let me tell you how the second half of the staff is creating less value than the first half of the staff. So in the worst case, that's what I've seen personally. I'm sure other people have horror stories as well. And I'm sure that people have I've had experiences where hiring an external team actually worked really well. And I mean, I've seen one example that I can pull from my own experience is having a team of expert web developers building a web application that is the primary product of the company. We have hired an external consultancy of webmasters or web designers, or I don't know what the correct term is, who knew a specific toolkit like Webflow or WordPress or whatever, and built a new marketing site and delivered it with the help of the marketing team and with no help whatsoever from the engineering team. Was the product as good as it would have been if the engineering team had built it? Of course not. Did it meet the needs of the company at the time? Sure it did. Did it allow us to actually get that project, which was completely independent and orthogonal, done in a timely manner without disrupting the critical work on the product? Yep. And so that was a case where bringing them in was actually very, very useful. Because we went back to step one, to principle one, which is you can only add capacity if you're working on independent, like adding people only adds capacity if you're working on independent

[00:23:17] Allan Stewart: things. Right. And adding people gets worse than that, right? So we had the ceiling is low, the floor is lava. And now we get to the third, which is that hell is other people, in which we discuss, the coherence costs. Or sometimes these are known as communication graphs. And basically, we're saying that it's a pretty intuitive principle at its core, is that if there are more people, it's going to be harder to get them all on the same page, right? It's going to be harder for everybody to know all the important things uniformly.

[00:23:56] Dave Adsit: And if you look at it mathematically, you have two people, that's two arrows, two one-directional arrows of communication that have to occur. Now, when you have three, you have three arrows. Oh, this is scaling linearly. Great. As soon as you have four people, you realize that you have six communication pathways between them all. And five people is 10, et cetera, et cetera. And it scales quadratically as you add new members to the team. So the actual formula is 2n minus n over 2.

[00:24:27] Allan Stewart: Right. So I read back at the beginning of this year, I read an article that somebody shared with me, a different article that was titled the non-issue of team size, AKA the incomplete understanding of complete graphs, which had some good ideas. And maybe we'll talk about that article some other time. I disagree with some of the details and conclusions, but overall, I think that there's some really good principles that come out of it. And one of those was that I liked the concept that you don't always have to engage the full graph, right? Just because you have six, six people or, you know, 12 people or 20 people, that doesn't mean that every line on that geometric graph of all possible communications is going to light up, right? For example, a lot of companies divide up into teams and teams will often have team leads. And the team lead goes to talk with the other team leads about integration. So we don't have to have him talk to, or her talk to every other team lead, um, nor do, nor does she have to talk with every member, the other team, nor does every member of their team have to talk to every member of our team. We can, we can streamline it. And so there's only, there's only one person, but ultimately the more people you are, you have the more opportunities there are. And I think that that's the key. It's, it's not that it's not that you're going to always engage the full graph, but you have to have this recognition that the full graph exists. And sometimes it engages in ways that you do not want where somebody on the team talks to another person on another team, and neither of them have the full context of what's going on, but they share a rumor that, or spread an anti-pattern, or they do something else that traverses through the dark areas of your graph that don't follow the org chart and cause

[00:26:24] Dave Adsit: systemic effects in your system. Right. And I would also like to point out that even if you are not engaging the whole graph because of an optimization that you think is reasonable, like we have three teams, they each have a front-end expert. And so once in a while, the front-end experts and the tech leads get together and they talk about front-end work. Great. That's an optimization that we've taken. Now, when we have a production outage and one of the backend developers is on call and it turns out that it's a front-end issue, now we've got a different class of problem, right? Because of how we... We've chosen to optimize for communication and for coherence across the group. We have accepted incoherence, or rather we have accepted gaps in our understanding and knowledge as a trade-off for efficiency of communication. And so that happens. That is very common in groups. But it is a failure mode that you need to be aware of if you are responsible for designing a technical system, a socio-technical system.

[00:27:34] Allan Stewart: Right. And because it grows quadratically, your optimizations still exhibit that same behavior, right? So if all of your... You're only having your tech leads and your front-end engineers come to the meeting. Well, if you grow from three teams to seven teams, well, now there's a whole lot more people who are interacting in those meetings. And you can further streamline it. Sure. There are ways to do that, but you are losing that... Incoherence. And I like that word, right? Especially when you turn it on its head and you're saying it's incoherent. And you're thinking of somebody, right? Like if a person is incoherent in their speech, there's kind of babbling, they're talking probably not unlike I tend to rant on, on occasion, not fully understandable, not completely cogent or making a point. But we're doing that organizationally rather than as a single person. As an organization, we don't quite make sense. And when we think about it that way, I think it gives it an interesting, or at least for me, it was an interesting realization. It's like, oh, yeah, we don't make sense as an organization when we do this. Right. So then moving along, after discussing that we've got this low ceiling and that we've got these other two principles that as we add people, it kind of drags us down, right? So like we've got the ceiling at the top and then we've got our, you know, this other, these other curves coming in at the bottom that are limiting our effective space. And so then the last part of the article talks about principles that help us work within these constraints that we find ourselves, right? We had a discussion about constraints in a previous episode where we talked about how great engineering comes... From constraints. And I think we can apply that to our organizational design. We can apply that to our thinking, our system's thinking to say, hey, when we recognize that these constraints exist and are there, what can we do about it?

[00:29:45] Dave Adsit: Yeah. So the first one, the first of the principles from beyond space and time, meaning they apply always and everywhere, is that you should keep work parallel and your groups should be small and your resources should be local. And of course, you all know, I don't like to use the word resource, just to refer to people. But in this case, it's like keep your work parallel. So you don't have a lot of dependencies on each other. Keep your groups small so that you can stay aligned and working in the same direction and keep all of the necessary people to get the work done local, right? So for example, if you're working on a team that is doing a lot of heavy database work, you should have a database expert DBA person on that team. Not, you know, someone available to be borrowed whenever you need it, but somebody who is actually a member of that team doing work with the rest of the team to build that product. If you are building a very design heavy product, you know, you're working closely in the front end, you're designing user experiences for particular users or for users whose behavior you're trying to influence in one way or another. Then keeping your designer on the team as a member of the team, working with everyone else is going to give you the maximum opportunity to stay aligned and to keep high coherence and to understand what is going on, what you're seeing, what is happening and how to build the best possible solution or product

[00:31:14] Allan Stewart: for your users. Yeah, I like that. And the resources don't all have to be people, right? So to make you a little bit happier, since you don't want to call people resources, I think that it extends to other things too, right? Like, can the team deploy what they need to do? independently without having to wait for separate other teams, right? Do they have to wait? Or other literal resources, right? Like if they have to wait for a build server, or if they have to wait for requisitioning a server that's going to be put in your colo, those are things that are not local, right? They have moved. So if you keep the group small I think this is, this is one of those cases where we learn a lot from domain driven design, concept of bounded contexts, where we can say, hey, within this context, we can keep most of the work within a team, right? We can, we can build up these areas of independent responsibility. And I think a lot of times we get this conflated with microservices, microservices is an architecture that is definitely built on some of these same principles, but by itself, it's just a deployment strategy. And it's, it's like the shadow of this principle, right? Like it may align really well, but you, you don't necessarily need it. You could within your monolith still achieve a lot of these same concepts by having dedicated teams for certain areas and, and strong architectural boundaries between the, between the teams.

[00:32:54] Dave Adsit: For sure. And this reminds me of the Amazon principle of the two pizza team. You know, you should have a team small enough to be fed with two pizza. And so that's not going to be a very big group. That's not going to be a dozen people. That's not going to be 25 people. That's going to be a handful of people. And it, you know, keeping your resources local. It just, it does remind me of all the times that I've had to requisition a server and wait. And now we don't do that very often because most of the stuff that I work on at least is cloud deployed and you can get new, you can get new, whatever you need almost instantly. Or within 15 minutes. So when we talk about the microservices architecture, I've more often seen it be the opposite of supporting this principle than something that is a shadow of, or in support of this principle. Almost every microservices architecture I've seen in real life has actually been a distributed monolith where the work is in fact, not parallel. You cannot in fact, deploy any one thing without deploying everything all at once. Mm-hmm. And the group is accidentally every engineer you have, plus a few who've quit and a few that have, you know, moved on to other things. And none of the resources that you actually need are local because everything is being done via RPC. And so this is one of the areas where your architecture does not necessarily mirror your intentions when you are trying to follow this principle. And so I would say, yes, you can use microservices well, Be very clear that you understand it and be very careful about refactoring over and over and over again as your understanding increases when you embark on that journey into microservice land.

[00:34:40] Allan Stewart: The next one is prioritizing development of force multipliers. And I think given our previous discussion about the 10X systems, we don't need to go too in depth here. I think there's a lot of relation there, but basically just saying that you can improve the system, right? I think that It might be people in some cases, it might be practices that you're following that make you more capable so that you can get more done, right? It's that economic concept of moving to another

[00:35:15] Dave Adsit: state and earning more. Exactly. One thing I would like to make sure that we touch on here though, is that knowledge is one of our force multipliers. And as anyone leading an engineering organization, overlooking the importance and the value of investing in the knowledge of the people on our team. There's an HBR article from probably 15 years ago now that talks about engineering groups that worked on aligning engineering with the goals of the company versus engineering groups that worked on improving their ability to do engineering work well. And overall, the ones that focused on learning and development and becoming better engineering organizations and better engineers outperformed the group that focused on alignment in almost every case. Perhaps it was in every case, but because they were able to deliver more and check in more and make sure that what they were delivering actually met the business needs. And so being more skilled, having more knowledge overcomes or supersedes the value of being better aligned with your business unit. one of the things, one of your force multipliers is continuous education for your development staff. So the next one is, the next principle is, if possible, factor work products into independent modules. If not, grow slowly and optimize. So that speaks a little bit to the concept of the microservices versus monolith, right? Sometimes we think, oh, if we do microservices, then we'll have independent things. Maybe. I mean, ideally, yes. Realistically, if you don't know how to factor code well to be independent in a monolith, you're probably not going to factor it well to be independent in a microservices environment. You're just going to have more deployable units to manage when you try

[00:37:15] Allan Stewart: to update something. Yeah. Yeah. I think it's putting the cart before the horse a little bit. You have to find out what is independent first or make it independent. This is oftentimes examining coupling and coupling. And cohesion within a system and figuring out what actually belongs together or doing some of the domain driven design exercises to find out what things belong together, what things have the same name, but they're actually separate concepts, or they can be the same concept, but they have different representations in different parts of the system. These kinds of things, understanding the personas of the users that are in your system so that you can actually understand that independence, what actually is independent from the rest, and then you extract it, whether that's full extraction into something like a separate service, or it might just be within your well-crafted monolith that this is just known. There is this internal boundary that we don't just reach over into the depths of it. We work along the interfaces so that it stays

[00:38:26] Dave Adsit: loosely coupled. Right. We build a separate module. We're a separate bounded context, however we choose to represent that in our database. One of the things I wanted to make sure to pull from this is a quote that says, some problems are not particularly amenable to partition. These are also problems which do not benefit much from additional workers. So if you have one of these parts of your system, which every system probably does, that you can't subdivide it further without losing the cohesion that you need from it. You've... Decoupled as far as it goes, and it's still kind of complex. Every system has something like that. Well, having a bigger team or having more people probably won't help much. There is still a minimum floor that I, in my opinion, for how many people should be on a team. I don't want to have one person be on call 24, 7, 365, because, you know, this was a, this was an atomic problem we couldn't subdivide. And so we only applied it. We only assigned one person to it. I like it when people can go on PTO and perhaps be sick occasionally and also have weekends. So I like to have teams of several people as a minimum. And then maybe those teams, instead of working independently as separate threads of execution, they all work together as a mob, as a single thread of execution with single piece flow when they're making enhancements to this atomic problem that cannot be subdivided without breaking it.

[00:39:52] Allan Stewart: Yeah. I like that a lot. The next one is scaling, scale organizational efforts across a port, portfolio of synergistic products. And to me, this, this really speaks to this idea that having separate things that can grow independently has a lot of power to it. And especially as we're looking at systems, even a well-factored microservices system, at some point you get so many different bounded contexts, so many different services that there are things that are just wildly different, right? Like you can't just keep growing. I think of like Microsoft Word. What was the quote? I think you've, you've said this before and Microsoft Word was done in like 1990 or something like that.

[00:40:44] Dave Adsit: This is my perspective on it is the last version of Microsoft Word that added any useful features was in like 2003. And after that, they just kept changing stuff, right? They were like, ah, we don't know what to do. We're going to invent this new ribbon UI that And then in like three years, we can have an update where we remove it. And like, it was done. You didn't, you didn't need to keep adding to it. It accomplished its goal, but things that are in flight organizations tend to perpetuate themselves. And so nobody wanted to shut down the group that added features to Microsoft Word. So, I mean, I think about this one as well, from the perspective of when you look at it from a business lens. You're like, okay, if we build one product and it meets one customer's needs and that customer gives us enough money to sustain the business, fantastic, right? Except now we have a problem of client concentration. If the client leaves, now the business is over. So we want to have multiple clients. Okay. But now multiple clients means we have to not just have one product that we build and deliver, but we should probably also serve adjacencies because that will help us grow our client base. I, it is a well-known, in my, in the group that I run in, that if you say the TAM, everyone will laugh because everybody remembers being hounded about the total addressable market. And if we could add more features, we have a bigger total addressable market and everything is great. And, you know, we talk about the TAM a lot. The TAM is huge, Dave. Right? The TAM is huge. Like, I don't know how many presentations I heard where the phrase, the TAM is huge was mentioned, but you actually can increase your, market. You can increase the value of your business by building in adjacencies. And the great thing about that is that now, if you build it right now, you have a bunch of independent things that work kind of together. And so your product becomes a portfolio of products that are all interact or all support each other in some way, hopefully through loose coupling, right? With high cohesion within each one. And now you can do things like add new features to try them out. And then remove those features if they don't work out. Every product I've ever worked in has had just dozens of dead features that no one ever cares about, except like one 10th of 1% of users. And it would be to the benefit of the customer or of the business and probably their customers as well. If you stopped supporting those features so that you could double down on the ones that actually matter or developing new ones that might matter even more. But it's very hard to do for a bunch of reasons, some of them technical, many of them social, but it, Back then, it was actually a physical device. And it was very, very, very common for physical devices like that to have a power switch.

[00:44:05] Allan Stewart: Or there's the other classic example of Steve Jobs when he came back to Apple in the late 90s, and they had so many different products, and he slashed it down. He's like, no, there's consumer and there's pro, there's mobile, by which he meant laptop, right? Like there's the laptop, and then there's the desktop. Those are your only quadrants. And they've grown since then, but that idea of bringing stuff down. And I also like that idea of you're basically looking at cohesion, but at a high level. Like which things are actually cohesive together? Can they be separate products inside the portfolio? Because if you try to make it all just one product, eventually you get into that problem of critical mass. And now you just... You can't sustain it. Yeah.

[00:44:59] Dave Adsit: There's a power law of functionality in applications. I believe that like 20% of the applications do 80% of the work. And that some large portion, and I'd have to go look it up to get the exact number, but something like 60% of features are never, ever used. I mean, I have a team in my organization right now that it's a volunteer group called the cleanup crew. And they go around my system looking for... Potentially dead features and adding aggressive logs with alerts to them and then waiting a week or two to see if anything ever calls that code and then deleting it. Now, if we were working in a strictly typed language instead of a dynamically typed language, then we might be able to short circuit some of that stuff. But it is what it is. And we're cleaning up dead code and dead functionality that is just not used by anyone ever. So the last example in this... Section is when he talks about Amazon Web Services, which is... I mean, by this point, it's got to be hundreds of products. I mean, it's definitely many dozen, if not hundreds. And if you were to try to build one product that met all of those needs, it would be madness. But instead, they have a whole bunch of products, each supported by separate teams, that deliver all of those different types of value. And they can add and remove them as needed by the market. If, heaven forbid, everybody gives up on AI and Amazon has to shut down their GPU division, well, they can do that. I don't think that's going to happen anytime soon.

[00:46:39] Allan Stewart: Perhaps not. The next principle is keep responsibility assignment matrices small, sparse, and local. And this one really just reminds me of the importance of the context that the people have who are down in the weeds, right? They're the boots on the ground who really know what's going on. And so pushing decisions down to leaf nodes, right, down to teams, gives you a lot of power, right? You have to decentralize some of your decision-making process in order to make this work. But it also removes bottlenecks from your system. And decision-making bottlenecks are particularly pernicious.

[00:47:21] Dave Adsit: Well, the next one is that we need to prioritize asynchronous. Information distribution over synchronous. Now, this is one that you and I have talked about many, many times over the years. Basically, the idea being any type of information, such as a status that people need, should be presented via some form of dashboard. Calling a status meeting to find out, you know, who's working on what and how long they think they have left on it, et cetera, et cetera. Or how many widgets have we sold today or whatever. Calling status meetings for those types of things. Is a huge waste of organizational capacity, time, effort, whatever. And if the engineers you know or anything like the engineers I know, having extra meetings is nothing but frustrating. And all of these things can be done asynchronously over passive dashboards that just reveal the information. Or perhaps over asynchronous communication structures like email or Slack. Like, I don't need a real-time update. We've talked about this before, right? When engineers talk real-time. Versus when business people talk real-time. It's definitely a different thing. Like, we start thinking about how we can make sure the hardware ensures in-order delivery of whatever. And how can we make sure the system fails when we miss a real-time requirement at some point. So that we don't ever get out of order and broken. And sometimes when people in your sales team say, this needs to be a real-time dashboard, they mean, I need all of the data updated by end-to-end. So that when I'm looking at it the next day, the data is accurate as of yesterday, close of business. Yeah. Right. That's kind of how banks work, right? We're like, ah, that's crazy. But banks will have a running stream of transactions. And then they will have an end-of-day operation where they add them all up and tell you what your end-of-day balance is. Right. And your balance could be all over the place during the day, depending on what order transactions came in. And it's kind of arbitrary and kind of doesn't matter. Right. When they have close of business, they're like, okay, these are all the transactions we're recognizing on this date. And this is the final total. And we're going to start tomorrow from here. And maybe some of the transactions that came in, have you ever seen transactions that come in after 3 p.m. will be processed the next business day. Like that's real time in some business situations. So asynchronous is very valuable and a useful tool for anyone designing an organization and the communication and cohesion flows between people.

[00:49:56] Allan Stewart: And I especially like how it says information distribution, right? It implies to me like almost like a clearinghouse of information where a lot of times these are, you know, broadcast kind of things. Sometimes that's top down. It doesn't have to be, but we're broadcasting to everybody the same information so that there's an awareness and that those can be done asynchronously. And then it talks about synchronous meetings should be reserved for low latency collaboration. On complex issues. And I think related to that is it should be reserved for small groups of the people who actually matter for a particular problem rather than let's get everybody together in a, in a big, you know, town hall style to try and solve a problem.

[00:50:45] Dave Adsit: Yeah. I was going to point out the same thing about broadcasting is that, you know, we, we in distributed systems design, things like pub sub are fantastic. We publish events when they occur. And you can subscribe to them if you're interested. I don't know or care who's subscribing to my events. They're just being published. And so a concrete example in the, in the workforce or in, in the system that I operate is using a Kanban board to track the current status of work. And so we have kind of constraints or rules around work. When you start something, you move it from the ready to start to the in progress column. And then you don't have to have a conversation about that thing with other people. And then you don't have to have a conversation about that thing with other people who want to know if this is started yet or not. Because you've already told them the information is available passively at no distraction to you. And so it costs you as the individual working on the thing. It costs you nothing to keep other people updated. If you just move it into in progress, as soon as you start it.

[00:51:45] Allan Stewart: Yeah. Yeah. One of the quotes you highlighted it at one point. I like it as well. It says a significant source of failure demand for meetings and status updates. Is the desire of organizational leaders to keep abreast of who's doing what. And you don't, you don't need to call a meeting for that. You don't get, you can make it available.

[00:52:08] Dave Adsit: One of our friends talks about the different purposes for a meeting. We should probably talk about those at some point, but what is not on the list is updating status. And if you start calling a status meeting with this person in it, he will say status is not a meeting. And then he will email it. He will email you or slack you a link to the Kanban board where you can see the status and then refuse to come to your meeting that you called specifically to discuss status. And it's fantastic, right? Because it enforces good behavior across the system. So the last thing, the last section is what happens inside the boundaries is important. And so in this one, we talk about, you know, the boundaries, the ceiling and the floor, and what happens in the, in between there is very important. And I, I really, really liked the way the article ended. The last, the last two sentences were really impactful. They came to tie the whole thing together for me. And I very much aligned with this, right? It says companies are groups of people being compensated for having to spend some of their finite lifetime, not being with their partners, children, pets, or super weird hobbies. They deserve to be members of an organizations, which honor that time by ensuring that their work, has value and meaning. And for me in engineering leadership, this is what I'm always striving to do. I'm always trying to build an organization where people can be motivated intrinsically, you know, all a Dan pink and drive with autonomy, mastery, and purpose. I want to build organizations where people feel like they're creating and delivering value. Every engineer I've ever interviewed has had a desire to have a positive impact on the organization that they work in. And, and we owe it to our staff to create systems, to create organizations that honor the time commitment and ensure that their work has meaning and is valuable. And so to me, this kind of goes back to what I was saying earlier about how I, I get frustrated when people talk about efficiency, when, what they really should be talking about is efficacy. Like, are we effective at delivering value? It does. If we are effectively delivering value, it doesn't matter if we're less efficient than some other group that can't deliver value,

[00:54:26] Allan Stewart: right?

[00:54:26] Dave Adsit: Like I don't want every, we go back to some of the concepts and lean. I don't want to maximally utilize every resource because that's a highway and a traffic jam. What I want is to flow value through the system effectively so that people can feel joy and in what they are doing and what they're delivering and how their contributions to the company and the world are actually useful. Yeah,

[00:54:49] Allan Stewart: absolutely. I think it's a, it's, I agree that it's a great way to end the article. Having taken all of these principles, some of them like mathematical principles and like these ideas that can help you guide your system. It's so important to remember that how you apply them matters because you're going to be working with humans. You're going to, it needs to be, you know, moral or ethical in a way that you are keeping the humanity, right? That it's, it's humane what you do. Um, within those constraints that, that you find yourself in a business and that you're not getting so tied up in these other concerns for, you know, that we have to deliver, we have to make money or we have to chase some other goal that we've lost sight of what was important to the people who are actually working within the system. So that's the article. Work is work. As I mentioned at the beginning, check out the show notes to find the link to the article. The link and go read this. If you haven't, it's a good article.