Crafting Code Podcast

[00:00:16] Allan Stewart: Welcome to the Crafting Code Podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately I've been thinking about the limits of doing everything yourself versus building a team to do the work.

[00:00:32] Dave Adsit: And I'm Dave Adsit, VP of Engineering, and recently I've been thinking a lot about maintaining team morale through organizational changes.

[00:00:40] Allan Stewart: Our episode topic is 10x systems. So out in the industry, on occasion, I have heard about this mythical 10x developer.

[00:00:52] Dave Adsit: Yeah, and you'll definitely hear some people believe that it is a real phenomenon. And other people believe that it is not.

[00:01:01] Allan Stewart: Yeah. And we've talked about this off and on. But there was one day where you sent a message to me and some other people in a Slack group that really kind of, I really caught on to. And you said that there are people skeptical of the idea of the 10x developer, which is understandable if you've never worked in a system that enabled people to be 10 times as productive as another system when they're not productive. Yeah. And you study economics, they talk about how crossing a border can make someone 10 or 20 or even 50 times as productive in terms of their earnings. And they say you can do, they say you think that one person is different, or do you think that the system that they're in is different? And that message, like that concept, like relating it to the economics of like how much you can get done or how much you can make depending on where you live. You cross a border, like to a different state or a different city, and all of a sudden that has changed dramatically. And that really resonated with me on this idea of 10x developers.

[00:02:10] Dave Adsit: Right. Well, and for a little more flavor around that example, the idea is that you can take someone who is a cab driver in Haiti or Cuba and bring them to Miami, Atlanta, or New York, and they will make orders of magnitude more. More money financially by doing the exact same job in a different location. And that person didn't become 10 times better at driving. In fact, in my experience driving in cabs before the advent of Uber, I never thought any of them were any good at driving anyway.

[00:02:46] Allan Stewart: Yeah. And I like the economic concept and the system concept, because depending on how you look at it, yeah, they are. They are making 10 times more money, but they might also be now in a situation where the cost of living has also increased. So maybe, and if it increased 10 times, then they're kind of, it's kind of a wash. They might be doing the same work and getting paid more, but it costs them more. But if it only costs them twice as much, well, then they're ahead. Mm-hmm.

[00:03:20] Dave Adsit: Well, and even so, different qualities of life can come with... Even if you are economically in the same spot, you're making 10 times as much, but you're spending 10 times as much, you may actually have a better quality of life overall. Maybe you're eating healthier food, you have air conditioning, you have reliable electricity and water, et cetera. And so the differences between different countries just by crossing a border can actually fundamentally alter someone's life experience and how their outcomes change. So yeah, this to me is the fundamental issue at hand when it comes to, are there 10x developers? Well, if you exclude people who are really bad at typing and people who are really masterful at typing, you probably can't find that many developers who are 10 times as fast at entering code, like solving a problem and typing it into the system. Yeah. Yeah. Yeah. Yeah. Yeah. !

[00:04:42] Allan Stewart: And so there's a little bit of a gap there there because developers have given themselves a little bit of a gap there because developers have given themselves a little bit of a gap there because developers have given themselves down here, the fact is that the system that people work in and the interaction with people may account for 90 or 95% of performance. Right. And, and Deming wrote that in probably the 60s.

[00:05:11] Dave Adsit: I don't remember when the new economics came out, but that was one of the things that he, one of the concepts he pushed in that book was that you should not be doing performance evaluations when most of the difference in performance between people in the same system boils down to luck. And I mean, of course, anybody who's going to write an article or a paper or a chapter in a book about the importance of eliminating the annual performance review is automatically on my side. I have been, I have been a champion against the annual performance review for the last 15 or 20 years. Of my career. So obviously this landed well for me, but when you think about the actual impact of the system that people have on what they can do, it's unbelievable what different systems allow you to produce or deliver to in terms of value. Yeah. Yeah. I think about an individual developer,

[00:06:15] Allan Stewart: just depending on the system they create for themselves can make a big difference. So like sometimes we'll hear about, you know, these really clever developers who automate pieces of their work and all of a sudden they're far more productive because they're, or they're, or they're using better tools or, you know, they got their employer to pay for that IDE or, or AI assistant or whatever. And now they're, now they're able to get more stuff done more quickly, right? Like those are systemic improvements that they're making potentially just for themselves individually. But I think it's more interesting, to think about the developers that are making systemic improvements for the, for the whole team that they're working on, right? As far as practices that they're using or changes that they instigate in the team that makes everything better. I think Deming also said that a rising tide lifts all boats. So if you can find, find those people who help implement change, you know, Then it can help the entire team to, to improve.

[00:07:26] Dave Adsit: Well, isn't, isn't that one of the more well-known Larry Wall quotes is that the chief virtues of the programmer are laziness, impatience, and hubris. And laziness is the quality that makes you go to great effort to reduce the overall energy expenditure that you have to put out to make something happen in the system. And to me, that's all about, you know, automation, right? When we, we put in effort that hopefully has a positive ROI in terms of it pays back, the effort is paid back and paid back quickly in a short cycle. So there's a lot of activities that we can do. And we'll probably talk about those in a bit that improve our effectiveness as developers. I think one of the things that we've talked about, in the past is that back in the day, Google gave all of their developers 20% time. They said, Hey, take 20% of your work time, the time you're going to work, whether whatever that is 40, 50 hours, I don't know what Google expected, but take 20% of the time that you have devoted to work and apply it to something that is not part of your official duties, something you're interested in, whatever. And the reason that they did that is because they wanted to see innovation, but also they gave and data and data. Don't forget the importance of that. They could try something immediately at scale in a way that it was going to work and not just break and fall apart. And they didn't have to wait 10 years to gather enough data to know if it was going to work or whatever. You could throw something out there and see if it improved query performance or if it improved... I mean, several of our favorite tools grew out of people just trying something and then running it on Google. I think one of the big ones that everybody talks about is Gmail. People didn't like their email options back in the day, which is understandable. I'm not going to throw those email options under the bus. I think some of them still exist, but I do prefer Gmail to most of the other clients I've tried. So yeah, there's a lot of ways that you can see how the system itself affects the outcome more than any individual or their effort. Yeah. One of the classic examples that people use, at least in the Deming training space, is they'll take a bunch of marbles and a sheet of paper and they'll put a funnel over the piece of paper and they'll drop the marbles into the funnel. So they spin around and then hit the paper at some point and you can see where they indent the paper. And if you recenter the funnel over each of the marble indentations, you will very quickly start to move into a larger and larger and larger pattern. Josh Bolton Josh Bolton Josh Bolton Josh Bolton Josh Bolton Josh Bolton Josh Bolton Josh Bolton Josh Bolton Josh Bolton Josh Bolton Josh Bolton Josh Bolton the productivity or efficacy of a developer, right? You don't want to measure productivity in terms of lines of code written. You want to measure it in terms of how effective are they at producing the outcomes we want from this system.

[00:11:04] Allan Stewart: Right. And if you just focus on just the job at hand, just doing our job doesn't necessarily improve the system. You're not getting that continuous improvement. And so you just continue to get the same kinds of results over and over.

[00:11:21] Dave Adsit: Right. How many times have you heard a developer say, oh, testing the code's not my job. That's what QA is for. Usually if they're on my team, you don't hear them say it more than once. Yeah.

[00:11:33] Allan Stewart: So when we look at this from the economic perspective, it makes me think of different economic concepts, right? Like kind of we can use to inform We can use to inform our thinking on practices that we do. Right. So should developers be engaged in testing? Maybe. Like what are the principles? What are the concepts that come behind that? So there was a few of these that we jotted down to talk about. First being transaction cost. What does it take to move something from one state to the next? Right. So these are all the ancillary costs around the thing. Right. So it's not just how much time how much time does it take to do X, but before we can do X, what do we need to do?

[00:12:19] Dave Adsit: Right. Well, if you think about it, there's so to understand transaction costs, you've got to think about all the types of things people like to do. We like to talk about Uber as a way of reducing transaction costs between a person who wants to hire a ride somewhere and a person who has time and a car and would be willing to sell it if they knew who wanted to buy it. Right. So Uber reduces the transaction costs from near infinite down to almost nothing or a few dollars. Right. And so there's a lot of things like that. I basically as an economic concept, a trend, the transaction cost is anything that you have to pay to make something happen. Either finding the right people do it, whatever. Right. Anything that's secondary to the actual primary cost of the transaction. Of the thing. So, yeah, some of the examples that we had put down in previously are like, if you want to find music, the transaction cost for finding music when I was a kid in the 80s was you spend an afternoon in the record store, trying albums, trying records, trying maybe if you're lucky, they'll open a CD and play part of it for you, whatever. Right. And you're like flipping through all the records and the transaction cost for finding music now is extremely high. you have potentially increased the transaction costs of finding really obscure music because people who work in record shops like to get really obscure stuff that they buy out of the back of magazines and put on the shelf for people who want obscure music. And maybe Spotify doesn't cater to that group of people very well. I'm not sure. But for the general case, the transaction

[00:14:33] Allan Stewart: cost has gone down substantially. Right. And so I think about the things that you have to do before you can get started on work in software. Yeah. There's a lot of transaction costs around organizing the work, organizing the team. Sometimes there's transaction costs around things like merging code. How do you get that through your process? Do you have a transaction cost around... Code reviews. So the code review itself might be part of the work. Like if you did a value stream map, well, the actual review is part of the work, but all the time that it takes to submit the code review, wait for somebody to get around to doing it, and then they give you feedback and now you have to go implement it and send it back. Right. There's all that waiting time is part of your transaction cost.

[00:15:31] Dave Adsit: Well, and I like to think... I like to think about one of them that we've drastically reduced is around servers, server operations, right? Back in the early days of my career, if you needed a new server to run a new service, you would have to, after you got the case study and after you got the approvals and everything, you still had to go to the system operators and say, hey, we need to order a machine for this. They would have to spec it out, which might take them a week, and then they'd order it, and it would take six to eight weeks to come in. And then once it... got there and they unboxed it, it would take them a week or two to get it set up. And now you're talking 10, 12 weeks to go from, I need a place to deploy this code to, I have a server to deploy this code on. And now with cloud services, we can spin up and spin down new servers for testing and deploying and whatever in a matter of seconds to minutes, which is phenomenal in terms of that. Yeah. Unnecessary time, that lead time that it used to take to get access to a machine. Yeah. And I would say that our lives have improved drastically for it, particularly in software development, right? The only downside is that we've reduced the transaction cost on spending money at the same time. You used to have to go get approval before you could go order that $4,000 At the very least can provision new servers and services in the staging environment or the dev environment. And so the transaction cost on spending more money is also lower. Hopefully they're not doing $4,000 at a time. Right. You have to provision a very large machine to spend $4,000 in even a day.

[00:17:25] Allan Stewart: Yeah. But if it takes five minutes to get my server, then I'm already tapping my toes and wondering what's taking so long. Right. So I can get a lot done in that. I can get a lot of servers generated in that 12-week window that you were describing before.

[00:17:39] Dave Adsit: Right.

[00:17:40] Allan Stewart: Right. Yeah. I like the quote, if it hurts, do it more often. Or since I couldn't find a great citation for that, Jez Humble said, if it hurts, do it more frequently and bring the pain forward.

[00:17:52] Dave Adsit: Yes.

[00:17:52] Allan Stewart: Right. That's a mechanism for lowering your transaction costs. Find those things like getting a server or whatever it is. Deploying code. Deploying code. Deploying code. Review or deploying code and figure out, well, how can we do it more? How can we automate it more? How can we make it easier so that it's not so painful?

[00:18:11] Dave Adsit: Well, so another economic concept that we've talked about in terms of having a 10X system is economies of scale. When it took 12 weeks to get a server, we would usually run maybe a whole system on that server. We might run the database on it. We might run the front end and the web server on it. We might run whatever we could. We might run whatever we could on it to take advantage of that server. But you never wanted to run above a certain baseline CPU because you knew that if there were any load spikes, you're going to blow the whole system up. But then we started using software to control multiple servers and run virtualization across multiple servers so that it was easier to scale horizontally with similar servers and then run a bunch of VMs on top of them. So we did that for quite a while. And we got to the point where as an industry, that became common practice for anybody running a private cloud or self-hosted or locally hosted servers. You would put all of the servers into a tool like VMware that ran them all as though they were a single giant server. And then you could run virtual machines inside of that server. And it would manage moving things around and keeping track of CPU across everything. And then you had this, you created an economy of scale and that you could run a lot more things on a lot fewer physical servers. And then we evolved one level past that to containerization, right? We use tools like Kubernetes with Docker containers to allow us to basically build an abstraction, a virtualization abstraction on top of our virtual servers that we vision to the cloud. And so you can basically have a bunch of services that if they all needed to spin, if they went hot, they would take one server down and you spread that load across a bunch of servers. You get an economy of scale there where you can have tens of thousands of network requests or HTTP requests coming into your server farm, your cluster, and you don't have to worry about it the same way. But also you can benefit tremendously from having that ability to distribute the load across multiple machines easily.

[00:20:38] Allan Stewart: On the more people and process side of it, I think about economies of scale that you can achieve as you standardize different ways of working within your development team. Right. And standardized makes it sound kind of draconian maybe, but yeah. But when people work in similar ways, you get economies of scale in just in how they work. For example, if you have many developers who all know mostly the same languages, then it's easier for them to pick up work from each other, right? Or they can transition between teams or things like that, right? And you've got an economy of scale in that regard. But if you have something that's more niche or, you know, each member of the team has, a specialization, well, then you might have to wait until that overutilized person is free and available to do the thing that you want them to do.

[00:21:36] Dave Adsit: Well, and you know, one of the things that we used to do is grab some old computers and set up a build system, right? And building and maintaining a CICD system to deploy one app, kind of expensive, all things considered. Not necessarily expensive in terms of hardware or the software, mostly open source, not all, but in terms of the time it takes to keep those things maintained over the long period. And if you can say, oh, well, you know what? We have 10 teams and they each have a couple of apps that they're using, and they're all going to use the same build and deploy service or tool, then building and maintaining that and running a bunch of build agents becomes a lot cheaper as teams start to use the same tools. Then those tools cost less per team member, per person or per team that are sharing that system together. There's also a lot of other benefits like, hey, maybe your team has never used the new build server. You've never set up a new pipeline for build and deploy, but a bunch of other people have. So you don't have to go spend all the time it takes to learn how to use the system from zero. You can say, hey, would somebody who knows how come, sit with us and teach us how to do it? And you basically can, you leverage some of those concepts like mob programming or pair programming, where somebody, the person with the knowledge directs the person who doesn't have the knowledge yet in how to do the thing so that they're, they become more familiar with the tool in the process so that they can start to do it themselves.

[00:23:19] Allan Stewart: Sure. Sure. If only one person is doing it, if only one person is using the build system, it's very expensive because they're doing all of the work. And the rest of the teams is breaking it, is breaking the build continuously. And if you get multiple teams all using the system, right, then it scales outward, right? If you only have one person who's doing unit testing, then everybody else is breaking the tests. But as more people do it, more teams are doing it and you, and you build up that collective capability, it just gets easier and easier.

[00:23:53] Dave Adsit: Well, and that's one of the concepts behind the shift into DevOps or DevSecOps or whatever, DevXOps is that you take a, take something that is only known by a few and you start to distribute the knowledge more broadly so that you can operate more quickly. You have a greater number of people with knowledge of how the build system works or how the infrastructure works in the staging or production environments, and you can operate more quickly as an organization. But again, that's a system investment, right? And that's a school of thought that says we should have specialists and we should keep the specialists from being distracted with non-specialist work. And that includes programmers and operators and network administrators and database administrators, et cetera, et cetera, et cetera. And if we keep them all from being distracted with any of the adjacent work, eventually we will run into deep siloing and lots of handoffs to get anything done. Um, but if we can break that down and shift that information, that knowledge across the team more generally, then we have fewer of those bottlenecks.

[00:25:04] Allan Stewart: Another economic concept is the cost of delay. And, uh, this more than anything reminds me of what I think must be your favorite book based on the number of copies you've purchased principles of product development flow. Right. But I mean, it's a great, it's a great book and cost of delay is a prioritization strategy, right? We, if you do the things that have a higher, uh, if you, if you take the cost of delay into account, you can get more value done. And I think from like a developer standpoint, it's, it's the difference between taking, you know, the, the convenient or easy tasks versus the ones that

[00:25:46] Dave Adsit: actually make a difference in the system. Right. Well, and, and I think you're probably right. It is probably my most favorite work related book. I have easily purchased 75, copies of this book. Most of them have been given away in trainings that I've run. Um, but it is phenomenal in the way that it teaches how to think about the concepts of productivity and efficacy. And so when you think about cost of delay, you think, Hey, if, if feature a is worth a thousand looking at a feature B, if feature B is worth a hundred dollars, whether we get it now or later, or $2,000, whether we get it now or later, now you have an economic framework that you can say, Hey, I'm comparing dollars to dollars. What, what is my total value of my system? If I do a, then B versus if I do B, then a, if, if B is worth the same to me, whether I get it now or later and a, the value falls off quickly, I should do it. And get the value before it goes away. Right? Maybe I need to release a feature before a competitor releases the same feature, because if they beat me to, if they beat me to the market in terms of marketing the feature, I'm going to lose customers who needed that feature, but maybe B they're not going to build it or it's similar regardless. I don't know. So a lot of situations where you can use the cost of delaying delivery to help you the priority of the features that you're building. And that helps actually very much think about the fact that your team is going to be busy and productive, regardless of how you prioritize work, but you can make your team substantially more effective by having them focus on the more important work or the more timely work, the work that is worth more if it is done sooner. Right? If you have, if you have a trade show and you have to deliver, a demo by the trade show. If the demo is a week late or even a day late, the demo is worthless because the trade show happened. But so they like the same is true. Maybe, maybe more relevant to some of us. I traveled just today. Uh, if you arrive at the airport five minutes before your flight, that's fantastic because you saved all that time to do something else and you are still on your flight. arrive at the airport five minutes after your flight departs, well, that just incurred a big expense somehow. Either time waiting for the next plane, money rebooking, whatever, right? And so you think about the closer I get to arriving at the airport right when they're boarding my zone, the better off I am because I could have used that time for something else. Now, I do definitely have some friends who do not take advantage of that opportunity. In fact, the later they arrive at the airport, the more stressed out they are about the whole thing. And so arriving late creates negative value for some people and positive value for others. But you've got to think about the fact that you're going to drive to the airport regardless of when, and it's the same actions are being taken regardless. So it's the same amount of productivity. Regardless of the order that we do it in. But is it more or less effective? And that is one of the things that I think is super critical to software developers who are working in any kind of a business context. Are we being not just productive, but also effective? And can we be equally or more effective while being equally or less productive? There's a concept that you and I have been state state state state state state state state state state state state state state state state state state

[00:30:07] Allan Stewart: state state state state state state state state state state state state state state state state state things you get done and what you delay, especially if it has to do with the maintainability of the system, could have a big effect, a big impact on future work. So if you put off the hard thing, you delay it, you've got a higher total cost of ownership for longer in your system. Everybody's fighting against the code because, oh, there's that weird code that is hard to deal with, or there's that database problem that we just are scared to try to address. And it just continues to sap away your energy with this additional maintenance cost until you finally tackle it and make everything easier going forward.

[00:31:06] Dave Adsit: Right. And there's a related concept in there around total productive maintenance, which comes from... Manufacturing and lean manufacturing, right? Which is you should spend time doing maintenance as you go. Each machine should be serviced at a certain rate in order to maximize its total lifespan and its total uptime, et cetera. When I was much younger, I had a car that was supposed to get an oil change every 3,000 miles. And I was far too busy to do such menial things. Up until the day when... I was in the engine seized because I hadn't changed the oil for a while. So the cost of dealing with that far exceeded the 30 minutes and $30 it takes to get your oil changed. But I was economizing wrong, right? I was putting off my maintenance to the point where I created a big problem for myself. And I've seen software teams do that. I've seen software teams do the exact same thing. Maybe you're one major version behind on your primary language or toolkit or framework or whatever. And maybe that's okay. And maybe you just keep letting that maintenance, that upgrade cost slide a little bit at a time until you realize now you're several major versions behind. You're going to have a really hard time getting back up to date or getting caught up at that point. Sometimes tools just disappear. Entirely in the interim. And what would have been a small cost of a day to refactor this code, to use a different HTTP library or whatever, is now going to cost us a week. And it's going to be part of an overall bigger project that takes who knows how long, months potentially. I've seen teams take months and months trying to get caught up on the major versions that they have failed to update. And eventually those things bite you because eventually they stop supporting, I don't know, let's say node eight. It's no longer available. Very recently, node eight became not available. Don't ask me how I know that. I definitely didn't have a system that relied on node eight that had to be upgraded to node 12 in an emergency while all of the build pipelines were blocked. That would be irresponsible. You always have to pay the cost of the maintenance, in other words. And you can pay it a little bit at a time. Over time, or you can pay it all at once in a panic.

[00:33:46] Allan Stewart: The bill comes due. Yes. Another cost of ownership one that I think about is the carrying cost for unused features or not very often used features or even dead code. I can't tell you how many code bases I get into and there's just unreachable sections of code or things that used to be used and were not deleted when they were removed from the system. But they were just removed from the UI. You can't get to it anymore, but you still have to deal with the fallout of that code existing in the system because either it is still used, but not very often, right? Infrequently used code. And it's like, well, we still have to maintain it just like all the code that gets used every day, or it's not used at all, but you have to figure out

[00:34:37] Dave Adsit: which ones are not being used. Right. Well, and sometimes you have really good static analysis unreachable. Nobody, nobody can call it. And sometimes you don't. And on the edges of the system, you usually run into trouble with things like that because it can't reach outside the system and know what calls this API. But I will tell you, I have very recent experience with doing a production, doing an update from a deprecated major version to the current version of a framework and having to upgrade code. Everybody was pretty sure was unreachable, sure. And we didn't have time to take, to measure whether that code ever got called. Right. So that cost us extra time in terms of hours and days. The last time we did a major version upgrade on one of our primary toolkits. And, um, and when it comes to unused features, I have no idea why this is, but it seems like every time there's a feature that's either very infrequently used or unused at all. That feature always seems to have the most complicated code. There's always some weird quirk in that code that makes it that much worse to maintain over time. It's almost like gremlins are in the system adding extra for loops. So like nobody has maintained this code in forever. So I'm going to add some extra for loops and jumps and loops area, uh, switches and stuff just to make

[00:36:06] Allan Stewart: it harder when they come back to it. And oftentimes those are also the untested parts of the code. Right. So it's where fear lives. And if somebody does have to touch it, they try really hard. Like they'll go out of their way to add complexity, cyclomatic complexity, at least to ensure that this will only execute in the one condition where I know for sure I have to fix this bug. And otherwise I'm going to just hope that I haven't broken everything.

[00:36:38] Dave Adsit: Right. Yeah. Yep. We, uh, we definitely make poor choices. When we're working in code that's untested and potentially unused. So yeah, like you reduce the total cost of ownership and the total cost of maintenance by cleaning up as you go. And the good thing is, is that if you're using any kind of modern toolkit, you have source control. The code is not gone forever. It's just gone from the, the active branch or the head of the code, right? The code, the version that we're using right now doesn't have stuff that's unnecessary. And if we do determine that it was necessary, we need to bring it back. It's still there. It's in our source control. Just go back a few versions and there it is.

[00:37:21] Allan Stewart: And perhaps more importantly, it's out of the view of the current developers. So they're not worrying about it, adding extra mental speed bumps and, and, uh, and complexity for our human brains.

[00:37:36] Dave Adsit: Yeah. So another economic concept that I think about a lot, especially in terms of systems, this one is definitely, from the, the systems thinking space is stocks and flows and cycles or regenerative cycles, right? So obviously your stock is how much of something you have and you'll have typically inflows and outflows. And if your inflow is faster than your outflow, you build up a stock, AKA your backlog, right? The inflow of ideas is always faster than the outflow of features being built. Your backlog always grows. Um, you can also have the reverse case where the inflow is slow and the outflow is fast. Maybe that's your, your revenue to burn ratio. If you're at a startup, we have a big stock of money because of investment, but we are definitely spending it faster than more money is coming in from sales. Uh, and then you get cycles, right? So you can do a thing that causes some of the outflow, to turn into inflow in the case of, Hey, we're, we're a startup. We have revenue coming in. We have a stock of cash in the bank and we're spending money. Maybe some of the money we're spending is on marketing. And so money spent on marketing probably has a return on ad spend or ROAS as they like to say in the marketing department. So that return on ad spend, maybe we spend a dollar and we get back $5. And so spending money, increases the total revenue that we have. And so maybe we're at a good healthy spot there where we can pump more money into that feedback cycle, uh, that, and that, that positive feedback cycle that just re regenerates the, the inflow reduces the total spend and preserve some of our stock of cash in the bank. Right. So there's a lot of opportunities when you think about the system of software development to think about what are the stocks of things we have, what are the flows of the things that are coming in and going out? If you think about, I don't know, think about your build system. If you have one or two build agents and you've got 10 or 15 dev teams that are all trying to do continuous integration. So they're kicking off builds all the time. You may find that you are building up a stock of, or a queue of waiting builds, and you can increase the pipe. Add more build agents and clear that stock out by increasing the outflow so that it exceeds the inflow. And that's usually a good place to be. That's usually a good thing to do. Uh, especially if you have the ability to dynamically control the number of agents that you're running, like during the work hours for our team, we want to have 10 agents up and running. And at night when almost no one is working and only, we only need to do builds in the case of an emergency, maybe we'll only run one or even zero. If the system can be fully responsive to the number of items in queue, maybe we run zero until somebody requests a build and then we spin it up. Or maybe we always run one because we know it takes five minutes to spin up an agent and have it register itself with the build master. And so maybe we don't want to wait that amount of time during the workday or even at night,

[00:41:08] Allan Stewart: especially in an emergency. Yeah. Yeah. Cause if you don't have enough, then you have to wait. And if you have too many, well, then you're just spending money. Yeah. You're just wasting money. So, yeah, I think, I think it's important to understand that like a lot of these stocks have a cost. You mentioned the backlog, right? If you, if you're building up a stock of work that you hope gets done at some point, well, usually there's going to be something that comes up like backlog grooming meeting or something like that, where all of a sudden people are wading through the list, trying to, trying to understand and identify the, the, the, the, the, what are all these things from two years ago that we said we were going to do. We've never gotten around to them. We probably will never get around to them still, but you're spending time on them. That's, that's wasted money. Yeah. Right. Design inventory is another one that I see a lot. It is often easier for a UX person to design the picture of something that's faster than actually building it and getting it deployed. Yeah. And so they can make a lot of pictures in the time that one, feature gets done. And now you have an inventory of design that costs you something. It's not, it's not free to have those things or, or even the cues. Like I think about cues between people. I think when I mentioned that, you know, are you waiting around for somebody to review your pull request? That's, that's a cue. And there's a cost associated with that, with that stock of unreviewed code. Well,

[00:42:43] Dave Adsit: and in terms of a 10 X system or 10 X more effective developers, if you think about the length of your queue, that is typically very closely related to the overall perception of the efficacy of an engineering team. If you have a backlog or a roadmap that's months and months and months long, and somebody adds something to the end of it, their perception is that this team takes nine to 12 months to deliver any software. The reality is the team may be delivering software all the time, but the queue is so long that the perception of efficacy is very low. And frankly, the real efficacy could be quite low as well. There are often opportunities in a market that are fleeting. If you can be, I mean, they say that the first mover almost never is the big winner, but if you can be the second or third mover in a market, like, Hey, this, this feature is important to customers. Let's build this feature now. And we'll capture customers and we'll keep customers, et cetera, et cetera. So versus if we don't build it for nine months, well now everybody has it. I mean, how many times have we seen something that we think is innovative become commoditized? We were talking in the airport earlier today about cloud providers and cloud vendors. And it's like, how many actual cloud providers are there right now that you could host a website on and, and do you want infrastructure as a service or do you want platforms? Yeah. There's a lot, there's a lot of things. And that used to be very, very innovative and now it's become fully commodity compute and network our commodity. Well, and one of the things that I like to think about a lot is storage is commodity. And are we using, this is a total different tangent that we should talk about at length at some point in the future, but are we designing our database based on the constraints that were current in the 1980s? Or are we building and designing our database schematically? Josh Bolton Josh Bolton Josh Bolton Josh Bolton Josh Bolton Josh Bolton Josh Bolton Josh Bolton Josh Bolton Josh Bolton Josh Bolton Josh Bolton Josh Bolton system? Is it very cheap? This was one of the things that Google was able to do with their 20% time, right? They made the cost of experimentation very inexpensive for their developers by saying, hey, here's the Google infrastructure and the Google data. What can you do with it?

[00:45:27] Allan Stewart: What can we build? Kind of like the VC betting game, right? Not every experiment is going to pan out, but if you can learn from the experiments and some of them provide a big value, then all of a sudden it's been worth it. But you didn't know, there was no way to know which experiment is going to be the one that moves the needle in a positive direction. I add that at the end because some experiments move it backwards. They move the needle, but not a way you want.

[00:46:00] Dave Adsit: Not the way that you want it to do. Wow. We added this new feature and now churn is through the roof.

[00:46:07] Allan Stewart: So you have to figure out which way are you going to go with it? Yeah. You're going to have to spend some money to figure those out, to do the experimentation, but when they pay off, it can be a real benefit to the system at large.

[00:46:23] Dave Adsit: Right. And so if you think about, hey, my product manager has an idea for an experiment. Can we build this experiment and have it running in production with a subset of our production traffic today, tomorrow, next week? Or are we in a situation where it's going to take us well over a To get this experiment started before we can even start measuring or capturing data to measure. And we want that delay to be very small so that we can learn more quickly because we're always learning. But if we are able to learn based on actual things happening in the market with the customers we have in the tools that we have built, then our learning is accelerated accelerated drastically over somebody who is in a situation where they're like, I want to run this experiment. I have this idea for it. I have this idea for something we should build and I want to run this experiment. And so it takes me 13 weeks to get it into production and start testing it. Well, I'm definitely going to make a bunch of assumptions based on that being a successful experiment. And I'm going to build a whole bunch of design and features, et cetera, assuming I guessed right, because we always assume we guess right. And if I guessed wrong, then all of the things that were built on top of that are waste. And now I am very ineffective. Even if I produced a lot of things in the meantime, none of it mattered. And so I appear to be the kind of person who gets four ideas out a year, one a quarter versus the kind of person who gets four ideas out a week and tested and validated and iterated upon and improved and improved and

[00:48:02] Allan Stewart: improved. Yeah. The value of fast feedback is, is incredible in a huge spectrum of things, right? Anything from this bit of code that I'm writing now, is it correct? If I can, you know, if it takes me 10 minutes to run a unit test suite versus 10 milliseconds. Yeah. There's a big difference there in what I'm getting, what I'm getting done. And then that goes up all the way through the pipeline. You're writing the code or even before you're writing the code, you're, you're doing those designs. You're your hypothesis. And then, you know, how, how quickly can you get feedback from your build system? How quickly can you get it deployed to production? How, how quickly can you get data back from actual customers, right? All of those have feedback cycles and the longer the feedback, the more lag and just like a video game, it's hard to play well if there's a lot of lag.

[00:49:00] Dave Adsit: Yeah. What is it they say? You made the mistake of shooting at me where I was at the time versus be. So I guess I would say our individuals 10X developers. No, but also yes, because that developer put into a worse system is not going to produce at the same level. But, and so that's a no, right? It's not like you can't take a developer and they're going to be 10 times as productive no matter where they go. However, there are developers who when put in any situation will start improving that system to the point where they become 10 times as productive as a similar developer was in that system before. And so no one is actually 10 times as productive as all the other developers in the industry, but some people make systems that are 10 times as productive. And those are your real 10X developers. Those are your people who are continuously learning and continuously improving and continuously improving the system that they work in, create systems that are 10, 20, 50 times as effective as the system would otherwise be without that person there improving it. And so I say, yes, I've seen this happen time after time after time. People come in and improve the system. They get familiar with the system. Then they start improving the system. They start automating things. They start accelerating the rate of growth, accelerating the feedback, and producing better code quicker. And it's not necessarily about faster. It's not necessarily about typing faster to become a more effective developer. It's about getting the right feature

[00:50:49] Allan Stewart: done sooner. Yeah. I totally agree. And going back to the cab driver analogy, there are so many interconnected pieces, which is why that idea of economics, really resonated with me because these things interconnect in interesting ways. Because if you, like we said, if our cab driver makes 10 times more, but their cost of living is 20 times more, well, they're going to be in trouble. If it's only two times more, then they've got a windfall. But even if they're making 10 times more money, but their cost of living is 10 times more, you mentioned before, their quality of life might have gone up significantly. Yeah, it costs more, but it's better.

[00:51:41] Dave Adsit: Yeah. You're still spending all the money, but now you're spending all the money on higher quality food, higher quality lodging, higher quality air conditioning,

[00:51:49] Allan Stewart: higher quality services in general. And so we have to look at not just the speed of typing or, or even the speed of how many things are getting done by the team or by an individual, you know, per week or per month or whatever cadence you're wanting to look at. But also it might be that they're 10 times more effective. They're getting the same number of cards across the board, but the cards that they're getting done are more valuable or more important in some other, some other way. And that really makes me think that there's these interconnected practices that create virtuous cycles. So, so some practices, are, are more or more difficult or less valuable if you don't have the reinforcing practice. So I think about like feature flags is a great feature. That's a great practice, but if you only deploy once a month, then the value of the feature flag may be less because it takes time to get it out. You know, there's that cost of delay to get it delivered. And then if, you know, if you want to make a change, it's going to, you know, you can change now dynamically. It's better. There's, there's, there's more value than if you had to wait an entire another month to turn off the feature flag, but to do the next thing, you know, you're, you're limited, but if you've got feature flags and continuous deployment, now you have a lot of flexibility to be able to change very rapidly. Some other virtuous cycles I think about is like mobbing, mob programming or ensemble programming helps you. It helps you, focus on one thing because the team is just doing one thing, which feeds into the, the lean concept of limiting work in process. You get smaller. If you add small batch sizes to that and trunk based development, then all of those things are helping each other out, right? Trunk based development is a lot easier to do if you're only doing one thing at a time, right? If you're doing small batches, then that's easy to have a small whip limit because you're not going to be working on one thing for the next six weeks. You're going to work on one thing for the next two hours. And, and that feeling that progression, the flow is there.

[00:54:15] Dave Adsit: Right. Well, and that's one of the things is like trunk based development requires that you commit to the main branch every single day. And if you are working in a mob, you'll only be making changes on one thing at a time, which makes it trivial to merge into the main branch. Yeah. And if you have the same team of six or eight people, and they're all working on their own thing, and they're all trying to merge into main every single day. Now I've got to have whoever's first wins and everybody else pays the cost of merging. And it gets more and more complex as more and more people merge ahead of you. And so then you actually discourage yourself from working on similar features, right? You're like, oh, it's easier if we work on different things. And now we're creating knowledge silos within the team, and we're no longer working collaboratively to deliver value to our customers. We're working in opposition to each other. Like, well, I would have gotten my merge in, but so and so did something in this, in this one module that I was, I had to make changes to. And then I spent the whole rest of the week fixing the merge conflicts. Like that doesn't happen if you're doing ensemble programming, mob programming with limited work in process. And so you can be more effective as a team by collaborating more. Yeah. And you can potentially get, I've seen teams that have gone from having five people each take a week to do one feature to be working in a collaborative mindset as a, as a mob do those five things sequentially over the week. And it's easier and more effective to do that.

[00:55:52] Allan Stewart: Yeah. And then it goes back to those economic concepts, right? So like the, the cost of delay, for example, if you farm out all your tasks to those five, developers and it takes them all the full week, well, then it's not till Friday that you can actually realize the value on any of those things. Right. And that is very optimistic. But if they work together and they can get the first one, yeah, that is optimistic.

[00:56:17] Dave Adsit: The most likely Friday they start merging. And then the next week you're doing all the merge conflicts and all the testing and all fixing the regressions. And you're actually at the end of the second week before you get any value. But anyway, go on, tell us about a better way, Allan.

[00:56:32] Allan Stewart: But if instead your five developers work on the same thing and let's say it takes them a day and there's no merge conflict. So at the end of Monday, maybe Tuesday morning, now all of a sudden you're realizing value on one of those things. No, it's not all five. We don't have, you know, those other four things have not been started, but in started and in progress, isn't delivering value. It's only done that gets the value.

[00:57:00] Dave Adsit: Right. Software that is 90% 99% done creates zero value. If you can't ship it, you can't, nobody can benefit from it.

[00:57:08] Allan Stewart: Yeah. Yeah. So I think there are some good virtuous cycles there. Another one is around, do you do testing? If you do testing, then that gives you faster feedback. And your testing might benefit from a CI server. And you might do tests. You might be better at testing if you do test driven development, which then improves your design. That makes your code more testable. Yeah. And so you're creating this virtuous cycle. And so these interconnected practices, I think they go back to the economic concepts. They go back to kind of a foundation of, of principles and values. Like what, what are you trying to accomplish? If you have, if you have that principle in, in mind that you want to have collective code ownership, what does that do for you? Right. It gives you better economies of scale. It helps with your cost of learning and ex experimentation. And so these things kind of build up on each other and, and the 10X developer or the 10X system really benefits from you adopting those set of practices. And so it's not, it isn't, it's not just one thing, not just one thing that we need to accomplish to create the 10X system. It's a whole bunch of interrelated things, which if you have the discipline, you have the mindset of accomplishment. Yeah. Makes everything that much better.