Crafting Code Podcast

[00:00:16] Allan Stewart: Welcome to the Crafting Code Podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately I've been thinking about older video games and whether they continue to maintain their online servers.

[00:00:33] Dave Adsit: I'm Dave Adsit, a VP of Engineering, and recently I've been thinking about the value of joining or building a community of professionals.

[00:00:41] Allan Stewart: Our topic today is constraints. I think it's pretty common to think of constraints as a bad thing. You're being constrained, help, help, I'm being constrained or repressed or something. thing. But being completely free of constraint can actually be more difficult. It's like paralyzing because you've got too many options, too many choices that you have to choose from. Too many constraints is also probably not going to work out because you don't have enough flexibility. But a manageable number of constraints makes things better.

[00:01:16] Dave Adsit: Yeah. I've thought a lot about constraints over the years. One of the things that I've thought about is, it's probably a common story, the story of the kite, right? You're flying a kite with your kids and you've, or as a kid probably, and you've got a string holding it down to the ground, tethering it to the ground. And you see the kite way up there in the air and the wind whipping and it's struggling. And you can tell it wants to go higher and higher and higher. And if you think I'm just going to cut the string and then it can fly as high as it wants, months, you'll notice that your kite sort of just falls out of the sky because that tether, that constraint is actually what enabled it to control the flow of air and stay aloft anyway. So we like to think about constraints as though they were sort of a negative thing, but there's actually a give and take between what constraints you have and what you are are unable to do because of them.

[00:02:14] Allan Stewart: Absolutely. I've heard it said that constraints make great engineering. I couldn't find an exact quote that sounded like that, but I think it sounds pretty good. And so I'm going to keep saying it.

[00:02:25] Dave Adsit: Yeah. I mean, I've heard similar things over the years, right? One of the things that I've heard is engineering is doing with $1 what any idiot can do with $2. And we had a previous training with Mary and Tom Poppendick when they came in and taught us at one of the companies I worked in. And one of the things that Mary said is that when you apply a date and say, hey, we're going to ship on this date, whatever's ready, that is what enables real engineering to happen because now you're making trade-offs. Now you're picking the most important things that can get done by that date. And so we like to think of constraints as holding us back from doing our best work possibly. But the reality is that sometimes the constraint is the thing that enables us to do good work and actually get something across the finish line.

[00:03:21] Allan Stewart: Yeah. Yeah. That's interesting. And we're going to talk about some different kinds of constraints here, but it just strikes me that time is one that you have very little control over. It marches forward where we are all time travelers moving forward at a rate of one second per second. Yeah. And always in the same direction too. Yeah. Always traveling forward, never backwards and always twirling, twirling into the future.

[00:03:46] Dave Adsit: Must be one of the constraints of our universe or at least our perception of it.

[00:03:50] Allan Stewart: Yeah.

[00:03:51] Dave Adsit: Let's talk about some of the different types of constraints. I think the first one and the most obvious one is technology. What technology can we use at this point with the problems we're trying to solve. So what are your thoughts on technology as a constraint?

[00:04:10] Allan Stewart: I think your existing tech stack is usually the first one that I think about. If you're Greenfield, you're starting a new project or there's a new company or there's a lot more options of what you can choose because you don't have anything existing that's limiting you. But as soon as you have a product, then those choices choices, constrain your future choices. That doesn't mean you can't change the tech stack, but it just, it becomes more difficult, especially if you want to avoid that tiger team trap, where you're going to get the people to come in and make the second version of the thing parallel to the existing version. And it's going to, this time we're going to do it all right. We're going to do it better. And it's only going to take just as long as it took to build the first one in the the first place, right?

[00:05:01] Dave Adsit: Or longer, maybe. Well, and even in a greenfield system, you are constrained by the technology that you know, or are willing to learn. Absolutely. There's a project out on GitHub that I look at every once in a while, the Ouroboros Quine, and it has 100 or 101 programming languages in it. I don't know most of them. And I've been around for a quarter of a century programming software professionally. And I've used a fair number of programming languages, including going through the book on seven languages in seven weeks that was popular a few years ago. And a lot of the languages that exist, I don't know and wouldn't dare try to pick up and use in production for anything that mattered.

[00:05:55] Allan Stewart: Yeah, I think about what is out there in the world? What can you get your hands on versus what are you going to have to build? Because there's a certain amount of, we use technology to build more things, new things. Nobody's interested in the software that has already been created, but what are you going to use to build something new? right now now that you can get an ai from as a commodity from a cloud provider what are you

[00:06:26] Dave Adsit: going to do with it yeah so there's a lot of different technology that you could pick and there's a lot of things around the technology with that would constrain you right say you want to use a specific database provider you're like this database would be perfect for our solution okay great are we cloud hosted is it available as a service inside of our cloud provider do we have have to self-host it? Do we have to spin up servers and manage the cluster and manage the storage and do all of those things in order to use this database? Okay, maybe that's worth it. Maybe it's not. Those are things that we need to consider when it comes to even a technical choice that if we're thinking, hey, it's a green field, I'm going to pick best of breed for every technology that we are going to use. And the only constraint that I have is that it has to be the best. Well, if I'm building a web application, I'm going to have to put JavaScript in the front end somehow, because that's what the browsers know how to talk. So am I going to do that through vanilla JavaScript? Am I going to do that through TypeScript? Like, there's still a lot of choices. And what does best even mean in that case? It's very hard to even know. And most of us aren't even working in that space. We are not fully unconstrained in the technology that we can use. We have some existing system somewhere, whether it's a prototype or whether it's the legacy code that's been in existence for the last decade. We have technology that we have to work with and hopefully enhance, right? Use or at the very least leverage it to solve problems for our users. And so when technology is a constraint, I think it enables us to think more about, well, what is the problem we're trying to solve? And if we don't have technology as a constraint, I have seen teams just spin and spin and spin in on picking the perfect tech before they even get started on solving the business problem. And you can frankly run out of budget, which is one of your other constraints without even making a technology choice.

[00:08:38] Allan Stewart: Yeah. I think sometimes it's a lot better to settle for something that's a little bit suboptimal while you're dealing with these other constraints and move forward. And you might find out a lot of times you find out that the the decisions you thought were important at one part of the project are not important later for various reasons, either because it's like, oh, this tech actually is just working fine. It's okay. Or something else has changed about your situation that makes it kind of dangerous to just assume that you can pick the perfect thing and never have to change it.

[00:09:15] Dave Adsit: Well, and I like to think, I realize that suboptimal is probably the perfect technical term, but I like to think we're making good enough technical choices for now. And I think that for now is important because at this point in time, given the other collection of constraints we have, this is a good enough choice for the tech we're going to use. And we can change it later if we need to. I think that if you and I were to get together on a new project to start up today, we would probably pick a certain tech stack that I can imagine in my head. it might not be the same tech stack that other people we know would choose but I think I know which back-end language we would pick I think I know what cloud provider we would pick I think I know what front-end toolkit we would pick and those are all perfectly adequate choices for any web-based software development problem today and then if we you you know, if that project got really big and we said, Hey, we've got these other features we need to build. There's other stuff that we need to do. We might introduce additional technology at that point to solve those problems.

[00:10:24] Allan Stewart: Right. So another constraint I think about is memory and time, basically these computational related things. And you could include things like bandwidth width in there, right? So mobile really brought bandwidth to the front. Some years ago, you know, iPhones were coming out and Android was coming out. There are things that you can do, but they can be computationally expensive or difficult. And in some ways, I think Moore's law has spoiled us because there's like a whole generation, maybe a couple of generations of engineers who don't think about some of these performance characteristics very often yeah because computers just get faster it's like oh well just buy just buy a new computer buy some more ram buy a gpu that was crafted for uh for bit bitcoin mining or something like that and and you move on your way but those those constraints are still there it's just that you are kind of

[00:11:33] Dave Adsit: of kind of working around them working around them solving them in a different way right i mean one of the one of the earlier home computers i had was a 386 um and it was it had 640k of core memory we had the expanded memory but most programs couldn't use that so the the uh the getting all the way up to a gig or not a gig a gigabyte no no it was getting all the way up to one megabyte of RAM was only feasible for programs specifically coded to use that extra RAM. Everything else kind of had to live within that 640 kilobytes. Today, I was talking with one of my engineers about a new program we're test driving within our company. And he was saying, hey, when I run this program and I have all of my Docker containers running and my three IDEs open at the same time. I'm running out of RAM and everything is getting really slow. And I'm like, well, what are the specs on your computer? He's like, well, it's an M1 Max with 64 gigabytes of RAM. And my response was just like, that's kind of it. I mean, that's not the top of the line computer in our department, but that's not too far down the list. You've got a pretty cutting edge computer. We're going to have to potentially give up something if you want all of these applications to run at the same time. Maybe you're going to have to cut back on the RAM and CPU that are allocated to Docker or turn off some of the features of this software project or software tool that you're using, or I don't know, maybe only run two full IDEs at a time instead of three. And all of these are memory and computation related constraints to what he was trying to accomplish with his laptop. I mean, I know that this doesn't mean anything to a younger generation of people who grew up with amazing laptops, but the 640K of RAM that I had in the laptop or in in my desktop computer was not even available for purchase and a portable computer, which, you know, is a precursor to a laptop.

[00:13:55] Allan Stewart: Yeah. Yeah. I was reading an article about how in, I think it was 96, Doom comes out and now today you can play Doom on all kinds of weird devices. Like people put it in like tweets and like run it on Arduinos or, you know, stuff like that, which is kind of amazing. Right. Um, and I, but I, but I try to load up a website and it's just as slow as it was 10 years ago. Yeah. It's, it's, it's slow. It's bloated. It's, it's kind of, uh, it's kind of a sad state of affairs sometimes.

[00:14:34] Dave Adsit: Yeah. Yeah. We all thought that the web was going to get really fast when we went off of dial up and onto cable modems, but it turns out that we just started downloading more crap. Every, Every website has to download a whole bunch of front-end frameworks in order to operate. And so if you don't already have those cached properly, you're going to have just as slow a web experience 15 years later.

[00:14:55] Allan Stewart: So I think about it in terms of when we're working on coding projects, a common place where this kind of performance consideration can come up is around databases. Databases are often a bottleneck in a system. And so then you have to start making some of these trade-offs of computation, right? Things like bandwidth and memory, the IOPS that they're using up in your cloud provider. Those are all things that you might have to work around because they're constraints that you have. So that might mean in some cases, the right answer is to denormalize your data because adding one more join to that table, to that query, one more table joining in there is just going to wreck the system. Those are just constraints that are kind of built into computation that although we can sometimes ignore them because we just get better and better machines, ultimately it's really fast. Like you take something like a database and you start doing the Cartesian product of multiple tables and the data just goes exponential and you can't do that forever.

[00:16:12] Dave Adsit: Well, one of the things that I like to think about is that I like to remind people that the normal forms were actually a response to a constraint, which is small disks. Small, very expensive disks. You can get a home server to be your media server or whatever and throw it in a closet and forget about it and have 20 terabytes of RAID 5 disk available for movies and music and whatever. But back in the day, nobody had a disk that big. All of the solid state storage in the world may not have been 20 terabytes if you go back far enough. The idea that your company would have virtually unlimited, functionally free solid state storage, or not even solid state, like magnetic storage for the database server, it was a ridiculous concept. And so you had to be conservative about every byte that you used. You had to conserve them all, use as few as possible. And so normalization was an attempt to trade off compute time for storage cost, right? You were pushing down storage costs by normalizing data, and it comes at the cost of more compute as you do a join and bring data together from multiple tables. Joins and filters cost more than flat table structures. And so we've trained a whole generation or several generations of developers that you should always normalize everything. thing. And so people believe it without thinking about what trade-offs you're making. Am I trading? What is my constraint in this case? Is it memory allocation? Is it storage on disk? Is it compute time? And is it, which of those applies more when I'm writing versus when I'm reading? I've, as recently as six months ago, had a discussion that went round and round and round in our architecture guild about whether we should store multiple rows in a data table, and then the updates would be more expensive because the updates would have to update anywhere between one and a hundred rows of data because the where clause wouldn't be where ID equals, they'd be where category name equals, right? And then, so you have to update multiple rows and do that as a set. Is that better or worse than normalizing that table? And this table would have ended up being three tables, right? Because it was a many to many concept. And then adding that complexity of compute every time we read and every time we update and all of these things. And it was a very heated debate because we had some very very dogmatic approaches to what good software is supposed to look like. When the reality is, what good software is supposed to look like depends on the constraints that you have. And in this case, our constraint was we needed very performant reads. And the writes happened very infrequently. And so it wouldn't really matter. But, oh, also, we host our database in the cloud and we have virtually unlimited storage space for the database and so adding a few more rows versus normalizing this table really wasn't going to cost us anything so yeah what makes me

[00:19:56] Allan Stewart: think like which is better and the answer is yes yeah they're all better for for some for some reason instead of constraints right for some set of constraints one of these is better but which Which one depends on a more holistic view?

[00:20:13] Dave Adsit: Well, I mean, you think about the very first spacecraft that we sent into low Earth orbit or to the moon had less compute power than your smartwatch. Yeah, you can guarantee that they were very cautious about how they used every byte of storage. Meanwhile, I use my smartwatch to set timers and know what time it is. I could have an analog watch that ran on a spring and get virtually the same functionality, but instead I have a supercomputer. So constraints change over time. Yep. And you don't even have to defrag the disk on it. I don't. So speaking of time, one of the other constraints that we have always is time. We hinted at it earlier, but you think about the time, the wall time, the calendar time, right? How long can we spend building this feature? I like to think about this in terms of bets. For me, it's not when would we like to ship it? We very rarely have a hard fixed date. Sometimes you do. Sometimes you have to launch a feature by a certain date for a trade show, or you've already bought an ad space and you have to do something. A lot of times we don't have deadlines. We have kind of sort of sad lines. Like no one is going to die. Someone might get a little bit sad for most of the software that most of us write. And I think it's important to think of time and date deadlines in that way. But I also like to think of them in terms of what is the bet we're willing to make? Because the time that an engineering team spends on something is literally money to the company. And I like to, when people ask me, well, how much money is it? What heuristic should we use? I always tell people your team of six to eight is about $20,000 a week fully burdened. And it might actually be more than that. And you might have a smaller team than that. But if you just say it's about $20,000 a week that we're spending on this, how big of a bet are you willing to make in terms of money before you need to know if you're on the right track or not? I like that.

[00:22:23] Allan Stewart: And I like what you were saying before. I think that time constraints are often used poorly. Yeah. Especially when time and scope are both constrained. And that happens so often. What's the thing we call the iron triangle? Yeah. Time, scope, and... Quality. Quality. Yeah. And so that happens so often that there's kind of been a visceral reaction from developers. A lot of them don't want to work in a deadline. line. But I liked what you were saying before that, you know, real engineering starts to happen when you're on the clock. Right. It makes me think about the Martian, right? It's like, oh, let's, let's do some, let's science the heck out of this problem.

[00:23:06] Dave Adsit: Yeah. Well, and we've had real experiences like that in the U S space program, right? We've sent people up into space and then had things go wrong. And then we've said, okay, we have this many hours to solve this problem before they run out of oxygen or they actually suffocate on CO2.

[00:23:25] Allan Stewart: Yeah.

[00:23:26] Dave Adsit: Let's fix it.

[00:23:27] Allan Stewart: And I think if we use time well as a constraint, I think you can get some of those benefits, but you're trading off with something else. And I think that that's always important as we examine any of these types of constraints is that they're all trade-offs and often against each other. So if you're going to trade off in time, then quality or the other one, or scope, they might have to change, right? You can't necessarily get everything that you want. You can't say, well, we want to get the Apollo 13 astronauts home and they're alive, but also we're going to extend their mission by a week. or also they need to bring back an extra, you know, 500 tons of moon rocks or something. Like you don't get to choose all those things.

[00:24:21] Dave Adsit: No, you have to give something up almost always. And I will say that, you know, I was just talking about how we've trained people to believe that normal, higher normal forms are always better without consideration for the trade-offs that we're making there. Well, I have been trained, I was trained either through directly or indirectly or by experience. And I think a lot of other people were too, that you should always work towards scope, build the right scope, ship the right thing, scope, scope, scope. And so it actually took me a long time and a lot of thinking and wrestling with that concept that having a date and letting the scope vary can actually be, it can actually be a much more powerful way to build a software product. Focusing on the high priority things and getting them in front of customers sooner pays tons tons and tons of dividends in ways that you wouldn't expect until you actually try it. And I wrestled with that for literally years before fully embracing that mindset shift from we're going to build the scope. And then when we hit the scope, we're going to ship it. And we're not going to let the scope creep. We're not going to whatever. If you build with a date, if you say we're going to ship this software in two weeks, what's the most valuable thing we could get done in two weeks? Well, you might be a week in and you discover that what you've been building isn't the most valuable. You actually could build something else instead. Perfectly fine. Do it. If you found something more valuable to build, you should shift focus and work on that because you still have that time. You still have that week. And, you know, honestly talking with Tom and Mary, when they came in, they were talking about, you know, uh, improving when Tom and Mary came in, they were talking to us about actually, um, six to eight weeks, right. They're talking about longer frames of delivery that we wouldn't even consider fast delivery anymore, but they were coming from a different context where people were shipping once a year, maybe twice a year, every six months. Maybe if you're super aggressive, you're shipping once a quarter, and they're talking about cutting that in half again, and delivering software two or even three times at a quarter. I mean, my teams now, most of my teams deliver more than once a week. And that feels like a good cadence. We haven't quite pushed it all the way down to start a feature in the morning, ship it in the afternoon, go home for the night. But maybe we'll get there. And maybe that's too too small of a batch. I don't know. Yeah.

[00:26:52] Allan Stewart: Yeah. There's an interesting question there about how, how often the customer even wants that level of change. I think you can get too fast. Right. It also makes me think about scope. There have been some projects where I feel like we've had a really good understanding of what the scope is. Like there's, there's a particular problem that we're trying to solve. We understand it really well. There are aspects of scope that we, It's just like, we just know. And so we can break it down along lines of scope and ship out incremental value. But I agree with you as far as it's taken me some time to realize that sometimes we don't know what we're supposed to be building. And so using time instead of scope can actually be better because the scopes are kind of like the features themselves. Like, what do we even want to build? We don't fully understand. And so we don't know how long it's going to take. and it's going to take a really long time. Well, what can we do in a week or two weeks or six weeks? Those can be better ways to constrain ourselves so that we learn. And then when we learn more, we might switch over and start thinking about scope again, because now we have specific things we want to accomplish. And then you just trade off between them depending on which one actually matters to you at a given time.

[00:28:13] Dave Adsit: Well, and I will say, I am definitely an advocate of low cost discovery techniques, getting things like prototypes, even paper prototypes in front of customers early, having interviews with customers to discover what their needs are. But I will say in almost every project I've ever worked on, it's not until you put the first version of the product in front of somebody that you really find out what they needed. And it might not even be the second version that they needed. You might be a few versions in before you really understand the problem and the solution that needs to be built. Because when people come and ask us for solutions and software, they very rarely understand what is possible. And when they see the solution that you built based on what they told you, it's almost never what they really wanted. it. So we need to plan for a few iterations. And that means going fast, at least at first, going fast enough to get the kind of feedback that we need so that we can build the right thing. And so rather than run away from the calendar, and I know we've all been beat up by, here is your fixed date. Oh, and by the way, here's your fixed scope. And by the way, it's going to double over the course of the next few weeks before we hit that date, because we don't know what we want. And of course it's going to grow, you know, rather than running away from those scars, if we can lean into, we're going to take a date and that's going to be our delivery cadence. We're going to deliver every two weeks or every four weeks, or I mean, once a week, I don't know, whatever the delivery cadence is, you're gonna say, we're going to spend this much time and then we're going to ship something. Whatever we have at that time is what we're going to ship. So let's make sure we build the best, most valuable, most important thing that we can in that time box. And then we put it out there. If we can actually get ourselves to that point, I mean, that doesn't mean that you have to give whatever you have at the end of a week to every customer in general release. You can definitely keep it behind a feature flag, give it to alpha customers, test it internally, give it to a handful of friendly beta, excuse me, give it to a handful of friendly beta customers and get some real feedback on working software so that when you have that bigger batch that is ready for all of your customers, you can be sure that it's something that's actually going to work and be valuable to them. I think we've developed techniques as an industry like feature flags that allow us to to bundle up our daily releases into something that we can give to customers on a less rapid cadence. Because you're right, some customers get burned out with the continuous change and some customers thrive in it. Some customers can't believe they get the opportunity to actually steer your product development by seeing the daily release that comes out. Depending on the software, I'll sign up for the alpha channel or I'll sign up for the slowest possible release channel because it's software. I don't want to see updates. I just need it to work the same every day for the next year.

[00:31:28] Allan Stewart: Well, if time is money, budget is definitely a constraint. So there's the budget that turns into time as far as how long can we afford to pay our developers, but then I think there's also budget constraints in the form of, can you pay your way out of a problem, right? Like if you, if you can just turn on the auto scaling for that cloud service, yeah, you're going to, you're going to be paying more, but, or you're going to, you know, vertically scale in some cases. And we don't think about that quite as often, or at least not in the same way, but you know, maybe you're moving up from a T whatever micros to a larger instance size or you're upgrading to the next tier of managed service, sometimes that's a great way to move forward and avoid really complicated things that are going to be difficult to solve. And basically you're buying yourself some time.

[00:32:28] Dave Adsit: It's 100% right. The question of what can we afford goes back to that whole idea of should we build it or buy it? Sometimes you build it because it's not available on the market, but a lot of times we build it because we have this not invented here syndrome, and we should consider buying more often than we do. If you have the money. I've been, I've gotten financial advice in the past that doesn't necessarily apply to your business but it applies to you as a person. And they basically said, if you can afford it and you, so you have a task, let's say the task is yard care, lawn maintenance, mowing, edging, fertilizing, weeding, all of that stuff. And if you enjoy it, you should do it. But if you don't enjoy it and you can afford to pay somebody to do it, you should pay somebody to do it and do something that's higher value to you as a person. And so higher value might be, I'm actually working on the business and I'm actually literally making more money. Or higher value might mean I am spending time with my family and creating bonds and memories with my family. and so we need to be thinking about budget in a in a way that is what are what are the trade-offs that i'm making by spending money in this way um but you could probably get any decent software team to build just about any piece of software right they'd have to do some research they'd have to do some learning but they and and you'd have to give them a lot of time but could you build your own identity service? Of course. Could you build your own web server with the right knowledge and materials? And if I can hire a couple of people that know a lot about building web servers, yeah. Should I do that when I could, for $47 a month, buy a web server as a service from a cloud provider? Probably not. I should probably spend that money more judiciously. And so, you know, getting ourselves out of that non-inventive here syndrome and leveraging the tools that are at our disposal is usually a good value for our money. I do think it's funny that a few minutes ago we were talking about memory and time, compute time as constraints. And here we're talking about the money is the constraint and we're just going to buy memory and compute and network and disk and all of that. Right. And it's, it's fair. Those are the kinds of trade-offs that as engineers, we have to be thinking about all the time. What do we have more of? Do we have more money or do we have more time to build something?

[00:35:16] Allan Stewart: I think you get bloat in the same ways, right? So in the same way that we just leaned on Moore's law and downloaded a gigabyte of JavaScript into the browser so that you can run your front-end framework. Well, we do the same thing with-

[00:35:32] Dave Adsit: We can validate a form. So you can validate a form.

[00:35:36] Allan Stewart: Well, sometimes we do the same thing with like, you know, well, we bought a bunch of services, we bought a bunch of online tools. And so you have to watch out with these different constraints, because if you aren't constrained by money, you can use it up really quick, right? You're not constrained by the number of operations that your CPU can do per nanosecond, or you're not constrained by how long it takes for the disk head to seek to the relocation on your spinning rust. Well, as soon as you're not constrained, you might go overboard. And so you have to watch out for those areas. And sometimes it's the right decision, but you need to go back and revisit those decisions periodically to see, are they still right? Or now that we're not a young scrappy startup and we have some money, maybe we should spend some of that money as an investment to pay down some debts. They might be technical debts. They might be literally we're spending hundreds of dollars recently I had an experience where we were spending hundreds of dollars on Google map API calls because we had just done maps the same way in every situation on the website. And in one situation, we needed to make an expensive call. And in other situations we did not, and we were paying the same amount regardless. So, you know, at some point you might save yourself hundreds of dollars a month by going back and looking at those things and say, oh, well, we can make a little change now that we've learned, now that we've grown, that wouldn't have made sense to do in the beginning.

[00:37:24] Dave Adsit: You could spend more engineering time to optimize multiple different calls, right? I think about constraints changing over time too. We talk about budget and we talk about things. One of the things that I used to spend a lot of time on when I was young and had no money was organizing my music collection. I would take all my CDs and carefully rip them to MP3s of the highest quality that fit on my player. And then I download all the ID3 tags and all of that stuff so that I could have a high quality music on the go experience and keep all my CDs and their jewel cases pristine, right? And I spent, I can't even count the number of hours that I spent on this type of project. and at this point in my life I would much rather just give Spotify $15 a month to give me unlimited access to most of the music I want there was still some of the weird stuff that I listened to as a kid that's not available anymore and and every once in a while I think oh man I wish I could listen to that one song by that one band that doesn't exist on the internet anymore and I don't I just listen to something else instead because the time it would take to manage all of that stuff is not worth the trade-off. It's not worth the cost. I would much rather spend the $20 a month or whatever to have somebody else handle all of the cataloging and identifying and ripping and managing and whatever needs to be done to have a good on-the-go music experience so that I can plug my phone into my car and just listen to whatever I want. So the constraints you have today may not be the constraints you have tomorrow.

[00:39:08] Allan Stewart: I think that shows up quite a bit as we transition over into company priorities and direction, right? We've started with a lot of the technical things. And as we move further and further in the socio-technical sphere, we find out that what the company wants to do today is different than what they wanted to do yesterday. And those things can change very rapidly. Sometimes because you've got the CEO squirrel syndrome, that every shiny thing that they see, they want it. And that's the kind of that immediacy. They want to take that. But other times it's just timing. The last customer you talk to has the most important problem. Well, yeah. Yeah, because they're the one that you're thinking about.

[00:39:57] Dave Adsit: Yeah, I mean, definitely your company direction can change. Sometimes drastically. I think we've worked at companies that have had drastic changes is in priority and direction and focus. We worked at one company where the customer we were trying to market to changed several times over the course of a couple of years. We were targeting the final user of our product. And then for a while, we were targeting the manager of that final user. We were like, oh, they'll buy it for their whole team, right? We're like, actually, it's not even that department at all. the department that uses our product, they don't have the budget to buy it. This other department has the budget to buy our thing and then they'll assign it to them. And then after a while we realized, oh, that other department doesn't care about the same things as the person who would use the software and they don't know how to evaluate it anyway. And so we went kind of around and around and around in circles until we finally ended up back at, hey, the person that uses our our software is the one who's most motivated to buy it. Right. So of course we were building different features for each of those different groups because they each had different, they valued different things and they had different constraints within their companies. But it can be hard to know what the right thing is and how to build it. But those are some of the constraints we have is like, Hey, today we're working as a digital product and tomorrow we're working as a physical product. And the day after that, we're working as, I don't know, some kind of a hybrid thing or cloud service type of a thing or whatever.

[00:41:39] Allan Stewart: Yeah. And I've often seen that the technical side of things, technical problems and business problems are not well aligned. So you might have a strategic goal at the company that gets upended by an incident or, or some, or a potential incident that is coming up because you can't scale to meet demand or something has fallen over in the data center and you have to deal with that. And so there was a technical concern that preempted the business concern. But I've also seen it the other way where there's a part of the system I'd really like to fix because it's hard for me to sleep at night knowing that there's this bad code just sitting there being used every day, but it kind of works well enough. And so you move on because there's something that's more important to the business. You ignore a problem for a while, as long as you responsibly can, because you're getting value doing something else.

[00:42:42] Dave Adsit: You know, I have often heard developers complain that marketing forgets that Black Friday is coming every single year until it's almost Black Friday. Meanwhile, developers forget that Black Friday is coming every single year until it's almost Black Friday as well. Of course, that means that now we have two competing priorities. One of them is how do we scale the system to meet the 10x load that we anticipate? And the other one is how do we launch all the new features that we already promised in the marketing campaign that goes out before Black Friday? And so, yeah, the company priorities, I don't know. No, they're hard and they change often. And honestly, if they didn't change often, that would be a bad sign for a lot of businesses because our ability to predict what the market conditions are going to be like in two years are very weak. We do not have that skill. And if you had that skill, you probably wouldn't be writing software. You'd probably be betting on the stock market. So one of the things that I've often seen people try to use to align or constrain company priorities is OKRs, objectives and key results. I've almost always seen it used poorly. And I don't think that we should go too far into detail there, too far in depth on this. We could probably do a whole podcast about the pitfalls of bad OKR implementations and what you should actually try to use them for. Not even just an episode, but a whole podcast, a series of episodes. I'm sure there could be a whole series. What do they call it on Netflix? A limited series on just OKR pitfalls and what you should be doing instead. But there actually is a lot of value in a good, well-implemented OKR strategy, right? If you were to go back to the original source material where it says have one or at most two objectives for your whole company, unless your whole company is a giant, giant megacorp that has a lot of divisions, then maybe you can do one or two per division. If you say, hey, our company's objective right now as a whole company, we are going to increase conversion rate, reduce churn, sell into this new market, and everybody is focused on the same objective. that can be very, very, very powerful because you create a ton of alignment across the whole company and you can align all of the technology and marketing and sales and finance and everybody getting everybody going in the same direction. It's just typically not implemented that way, which is unfortunate. The other thing that I've been thinking about is just today in one of the the Slack teams that I'm part of, there was a post about how we should be mostly using boring old technology, a boring old architecture until it's no longer sufficient. Where typically as developers, we want to reach for the new shiny things and the new shiny ways of doing things, right? My company has gone through a process over the last couple of years where we were a monolith that was not meeting customer needs. And then we tried to scale up and move into kind of a microservices architecture. And then we had some other financial constraints and now we're a smaller team. And we have retreated to the monolith architecture as something that we can well support within our current company structure. We could probably have multiple hours of conversation about the times and places where monolith versus microservices are more appropriate and which, how to choose which and why. But, you know, we went through a phase of growth and then we went through a phase of contraction. And with the smaller team that we have now, having a monolith allows us to actually deliver more features more quickly and meet other business needs, needs given that constraint. And there's just so many things about the company direction and priorities that become super important. And I will say that one of our more sarcastic or snarky friends immediately posted after that, but what about my resume? If we're using boring old architecture, what about my resume? And I would say, I, not just as a technical leader, but also also as an engineer on a team, have always discouraged resume-driven development. You more often than not create problems that you didn't need to have and implement solutions that solve problems no one actually had. And more often than not, it's just a bad choice to focus on adding new three-letter acronyms to your resume rather than solving a business problem.

[00:47:42] Allan Stewart: Well, that does also highlight the next constraint we were going to talk about is people. And that could be the number of people that you have available. That could be just like the personalities, the skills of people that are on the team. And so like you mentioned, microservices, organizationally, do you have the skill set in the people to be able to pull off

[00:48:10] Dave Adsit: off microservices. Well, and do you have the need? If you have a team of 12 people, you don't really need the autonomy that is granted by a microservices architecture, a well-implemented microservices architecture, not talking about a distributed monolith, which is typically what people end up with. But if you had true microservices, do you need that for 12 engineers? Or do you have 300 engineers and you actually truly need to have 30 different teams operating independently across your system without having to coordinate releases and coordinate dependency updates, et cetera, et cetera. Yeah. Yeah.

[00:48:50] Allan Stewart: The presence and the skill of people makes a big difference. If a key player is on vacation, that might constrain what you're able to do during that time period. If you, if you need a skillset that, that you lack, hack you want to want to create a new mobile app or you want to adopt some kind of machine learning AI thing but you don't have that skill in the people then you're going to have to do something about it right are you going to train train up that skill are you going to have somebody come and you know and teach you are you going to hire an expert contract it out there's a lot of options but you're constrained around what your people are able to do or willing to do even.

[00:49:36] Dave Adsit: Well, and I think about it from the terms of a lot of people don't know what engineers do. They don't know the difference between an operations engineer versus a programmer versus a web developer. A lot of people don't know. And they certainly don't know how to tell the difference between a junior, a mid, a senior, a staff, et cetera. And I think about it from the perspective, if we were to step out of our own industry and we think, if somebody gave me a budget of $600,000 a year to hire medical professionals, okay, well, do I need two heart surgeons or seven nurses? I don't know. I don't know what they're going to do. I know that two heart surgeons and seven nurses are not interchangeable. They're not such a thing is a fungible medical professional. I can't just go down and hire a group of them and then expect whatever medical problem I put in front of them is gonna work out well. Having recently had surgery, I think, man, I'm really glad that I had a skilled surgeon and he had a skilled surgical team that included several other professions, especially anesthesia. That guy is the best guy to have. I don't wanna have surgery on my sinuses without anesthesia. so i wouldn't have hired two surgeons and no anesthesiologist but you have to know what the problem is and sometimes the people you have available constrain what problem you can solve right when uh years ago i worked for a company that had an outsource team in southeast asia somewhere and we we kept having people churn off of our project and and so we were always getting new people. And what we found out after digging in a little bit and asking some questions is in that market, software developers know exactly what their hourly rate is based on number of years of experience. And the pay band that we were offering allowed us to hire people who had had something like three to three and a half years of experience. And when somebody hit three years and six months of experience, they could go down the street and get a raise. And we didn't understand that in our company in the US because we didn't realize developers in that place were paid quite so little and with quite so tight constraints. But as soon as somebody got three years and six months of experience, they would start applying for jobs. And as soon as they got a job, they would leave because it was always a raise because now they weren't a developer with three years of experience. They were a developer with three and a half years of experience. And we adjusted some of our HR practices around that so that we could keep people for 18 to 24 months instead of six. And it allowed us to have more continuity on the team and deliver a higher quality product. I like that.

[00:52:35] Allan Stewart: So yeah, people are kind of crazy. They're not interchangeable. anytime you change like you don't even you don't even have to hire and fire but just reorg your company and you'll find out how disruptive it is to change people around

[00:52:53] Dave Adsit: right the uh the the concept of your organization i've i've recently had multiple changes to my engineering org and it is very disruptive every time it takes time i had one of my principal Principal engineers say every team is immutable. When you add, remove, or change a member, you've created a new team. And you now have to go through the process of forming, norming, storming, performing. Right. And while that concept of team formation is actually far too rigid to match the reality of what happens, the idea that your team, any change you make to your team is going to change everything about your team is actually somewhat true. Yeah. I think most of us are familiar with Conway's law, which is that organizations which design systems are constrained to produce designs which are copies of the communication structures of those organizations. I think we've all heard that. We've probably lived it. We've seen that microservices is a technical solution to a people problem, which that people problem is that it's very hard for everyone to communicate with everyone else at the same level all the time. As you add more people, the number of communication channels grows exponentially and microservices allows us to reduce it within a scope and create constrained communication with other groups. so that we can have more effective communication within our group. Yeah, I think it's pretty undersold, generally,

[00:54:29] Allan Stewart: how much this gets encoded into the software, into a business. You'd like to think that how you organize the people is separate, and that doesn't matter because, you know, these are how we were going to report up to the business, and it's not going to affect the software, but it does. It literally gets encoded in there in various interesting ways as you're looking at the org chart and how the system is structured, how you just tackle the problems, right? Right. So there's the famous Eric Raymond statement that if you have four groups working on a compiler, you get a four pass compiler. Right. Those are those are the kinds of things that happen as you move those boxes and arrows around. And I think there's also some interesting things that happen just kind of almost politically within the teams, what teams are willing to do, or whether they're able to make forward progress can sometimes really depend on how those things are structured. I've worked on some teams where different members of the team have different bosses because they're aligned around their functions rather than having a single set of directions for the team. And it can be really confusing. You don't make a lot of progress, but you can make a lot of messes. Right.

[00:55:57] Dave Adsit: I've been in similar situations in the past, and it can create a lot of conflict within a team. And the team is the unit where you want the least conflict. You want people to, of course, have healthy conflict, but you want people to be aligned around purpose and direction at the team level. And if you've got some members that have one purpose and one direction and one set of goals and other members that have a different set of goals, you're going to run into problems inevitably. And one of the things I think about is that as your team grows, when you have one team, you might only need one engineering manager. And that person might also be the tech lead and the architect and possibly, but not ideally, the product manager for the team as well, right? They are making all the decisions and it's really easy because they don't have to do that much communication outside the team. They just figure out what needs to be built and they guide the team to build it. And then if you have multiple teams, you'll often find that the team itself is able to execute, but when things cross team boundaries, they might fall into the gaps between and nobody ever works on them and they get lost, right? And so you end up having additional overhead that you might think of as unfortunate or unnecessary, but it actually is the, it's the grease or the facilitation that keeps things moving through the system, right? You have to to create additional responsibilities, which means hiring additional people. And so if you have a system with 10 development teams and no engineering managers, what does that look like? If you have a perfectly flat system, it probably looks like not very many people get very much attention from their leader, their manager. And it probably looks like most of the time they're operating autonomously independently. And the leader doesn't really know that much about what's going on. So you end up needing those additional layers of organizational structure to both facilitate broader cross-team projects, but also to deliver the hands-on needs of the team members, right? Engineers are always asking, how do I get to the next level? I would like to level up my career. I'd like to make progress in my career. I'm a junior developer. I would like to drop junior and be a developer. I'm a staff developer and I'd like to be principal. I'd like to be a tech lead. I want to be an architect. I want to be an engineering manager. How do I do that? And if you don't have those people to help facilitate that work or that understanding or that growth, then it just gets stagnated. And then you can actually lose really good people because they don't have a growth path.

[00:58:31] Allan Stewart: Yeah. And the context, the context of the business will matter there too, right? If you, if you're running a really flat organization like that, it might not work in some contexts, but if you're doing a lot of ideation and the, the goal is lots of experiments and, and interesting tech that you're going to then go and utilize somewhere else or sell off or, or something like that, then that might be a fantastic way to structure your system. depending on the constraints of the people, the kinds of people that you could afford and that you hired. If you have a lot of really autonomous, smart people, that might work out. And it's just, it's going to depend on the context of the rest of the system, whether that works out or not. The last constraint that I had thought about is kind of related to these other ones around people and your org chart and your company priorities. But it's also just kind of the culture culture of the company in general, the combination of people that you have and the philosophies that they, or, you know, ideals that they espouse, how things actually get done in your company matters, right? There are constraints around what beliefs or traditions or policies are in place within the company. So for example, I remember a company that I was working for many years ago had a change advisory board. And because of the cadence of the change advisory board, that affected what we did and what we could work on. And it affected things like how often we could release to customers. It affected the batch sizes of what we were doing. And sometimes that was often very negative and I wouldn't recommend it as a good practice for anybody. But those kinds kinds of concepts, the way people want to make decisions constrains what you're able to do

[01:00:32] Dave Adsit: in code. Yeah. I have a couple of stories about change advisory boards that I will share with you at a later time, or perhaps if we meet up in person. They're not particularly positive experiences, I would say. But there's a lot about different things about your company culture. Is your company risk averse? Are you willing to accept risk? I remember at one company I worked at, we had basically forklifted the move fast and break things motto. And we moved fast and we broke things occasionally. We moved pretty fast and we broke things once in a while. And eventually somebody came in, there was a change in leadership and the new leader did not like breaking things. And so he tried to change the motto to move fast, but don't break things. And I'm sure you can imagine what happened is that when we could no longer break things, we could no longer move fast because we had to introduce additional layers of validation and testing before anything was allowed to go out because we didn't dare break things anymore. more, right? So I've worked in several organizations that have had a QA department that comes around behind developers. The development works on sprint one and in sprint two, QA is validating everything before they hand it to operations for deployment in sprint three. And I've worked in organizations where we have eliminated the QA function entirely and put responsibility for quality on the the developers who write the code. And in my experience, you get better code when you do that. If somebody knows, if I release bad code, I'm going to get a paged in the middle of the night when it breaks, and I'm going to have to get up and fix it or resolve the issues related to it. And then the next day I'm going to have a fire drill where I try to fix all the problems and write a patch and release it and all of that. You get better code because nobody wants to do any of that. They just want to work on new features. And so you're going to take a little bit more time to put in place a few more automated tests and make sure that you understood the problem well before you release the solution. So there's a lot of things about the culture of the company. If we have a good, if we've worked hard to create psychological safety and we do our blameless incident reviews when there's a problem and we truly don't try to lay blame at the feet of any one individual, we're going to have a different type of software development culture than if we are finger pointing people all the time as soon as anything goes wrong. Well, I know who touched it last and it was this person and we should string them up by their toenails. I don't know. I've honestly seen people get fired over it in the past and release a bad feature too many times in a row and you lose your job. And whether that was justified or not, it's hard to say, but it does speak to the culture of the company, right? That's one of the quips or adages about culture or is it your company culture is who you hire, who you promote and who you fire?

[01:03:44] Allan Stewart: Yeah, if you have a culture of meetings that affects whether you're going to be able to code or whether the developers pay attention in the meetings because they're actually coding instead. Trying to code instead. Thanks, Zoom. If you have a culture around experimentation that could change how you code, you do a lot more A-B testing, perhaps. If you have a culture that allows allows a lot of flexibility in what kind of programming languages you use that'll have a different a different outcome in how people are actually writing code than if you have a culture of standardization and say no we only we only support these three languages or databases or

[01:04:33] Dave Adsit: whatever. Yeah. You can write in any language you want, as long as it's Java. There's, there's well, and, and these are probably not the kinds of things that make it to the our five culture values are chart on the wall, but they're things that you figure out as you work in a company over time. Right. Yeah.

[01:04:54] Allan Stewart: So to kind of summarize all of our discussion, I think some of the important points here are constraints aren't bad. You need to recognize what constraints exist in your system and then use those to make great engineering, right? Know how you are constrained, what limitations you have, and use that like your kite example to fly high and make things work better. And then also being aware of those constraints means that you're aware when it changes because your strategy is going to need to change as your context and your constraints change.

[01:05:36] Dave Adsit: Yeah, I was going to say, you have to be reactive when constraints change over time. We've talked about that multiple times, how different constraints apply at different points and how you have to be adaptive to that. And I mean, we've all worked in places where someone has ignored the constraints that exist, right? One example from my history is picking a programming language because it was kind of fashionable. And not just fashionable, but also it has some technical merit to it. Like, oh, this is the fastest programming language in the market right now. And we could say it was C++ or Go or Rust because all of them have the same problem, right? The problem is there aren't any developers who know those languages. And so you put your most important thing in your company into that tech stack. And now you have to hire people to build a team. And you might find a couple of people who have spent time learning those languages and are okay at them or even pretty good. but what do you do if you need a bigger team than that? Now you've got to hire people and train them and that makes everything slower. And do you want your flagship product, even if it's written in the best tech stack, do you want it to be written by novices to that tech stack? There's a story from the small talk era of one of the big small talk projects that they, at the beginning, they estimated they needed about a dozen small talk experts to pull this thing off. And the management and the HR came back and said, what if we gave you a hundred novices instead? And they said, we would rather have the 12 experts. And they're like, I don't think you understand what we're saying. We have a hundred novices and zero experts, right? And so over the course of the project, of course, the project was over budget and over time and all of the things that happen when you don't have the right people to do it. But at the end, when they did their post-mortem and they said, hey, what did we learn? They said, we probably could have pulled this off within the time constraints and the budget constraints if we'd had a dozen experts from the beginning. Now we've got a few dozen experts because we all worked on this project for the last couple of years, but they didn't have them available at the time. And I've worked on on these projects where someone made a technical choice without considering the greater constraints in the market. We picked Go instead of C Sharp because Go is the hot new sexy and C Sharp was originally built by Microsoft and must therefore be suspect at all turns. It turns out there's hundreds of thousands of C Sharp developers on the market and we could have filled a team almost immediately with very high skilled and very qualified developers. But instead, we're struggling to find the handful of Go developers who aren't already happy working at one of the few big Go shops. In fact, in one example that I'm thinking of, we accidentally hired about eight to 10 expert level C sharp developers and then taught them Go so that they They could work on the Go project. And the question then becomes, are we fighting against sunk costs? Should we just start over and rewrite it in a language and a toolkit that we all know really well or not? I don't know. But there are definitely costs when you ignore your constraints. And so I would encourage everyone to understand the constraints that you're operating in and don't necessarily fight against them. Embrace them and leverage them to fly to higher heights.