Crafting Code Podcast

[00:00:16] Allan Stewart: Welcome to the Crafting Code Podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately I've been thinking about older video games and whether they continue to maintain their online servers.

[00:00:33] Dave Adsit: I'm Dave Adsit, a VP of engineering, and recently I've been thinking about the value of joining or building a community of professionals.

[00:00:41] Allan Stewart: Our topic today is constraints. I think it's pretty common to think of constraints as a bad thing. You're being constrained, help, help, I'm being constrained or repressed or something. But being completely free of constraint can actually be more difficult. It's like paralyzing because you've got too many options, too many choices that you have to choose from. Too many constraints is also probably not going to work out because you don't have enough flexibility. But a manageable number of constraints makes things better.

[00:01:16] Dave Adsit: Yeah, I've thought a lot about constraints over the years. One of the things that I've thought about is... It's probably a common story. The story of the kite, right? You're flying a kite with your kids or as a kid, probably, and you've got a string holding it down to the ground, tethering it to the ground, and you see the kite way up there in the air and the wind whipping and it's struggling and you can tell it wants to go higher and higher and higher. And if you think, I'm just going to cut the string and then it can fly as high as it wants, you'll notice that your kite sort of just falls out of the sky because that tether, that constraint... That constraint is actually what enabled it to control the flow of air and stay aloft anyway. So we like to think about constraints as though they were sort of a negative thing, but there's actually a give and take between what constraints you have and what you are enabled to do because of them.

[00:02:14] Allan Stewart: Absolutely. I've heard it said that constraints make great engineering. I couldn't find an exact quote that sounded like that, but I think it sounds pretty... And so I'm going to keep saying it. Yeah.

[00:02:26] Dave Adsit: I mean, I've heard similar things over the years, right? One of the things that I've heard is engineering is doing with $1 what any idiot can do with $2. And we had a previous training with Mary and Tom Poppendick when they came in and taught us at one of the companies I worked in. And one of the things that Mary said is that when you apply a date and say, I'm going to ship on this date, whatever's ready, that is what enables real engineering to happen because now you're making trade-offs. Now you're picking the most important things that can get done by that date. And so we like to think of constraints as holding us back from doing our best work possibly. But the reality is that sometimes the constraint is the thing that enables us to do good work and actually get something across the finish line.

[00:03:21] Allan Stewart: Yeah. Yeah. That's interesting. Yeah. Yeah. We're going to talk about some different kinds of constraints here, but it just strikes me that time is one that you have very little control over. It marches forward where we are all time travelers moving forward at a rate of one second per second. Yeah. And always in the same direction too. Yeah. Always traveling forward, never backwards and always twirling, twirling into the future.

[00:03:46] Dave Adsit: Must be one of the constraints of our universe or at least our perception of it.

[00:03:50] Allan Stewart: Yeah.

[00:03:51] Dave Adsit: Let's talk about some of the different types. Yeah. I think the first one and the most obvious one is technology. What technology can we use at this point with the problems we're trying to solve? So what are your thoughts on technology as a constraint?

[00:04:09] Allan Stewart: I think your existing tech stack is usually the first one that I think about. If you're Greenfield, you're starting a new project or there's a new company or there's a lot more options. I think in this case, I think in this case, I think in this case, I think in this case, I think in this case, I think in this case, I think in this case, I think in this case, time we're going to do it all right. We're going to do it better. And it's only going to take just as long as it took to build the first one in the first place. Right. Or longer maybe. Well, and,

[00:05:03] Dave Adsit: and even in a greenfield system, you are constrained by the technology that you know, or are willing to learn. Absolutely. There, there's a project out on GitHub that I look at every once in a while, the Ouroboros Quine, and it has 100 or 101 programming languages in it. I don't know most of them. I, and I've been around for a quarter of a century programming software professionally, and I've used a fair number of programming languages, including going through the book on seven languages in seven weeks that was popular a few years ago. And And a lot of the languages that exist, I don't know, and wouldn't dare try to pick up and use in

[00:05:53] Allan Stewart: production for anything that mattered. Yeah. I think about like what, what is out there in the world, you know, what, what can you get your hands on versus what are you going to have to build? Cause there's a certain amount of, you know, we use technology to build more things, new things, right? Nobody, nobody's interested in the software that has already been created, but what are you going to use? To build something new right now, now that you can get an AI from as a commodity from a cloud provider, what are you going to do with it? Yeah. So there's a lot of different technology

[00:06:30] Dave Adsit: that you could pick and there's a lot of things around the technology with that would constrain you, right? Say you want to use a specific database provider. You're like this, this database would be perfect for our solution. Okay, great. Are we cloud hosted? Is it available as a in our cloud provider. Do we have to self host it in that provider? Do we have to spin up servers and manage the cluster and manage the storage and do all of those things in order to use this database? Okay. Maybe that's worth it. Maybe it's not. Those are things that we need to consider when it comes to even a technical choice that if we're thinking, hey, it's a green field, I'm going to pick best of breed for every technology that we are going to use. And the only constraint that I have is that it has to be in a cloud provider. the best? Well, if I'm building a web application, I'm going to have to put JavaScript in the front end somehow, because that's what the browsers know how to talk. So am I going to do that through vanilla JavaScript? Am I going to do that through TypeScript? There's still a lot of choices. And what does best even mean in that case? It's very hard to even know. And most of us aren't even working in that space. We are not fully unconstrained in the technology that we can use. We have some existing system somewhere, whether it's a prototype or whether it's the legacy code that's been in existence for the last decade. We have technology that we have to work with and hopefully enhance, right? Or at the very least, leverage it to solve problems for our users. And so when... When technology is a constraint, I think it enables us to think more about, well, what is the problem we're trying to solve? And if we don't have technology as a constraint, I have seen teams just spin and spin and spin on picking the perfect tech before they even get started on solving the business problem. And you can frankly run out of budget, which is one of your other constraints

[00:08:34] Allan Stewart: without even making a technology choice. Yeah. I think sometimes it's a lot better to settle for... Yeah. ...something that's a little bit suboptimal while you're dealing with these other constraints and move forward. And... Yeah. ...you might find out... A lot of times you find out that the decisions you thought were important at one part of the project are not important later for various reasons, either because it's like, oh, this tech actually is just working fine. It's okay. Yeah. Or something else has changed about your situation that makes it kind of dangerous to just assume that you can... Yeah. ...pick the perfect thing and never have to change it.

[00:09:15] Dave Adsit: Well, and I like to think... I realize that suboptimal is probably the perfect technical term, but I like to think we're making good enough technical choices for now. And I think that for now is important because at this point in time, given the other collection of constraints we have, this is a good enough choice for the tech we're going to use. And we can change it later if we need to. I think that if you and I were to get together on a new project... Yeah. ...to start up today, we would probably pick a certain tech stack that I can imagine in my head. It might not be the same tech stack that other people we know would choose, but I think I know which backend language we would pick. I think I know what cloud provider we would pick. I think I know what front end toolkit we would pick. And those are all perfectly adequate choices for any web-based software development problem today. And then if that project got really big and we said, hey, we've got these other features we need to build, there's other stuff that we need to do, we might introduce additional technology at that point to solve those problems.

[00:10:24] Allan Stewart: Right. So another constraint I think about is memory and time, basically these computational related things. And you could include things like bandwidth in there, right? So mobile really brought bandwidth. Yeah. To the front. Some years ago, iPhones were coming out and Android was coming out. There are things that you can do, but they can be computationally expensive or difficult. And in some ways, I think Moore's law has spoiled us because there's like a whole generation, maybe a couple of generations of engineers who don't think about some of these performance characteristics very often.

[00:11:09] Dave Adsit: Yeah.

[00:11:09] Allan Stewart: Yeah. Yeah. Yeah. And computers just get faster. It's like, oh, well, just buy a new computer, buy some more RAM, buy a GPU that was crafted for Bitcoin mining or something like that, and you move on your way. But those constraints are still there. It's just that you are kind of working

[00:11:34] Dave Adsit: around them. Working around them, solving them in a different way. Right. I mean, one of the constraints Everything else kind of had to live within that 640 kilobytes. Today, I was talking with one of my engineers about a new program we're test driving within our company. And he was saying, hey, when I run this program and I have all of my Docker containers running and my three IDEs open at the same time, I'm running out of RAM and everything is getting really slow. And I'm like, well, what are the specs on your computer? He's like, well, it's an M1 Max with 64 gigabytes of RAM. And my response was just like, that's kind of it. That's, I mean, that's not the top of the line computer in our department, but that's not too far down the list. You know, you've got a pretty cutting edge computer. We're going to have to potentially give up something if you want all of these applications to run at the same time. Maybe you're going to have to cut back on the RAM and CPU that are allocated. Or turn off some of the features of this software project or software tool that you're using, or I don't know, maybe only run two full IDEs at a time instead of three. And all of these are memory and computation related constraints to what he was trying to accomplish with his laptop. Keep in mind, I mean, I know that this doesn't mean anything to a young, younger generation of people who grew up with amazing laptops, but, you know, the 640K of RAM that I had in the laptop or in my desktop computer was not even available for purchase in a portable computer, which, you know, is a precursor to a laptop.

[00:13:55] Allan Stewart: Yeah. Yeah. I was reading an article about how in, I think it was 96, Doom comes out. And now today you can play Doom on all kinds of computers. Like people put it in like tweets and like run it on Arduinos or, you know, stuff like that, which is kind of amazing. Right. But I try to load up a website. And it's just as slow as it was 10 years ago. Yeah. It's slow. It's bloated. It's kind of a sad state of affairs sometimes.

[00:14:33] Dave Adsit: Yeah. Yeah. We all thought that the web was going to get really fast when we, when we went off of dial up and onto cable modems, but it turns out that we just started downloading more crap. Every, every website has to download a whole bunch of front end frameworks in order to operate. And so if you don't already have those cached properly, you're going to have just as slow of a web experience 15 years later. Yeah.

[00:14:56] Allan Stewart: So I think about it in terms of when we're working on coding projects, a common place where this kind of performance consideration can come up is around databases. Mm-hmm. So databases are often a bottleneck in a system. And so then you have to start making some of these trade-offs of computation, right? Um, things like bandwidth and memory, uh, the IOPS that you're running, that they're using up in your cloud provider. Those are all things that you might have to, uh, work around because they're constraints that you have. So that might mean in some cases, the right answer is to denormalize your data because. Yeah. Adding one more join to the, to that table, uh, to that query, one more table joining in there is just going to wreck the system. Yeah. Um, those are just constraints that are kind of built into computation that although we can sometimes ignore them because we just get better and better machines, ultimately it's really fast. Like you take something like a database and you start doing the Cartesian product of multiple tables and the data. Just goes exponential. And you, you can't, you can't do that forever.

[00:16:12] Dave Adsit: Well, and one of the things that I like to think about is that I like to remind people that the normal forms were actually a response to a constraint, which is small disks, small, very expensive disks. You know, you can get a home server to be your, your media server or whatever, and throw it in a closet and forget about it and have 20 terabytes of. Raid five disk available for movies and music and whatever. But back in the day, nobody had a disc that big, all of the solid state storage in the world may not have been 20 terabytes. If you go back far enough, the, the idea that your company would have virtually unlimited functionally free. Solid state. Storage or not even solid state, like magnetic storage for the database server. It was a ridiculous concept. And so you had to, you had to be conservative about every bite that you used. You had to, you had to conserve them all use as few as possible. And so normalization was an attempt to trade off compute time for storage costs. Right. You were, you were pushing down storage costs by normalizing data and it comes at the cost of more compute as you do a join and bring data together for multiple tables, you know, joins and filters cost more when you're. Or joins costs more than flat table structures. Uh, but now, and so we've trained a whole generation or several generations of developers that you should always normalize everything. And so. People believe it without thinking about what trade offs you're making. Am I trading? What is my constraint in this case? Is it memory allocation? Is it storage on disc? Is it compute time? And is it. Which of those applies more when I'm writing versus what I'm reading? Um, I've re uh, as recently as six months ago, had a discussion that went round and round and round in our architecture guild about whether we should store multiple rows. In a data table. And. And then the updates would be more expensive because the updates would have to update. Anywhere between one and a hundred rows of data, because the, the where clause wouldn't be where ID equals they'd be where category name equals. Right. And then, so you have to update multiple rows and do that as a set. Is that better or worse than normalizing that table? And this table would have ended up being three tables, right? Cause it was a many to many concept. Sure. And then adding that complexity of compute every time we read and every time we update and all of these things. And we, it was a very heated debate because we had some very dogmatic approaches to what. What good software is supposed to look like when the reality is what good software is supposed to look like depends on the constraints that you have. And in this case, our constraint was we needed very performant reads and the rights happened very infrequently. And so it wouldn't really matter. But, oh, also we host our database in the cloud and we have virtually unlimited storage space for the database. And so adding a few more rows versus normalizing this table really wasn't going to cost us anything. So, yeah.

[00:19:56] Allan Stewart: What makes me think like, which is better? And the answer is yes. They're all better for, for some, for some reason, instead of constraints, right. For some set of constraints. One of these is better. But which one depends on a more holistic, holistic view.

[00:20:13] Dave Adsit: Well, I mean, you think about the, this, the, the very first spacecraft that we sent into. Low earth, low earth orbit, or to the moon had less compute power than your smartwatch. Yeah. You can guarantee that they were very cautious about how they used every bite of storage. Meanwhile, I use my smartwatch to set digital, you know, set timers and, and. I can have an analog watch that ran on a spring and get virtually the same functionality, but instead I have a supercomputer. So constraints change over time. Yep. And you don't even have to defrag the disc on it. I don't. So speaking of time, one of the other constraints that we have always is time. We hinted at it earlier, but you think about the time, the wall time, the calendar time, right? How long can. We spend building this feature? I like to think about this in terms of bets. For me, it's, it's not, when would we like to ship it? We very rarely have a hard fixed date. Sometimes you do. Sometimes you have to launch a feature by a certain date for a trade show, or you've already bought an ad space and you have to do something right. A lot of times we don't have deadlines. We have kind of sort of sad lines. Like no one is going to die. Someone might get a little bit sad. For most of the software that most of us write. And I think it's important to think of time and date. Yeah. Date deadlines in that way. But I also like to think of them in terms of what is the bet we're willing to make? Because the time that an engineering team spends on something is literally money to the company. And I like to, when people ask me, well, how much money is it? What, what heuristic should we use? I always tell people your team of six to eight is about $20,000 a week, fully burdened. Yeah. And it might actually be more than that. And you might have a smaller team than that, but if you just say it's about $20,000 a week that we're spending on this, how big of a bet are you willing to make in terms of money before you need to know if you're on the right track or not? I like that.

[00:22:23] Allan Stewart: And I like what you were saying before. I think that time constraints are often used, uh, poorly. Yeah. Especially when time and scope are both constrained and, and that happens so often. Um, what's the thing we call the, the iron triangle time scope and quality quality. Yeah. And so that, that happens so often that there's kind of been a visceral reaction from developers. You know, a lot of them don't want to work in a deadline, but I liked what you were saying before that, you know, real engineering starts to happen when you're on the clock. Right. Um, it makes me think about the Martian, right? It's like, oh, let's, let's do some, let's science the heck out of this.

[00:23:05] Dave Adsit: Yeah. Problem. Yeah. Well, and we've had real experiences like that in the U S space program, right? No. Yeah. We've sent people up into space and then had things go wrong. And then we've said, okay, we have this many hours to solve this problem before they run out of oxygen or they actually suffocate on CO2.

[00:23:25] Allan Stewart: Yeah.

[00:23:26] Dave Adsit: Let's fix it.

[00:23:27] Allan Stewart: And I think if we, if we use time, well, as a constraint, I think you can get some of those benefits. Yeah. But you're trading off with something else. And, and I think that that's always important as we examine any of these types of constraints is that they're, they're all trade-offs and often against each other. So if you're going to trade off in time, then quality or, um, the other one, all right. It's scope that they might have to change, right? You can't necessarily get everything that you want. You can't, you can't say, well, we want to get the Apollo 13 astronauts. Home and, and, and they're alive, but also we're going to extend their mission by a week or also they need to bring back an extra, you know, 500 tons of moon rocks or something like you, you don't get to choose all those things.

[00:24:21] Dave Adsit: No, you have to give something up almost always. And I will say that, you know, I was just talking about how we've trained people to believe that normal, higher normal forms are always better without consideration for the trade-offs that we're making there. Well, we've, I, I have been trained. I was trained either through directly or indirectly or by experience. And I think a lot of other people were too, that you should always work towards scope, build the right scope, ship the right thing, scope, scope, scope. And so it actually took me a long time and a lot of thinking and wrestling with that concept that having a date and letting the scope vary can actually be, it can actually be a much more powerful way to build a software product. Focusing on the high priority things and getting them in front of customers sooner pays tons and tons of dividends in ways that you wouldn't expect until you actually try it. And I wrestled with that for literally years before fully embracing that mindset shift from we're going to build the scope. And then when we hit the scope, we're going to ship it and we're not going to let the scope creep. We're not going to whatever. If you build with a date, if you say, we're going to ship this software in two weeks, what's the most valuable? What's the most valuable thing we could get done in two weeks? Well, you might be a weekend and you discover that what you've been building isn't the most valuable. You actually could build something else instead. Perfectly fine. Do it. If you found something more valuable to build, you should shift focus and work on that because you still have that time. You still have that week. And, you know, honestly talking with Tom and Mary, when they came in, they were talking about, you know, improving. When Tom and Mary came in, they were talking to us about. Actually, six to eight weeks, right? They're talking about longer frames of delivery that we wouldn't even consider fast delivery anymore. But they were coming from a different context where people were shipping once a year, maybe twice a year, every six months. Maybe if you're super aggressive, you're shipping once a quarter. And they're talking about cutting that in half again and delivering software two or even three times at a quarter. I mean, my team's now. Most of. My team's deliver more than once a week. And that feels like a good cadence. We haven't quite pushed it all the way down to start a feature in the morning, ship it in the afternoon, go home for the night. But, you know, maybe we'll get there. And maybe that's too small of a batch. I don't know. Yeah.

[00:26:52] Allan Stewart: Yeah. There's an interesting question there about how, how often the customer even wants that level of change. I think you can get too fast. Right. It also makes me think about scope. There. There's been some projects where I feel like we've had a really good understanding of what the scope is. Like there's, there's a particular problem that we're trying to solve. We understand it really well. There are aspects of scope that we, it's just like, we just know. And so we can break it down along lines of scope and ship out incremental value. But I agree, agree with you as far as it's taken me some time to realize that sometimes we don't know what we're supposed to be. And so using time instead of scope can actually be better because the scopes are kind of like the features themselves. Like what do we even want to build? We don't fully understand. And so we don't know how long it's going to take. And it's going to take a really long time. Well, what can we do in a week or two weeks or six weeks? Those can be better, better ways to constrain ourselves so that we learn. And then when we learn more, we might switch over and start thinking about scope again, because now we have scope. And then you just trade off between them, depending on which one actually matters to you at a given time.

[00:28:13] Dave Adsit: Well, and I will say I am definitely an advocate of low cost discovery techniques, getting things like prototypes, even paper prototypes in front of customers early, having interviews with customers to discover what their needs are. But I will say in almost every project I've ever worked on. It's not until you put the first version. Of the product in front of somebody that you really find out what they needed. And it might not even be the second version that they needed. They might act. You might be a few versions in before you really understand the problem and the solution that needs to be built. Because when people come and ask us for solutions and software, they very rarely understand what is possible. And when they see the solution that you built based on what they told you, it's almost never what they really wanted. So. We need to plan for a few iterations. And that means going fast, at least at first, going fast enough to get the kind of feedback that we need so that we can build the right thing. And so rather than run away from the calendar, and I know we've all been beat up by here is your fixed date. Oh, and by the way, here's your fixed scope. And by the way, it's going to double over the course of the next few weeks before we hit that date. Because. We don't know what we want. And of course it's going to grow. You know, rather than running away from those scars, if we can lean into, we are going to take a date and that's going to be our delivery cadence. We're going to deliver every two weeks or every four weeks, or I mean, once a week, I don't know, whatever the delivery cadence is, you're gonna say, we're going to spend this much time in the, we're going to ship something. Whatever we have at that time is what we're going to ship. So let's make sure we build the best, most valuable, most important thing that we can in that time box. And then. We put it out there. If we can actually get ourselves to that point. I mean, that doesn't mean that you have to give whatever you have at the end of a week to every customer in general release. You can definitely. Keep it behind a feature flag, give it to alpha customers. Test it internally. Give it to a handful of friendly beta. Customers. Excuse me, give it to a handful of friendly beta customers and get some real feedback on working software so that when you have that bigger batch that. Is ready for. All of your customers. You can be sure that it's something that's actually going to work and be valuable to them. I think we've. We've gotten, we've developed techniques as an industry like feature flags. That allow us. To. Bundle up. Our daily releases into. Something that we can give to customers on a less rapid cadence. Because you're right. Some customers get burned out with the continuous change and some customers. Thrive in it. Some customers can't believe they get the opportunity to actually steer your product development. By seeing the daily release that comes out. Depending on the software, I'll sign up for the alpha channel or I'll sign up for the. Slowest possible release channel. Because it's software. I don't want to see updates. I just need it to work the same every day for the next year. Yeah.

[00:31:28] Allan Stewart: Well, if time is money. Budget is definitely a constraint. So there's the budget that turns into time as far as. You know, how long can we afford to pay our developers? But then I think there's also budget constraints in the form of, can you pay. Your way out of a problem. Right? Like if you, if you can just turn on the auto scaling for that cloud service. Yeah. You're going to, you're going to be paying more, but, or you're going to, you know, vertically scale in some cases. And we don't think about that quite as often or like, you know, we're not going to be paying.

[00:32:30] Dave Adsit: right? The question of what can we afford goes back to that whole idea of should we build it or buy it? Sometimes you build it because it's not available on the market, but a lot of times we build it because we have this not invented here syndrome and we should consider buying more often than we do if you have the money. I've gotten financial advice in the past that doesn't necessarily apply to your business, but it applies to you as a person. They basically said, if you can afford it and you have a task, let's say the task is yard care, lawn maintenance, mowing, edging, fertilizing, weeding, all of that stuff. If you enjoy it, you should do it. But if you don't enjoy it and you can afford to pay somebody to do it, you should pay somebody to do it and do something that's higher value to you as a person. Higher value, might be I'm actually working on the business and I'm actually literally making more money or higher value might mean I am spending time with my family and creating bonds and memories with my family. And so we need to be thinking about budget in a way that is, what are the trade-offs that I'm making by spending money in this way? as a service from a cloud provider? Probably not. I should probably spend that money more judiciously. And so getting ourselves out of that non-invented here syndrome and leveraging the tools that are at our disposal is usually a good value for our money. I do think it's funny that a few minutes ago, we were talking about memory and time, compute time as constraints. And here we're talking about the money is the constraint and we're just going to buy memory and compute and network and disk and all of that, right? And it's fair. Those are the kinds of trade-offs that as engineers, we have to be thinking about all the time. What do we have more of? Do we have more money or do we have more time to build something?

[00:35:16] Allan Stewart: And I think you get bloat in the same ways, right? So in the same way that we just leaned on Moore's law and downloaded a gigabyte of JavaScript into the internet, into the browser so that you can run your front-end framework, well, we do the same thing with-

[00:35:32] Dave Adsit: We can validate a form. So we can validate a form.

[00:35:36] Allan Stewart: Well, sometimes we do the same thing with like, well, we bought a bunch of services. We bought a bunch of online tools. And so you have to watch out with these different constraints because if you aren't constrained by money, you can use it up really quick, right? You're not constrained by the number of operations that you're... Yeah. Your CPU can do per nanosecond or you're not constrained by how long it takes for the disk head to seek to the relocation on your spinning rust. Well, as soon as you're not constrained, you might go overboard. And so you have to watch out for those areas. And sometimes it's the right decision, but you need to go back and revisit those decisions periodically to see are they still right? Or now that we're... Now that we're not a young scrappy startup and we have some money, maybe we should spend some of that money as an investment to pay down some debts. They might be technical debts. They might be literally we're spending hundreds of dollars. Recently, I had an experience where we were spending hundreds of dollars on Google Map API calls because we had just done maps the same way in every... situation on the website. And in one situation, we needed to make an expensive call. And in other situations, we did not. And we were paying the same amount regardless. So, you know, at some point, you might save yourself hundreds of dollars a month by going back and looking at those things and say, oh, well, we can make a little change now that we've learned, now that we've grown, that wouldn't have made sense to do in the beginning.

[00:37:24] Dave Adsit: You can change... You could spend more engineering time to optimize, multiple different calls, right? When I think about constraints changing over time too, we talk about budget and we talk about the... One of the things that I used to spend a lot of time on when I was young and had no money was organizing my music collection. I would take all my CDs and carefully rip them to MP3s of the highest quality that fit on my player. And then I download all the ID3 tags and all of that stuff so that I could have a high quality music on the game. and keep all of my CDs and their jewel cases pristine, right? And I spent... I can't even count the number of hours that I spent on this type of project. And at this point in my life, I would much rather just give Spotify $15 a month to give me unlimited access to most of the music I want. There were still some of the weird stuff that I listened to as a kid that's not available anymore. And every once in a while, I think, oh man, I wish I could listen to that one song by that one band that doesn't exist on the internet anymore. And I don't. I just listen to something else instead because the time it would take to manage all of that stuff is not worth the trade-off. It's not worth the cost. I would much rather spend the $20 a month or whatever to have somebody else handle all of the cataloging and identifying and ripping and managing and whatever needs to be done to have a good, good music. To have a good, on-the-go music experience so that I can plug my phone into my car and just listen to whatever I want. So the constraints you have today may not be the constraints you have tomorrow.

[00:39:08] Allan Stewart: And I think that shows up quite a bit as we transition over into company priorities and direction. We've started with a lot of the technical things. And as we move further and further in the socio-technical sphere, we find out that what the company wants to do today It's different than what they wanted to do yesterday. And those things can change very rapidly. Sometimes because you've got the CEO squirrel syndrome, that every shiny thing that they see, they want it. And that's kind of that immediacy. They want to take that. But other times it's just timing. The last customer you talk to has the most important problem. Well, yeah. Yeah, because they're the one

[00:39:55] Dave Adsit: that you're thinking about. Mm-hmm. Yeah. I mean, definitely. Definitely your company direction can change. Sometimes drastically. I think we've worked at companies that have had drastic changes in priority and direction and focus. We worked at one company where the customer we were trying to market to changed several times over the course of a couple of years. We were targeting the final user of our product. And then for a while, we were targeting the manager of that final user. We were like, oh, they'll buy it for their whole team, right? Yeah. We're like, actually, it's not even that department at all. The department that uses our product, they don't have the budget to buy it. This other department has the budget to buy our thing, and then they'll assign it to them. And then after a while, we realized, oh, that other department doesn't care about the same things as the person who would use the software, and they don't know how to evaluate it anyway. And so we went kind of around and around and around in circles until we finally ended up back at, hey, the person that uses our software is the one who's most motivated to buy it, right? So of course, we were building different features for each of those different groups because they each had different, they valued different things, and they had different constraints within their companies. But it can be hard to know what the right thing is and how to build it. But those are some of the constraints we have. It's like, hey, today we're working as a digital product, and tomorrow we're working as a physical product. And the day after that, we're working as, I don't know, some kind of a hybrid thing or cloud service type of a thing or whatever.

[00:41:39] Allan Stewart: Yeah. And I've often seen that the technical side of things, technical problems and business problems are not well aligned. So you might have a strategic goal at the company that gets upended by an incident or some, you know, or a potential incident that is coming up because you can't scale to meet demand or something has fallen over in the data center and you have to deal with that. And so there was a technical concern that preempted the business concern. But I've also seen it the other way where there's a part of the system I'd really like to fix because it's hard for me to sleep at night knowing that there's this bad code just sitting there being used every day. but it kind of works well enough. And so you move on because there's something that's more important to the business. You ignore a problem for a while as long as you responsibly can because you're getting value doing something else.

[00:42:42] Dave Adsit: You know, I have often heard developers complain that marketing forgets that Black Friday is coming every single year until it's almost Black Friday. Meanwhile, developers forget that Black Friday is coming every single year until it's almost Black Friday as well. Of course, that means that now we have two competing priorities. One of them is how do we scale the system to meet the 10X load that we anticipate? And the other one is how do we launch all the new features that we already promised in the marketing campaign that goes out before Black Friday? And so, yeah, the company priorities, I don't know. They're hard and they change often. And honestly, if they didn't change that would be a bad sign for a lot of businesses because our ability to predict what the market conditions are going to be like in two years are very weak. We do not have that skill. And if you had that skill, you probably wouldn't be writing software. You'd probably be betting on the stock market. So one of the things that I've often seen people try to use to align or constrain company priorities is OKRs, objectives and key results. I've almost always seen it used poorly. And I don't think that we should go too far into detail there, too far in depth on this. We could probably do a whole podcast about the pitfalls of bad OKR implementations and what you should actually try to use them for. Not even just an episode, but a whole podcast,

[00:44:10] Allan Stewart: a series of episodes.

[00:44:11] Dave Adsit: I'm sure there could be a whole series. What do they call it on Netflix? A limited series on just OKR pitfalls and what you should be doing instead. But they're actually, is a lot of value in a good, well-implemented OKR strategy, right? If you say, if you were to go back to the original source material where it says have one or at most two objectives for your whole company, unless your whole company is a giant, giant mega corp that has a lot of divisions, then maybe you can do one or two per division. If you say, hey, our company's objective right now as a whole company, we are going to increase, we're going to reduce conversion rate, reduce churn, sell into this new market and everybody is focused on the same objective. That can be very, very, very powerful because you create a ton of alignment across the whole company and you can align all of the technology and marketing and sales and finance and everybody getting, everybody going in the same direction. It's just typically not implemented that way, which is unfortunate. The other thing that I've been thinking about is just today in one of the Slack teams that I'm part of, there was a post about how we should be mostly using boring old technology, a boring old architecture until it's no longer sufficient, where typically as developers, we want to reach for the new shiny things and the new shiny ways of doing things, right? My company has gone through a process over the last couple of years where we were a monolith that was not meeting customer needs. And then we tried to scale up and move into kind of a microservices architecture. And then we had some other financial constraints and now we're a smaller team. And we have retreated to the monolith architecture as something that we can well support within our current company structure. We could probably have multiple hours of conversation about the times and places where monolith versus microservices are more appropriate and how to choose which and why. But we went through a phase of growth and then we went through a phase of contraction. And with the smaller team that we have now, having a monolith allows us to actually deliver more features more quickly and meet other business needs given that contraction. So, I mean, there's just so many things about the company direction and priorities that become super important. And I will say that one of our more sarcastic or snarky friends immediately posted after that, but what about my resume? If we're using boring old architecture, what about my resume? And I would say I, not just as a technical leader, but also as an engineer on a team, have always discouraged resume-driven development. You more often than not create problems that you didn't need to have and implement solutions that solve problems no one actually had. And more often than not, it's just a bad choice to focus on adding new three-letter acronyms to your resume rather than solving a business problem.

[00:47:42] Allan Stewart: Well, that does also highlight the next constraint we were going to talk about is people. And that could be the number of people that you have available. That could be just like the personalities, the skills of people that are on the team. And so, like you mentioned, microservices. Organizationally, do you have the skill set in the people to be able to pull off microservices?

[00:48:12] Dave Adsit: Well, and do you have the need? If you have a team of 12 people, you don't really need the autonomy that is granted by the team. a well-implemented microservices architecture. Not talking about a distributed monolith, which is typically what people end up with. But if you had true microservices, do you need that for 12 engineers? Or do you have 300 engineers and you actually truly need to have 30 different teams operating independently across your system without having to coordinate releases and coordinate dependency updates, etc., etc.?

[00:48:49] Allan Stewart: Yeah, the presence and the skill of people makes a big difference. If a key player is on vacation, that might constrain what you're able to do during that time period. If you need a skill set that you lack, you want to create a new mobile app or you want to adopt some kind of machine learning, AI thing, but you don't have that skill in the people, then you're going to have to do something about it. Right? Are you going to train up that skill? Are you going to have somebody come and teach you? Are you going to hire an expert, contract it out? There's a lot of options, but you're constrained around what your people are able to do or willing to do even.

[00:49:36] Dave Adsit: Well, and I think about it from the terms of a lot of people don't know what engineers do. They don't know the difference between an operations engineer versus a programmer versus a web developer. A lot of people don't know. And they certainly don't know how to tell the difference between a junior, a mid, a senior, a staff, etc. And I think about it from the perspective if we were to step out of our own industry and we think if somebody gave me a budget of $600,000 a year to hire medical professionals, okay, well, do I need two heart surgeons or seven nurses? I don't know. I don't know what they're going to do. I know that two heart surgeons and seven nurses are not interchangeable. There's not such a thing as a fungible medical professional. I can't just go down and hire a group of them and then expect whatever medical problem I put in front of them is going to work out well. Having recently had surgery, I think, man, I'm really glad that I had a skilled surgeon and he had a skilled surgical team that included several other professions, especially anesthesia. That guy is the best guy to ask. I don't want to have surgery on my sinuses without anesthesia. So I wouldn't have hired two surgeons and no anesthesiologist, but you have to know what the problem is. And sometimes the people you have available constrain what problem you can solve. Years ago, I worked for a company that had an outsource team in Southeast Asia somewhere. And we kept having people churn off of our project. And so we were always getting new people. And what we found out after digging in a little bit and asking some questions is in that market, software developers know exactly what their hourly rate is based on number of years of experience. And the pay band that we were offering allowed us to hire people who had something like three to three and a half years of experience. And when somebody hit three years and six months of experience, they could go down the street and get a raise. And we didn't understand that in our company in the US because we didn't realize developers in that place were paid quite so little and with quite so tight constraints. But as soon as somebody got three years and six months of experience, they would start applying for jobs. And as soon as they got a job, they would leave because it was always a raise because now they weren't a developer with three years of experience. They were a developer with three and a half years of experience. And we adjusted some of our HR practices around that so that we could keep people for 18 to 24 months instead of six. And it allowed us to have more continuity on the team and deliver a higher quality product. I like that.

[00:52:35] Allan Stewart: So yeah, people are kind of crazy. They're not interchangeable. Anytime you change, you don't even have to hire and fire, but just reorg your... your company and you'll find out how disruptive it is to change people around.

[00:52:53] Dave Adsit: Right. The concept of your organization, I've recently had multiple changes to my engineering org and it is very disruptive every time. It takes time. I had one of my principal engineers say, every team is immutable. When you add, remove or change a member, you've created a new team. And you now have to go through the process of forming, norming, storming, performing. Right. And while that concept of team formation is actually far too rigid to match the reality of what happens, the idea that your team, any change you make to your team is going to change everything about your team is actually somewhat true. Yeah. I think most of us are familiar with Conway's Law, which is that organizations, which design systems are constrained to produce designs, which are copies of the communication structures of those organizations. I think we've all heard that. We've probably lived it. We've seen that microservices is a technical pro... Sorry, microservices is a technical solution to a people problem, which that people problem is that it's very hard for everyone to communicate with everyone else at the same level all the time. As you add more people, the number of communication channels grows exponentially and microservices allows us to reduce it within a scope and create constrained communication with other groups so that we can have more effective communication within our group. I think it's pretty undersold generally

[00:54:29] Allan Stewart: how much this gets encoded into the software, into a business. You'd like to think that how you organize the people is separate and that doesn't matter because, these are how we were going to report up to the business and it's not going to affect the software, but it does. It literally gets encoded in there in various interesting ways as you're looking at the org chart and how the system is structured, how you just tackle the problems. There's the famous Eric Raymond statement that if you have four groups, working on a compiler, you get a four pass compiler. Those are the kinds of things that happen as you move those boxes and arrows around. I think there's also some interesting things that happen almost politically within the teams. What teams are willing to do or whether they're able to make forward progress can sometimes really depend on how those things are structured. I've worked on some teams where different members of the team have different bosses because they're aligned around their functions rather than having a single set of directions for the team. It can be really confusing. You don't make a lot of progress, but you can make a lot of messes.

[00:55:57] Dave Adsit: I've been in similar situations in the past and it can create a lot of conflict within a team. The team is the unit where you want the least conflict. You want people to, of course, have healthy conflict, but you want people to be aligned around purpose and direction at the team level. If you've got some members that have one purpose and one direction and one set of goals and other members that have a different set of goals, you're going to run into problems inevitably. One of the things I think about is that as your team grows, when you have one team, you might only need one engineering manager. That person might also be the tech lead and the architect and possibly, but not ideally, the product manager for the team as well. They are making all the decisions, and it's really easy because they don't have to do that much communication outside the team. They just figure out what needs to be built and they guide the team to build it. Then if you have multiple teams, you'll often find that the team itself is able to execute, but when things cross team boundaries, they might fall into the gaps between and nobody ever works on them and they get lost. You end up having additional overhead that you might think of as unfortunate or unnecessary, but it actually is the greatest thing that you can do. You have to create additional responsibilities, which means hiring additional people. If you have a system with 10 development teams and no engineering managers, what does that look like? If you have a perfectly flat system, it probably looks like not very many people get very much attention from their leader, their manager, and it probably looks like most of the time they're operating on a flat surface. You're operating autonomously independently and the leader doesn't really know that much about what's going on. You end up needing those additional layers of organizational structure to both facilitate broader cross-team projects, but also to deliver the hands-on needs of the team members. Engineers are always asking, how do I get to the next level? I would like to level up my career. I'd like to make progress in my career. I'm a junior developer. I would like to drop junior. I'd like to be a junior and be a developer. I'm a staff developer and I'd like to be principal. I'd like to be a tech lead. I want to be an architect. I want to be an engineering manager. How do I do that? And if you don't have those people to help facilitate that work or that understanding or that growth, then it just gets stagnated and then you can actually lose really good people because they don't have a growth path.

[00:58:32] Allan Stewart: And the context of the business will matter there too. If you're running a really flat organization like this, it might not work in some contexts, but if you're doing a lot of ideation and the goal is lots of experiments and interesting tech that you're going to then go and utilize somewhere else or sell off or something like that, then that might be a fantastic way to structure your system. Depending on the constraints of the people, the kinds of people that you could afford and that you hired, if you have a lot of really autonomous, smart people, that might work out. And it's going to depend on the context of the rest of the system, whether that works out or not. The last constraint that I had thought about is kind of related to these other ones around people and your org chart and your company priorities. But it's also just kind of the culture of the company in general. The combination of people that you have and the philosophies that they, or ideals that they espouse, how things actually get done in your company matters, right? There are constraints around what beliefs or traditions or policies are in place within the company. So for example, I remember a company that I was working for many years ago had a change advisory board. And because of the cadence of the change advisory board that affected what we did and what we could work on, and it affected things like how often we could release to customers. It affected the batch sizes of what we were doing. And sometimes that was often very negative, and I wouldn't recommend it as a good practice for anybody. But those kinds of concepts, the way people want to make decisions, constrains what you're able to do in code.

[01:00:33] Dave Adsit: Yeah, I have a couple of stories about change advisory boards that I will share with you at a later time, or perhaps if we meet up in person. They're not particularly positive experiences, I would say. But there's a lot about different types of, different things about your company culture. Is your company risk averse? Are you willing to accept risk? I remember at one company I worked at, we had basically forklifted the move fast and break things motto. And we moved fast, and we broke things occasionally. We moved pretty fast, and we broke things once in a while. And eventually, somebody came in, there was a change in leadership, and the new leader did not like breaking things. And so, he tried to change the motto to move fast but don't break things. And I'm sure you can imagine what happened is that when we could no longer break things, we could no longer move fast. Because we had to introduce additional layers of validation and testing before anything was allowed to go out. Because we didn't dare break things anymore. Right? So I've worked in several organizations that have had a QA department that comes around behind developers. The development works on Sprint 1, and in Sprint 2, QA is validating everything before they hand it to operations for deployment in Sprint 3. And I've worked in organizations where we have eliminated the QA function entirely and put responsibility for quality on the developers who write the code. And in my experience, you get better code when you do that. If somebody knows, if I release bad code, I'm going to get paged in the middle of the night when it breaks, and I'm going to have to get up and fix it or resolve the issues related to it. And then the next day, I'm going to have a fire drill where I try to fix all the problems and write a patch and release it and all of that. You get better code because nobody wants to do any of that. They just want to work on new features. And so you're going to take a little bit more time to put in place a few more automated tests and make sure that you understood the problem well before you release the solution. Yeah. So there's a lot of things about the culture of the company. If we have a good, if we've worked hard to create psychological safety and we do our blameless incident reviews when there's a problem and we truly don't try to lay blame at the feet of any one individual, we're going to have a different type of software development culture than if we are finger pointing people all the time as soon as anything goes wrong. Well, I know who touched it last. It was this person and we should string them up by their toenails. I don't know. I've honestly seen people get fired over it in the past and release a bad feature too many times in a row and you lose your job. And whether that was justified or not is hard to say. But it does speak to the culture of the company, right? That's one of the quips or adages about culture is that your company culture is who you hire, who you promote, and who you fire.

[01:03:44] Allan Stewart: Yeah. If you have a culture of meetings that affects whether you're going to be able to code or whether the developers pay attention in the meetings because they're actually coding instead. Trying to code instead. Thanks, Zoom. If you have a culture around experimentation that could change how you code, you do a lot more A-B testing, perhaps. If you have a culture that allows a lot of flexibility in what kind of programming languages you use, that'll have a different outcome in how people are actually writing code than if you have a culture of standardization and say, no, we only support these three languages or databases or whatever. Yeah.

[01:04:34] Dave Adsit: You can write in any language you want as long as it's Java. There's... And these are probably not the kinds of things that make it to the our five culture values are chart on the wall. But they're things that you figure out as you work in a company over time, right?

[01:04:54] Allan Stewart: Yeah. So to kind of summarize all of our discussion, I think some of the important points here are constraints aren't bad. Yeah. You need to recognize what constraints exist in your system and then use those to make great engineering, right? Know how you are constrained, what limitations you have, and use that like your kite example to fly high and make things work better. And then also being aware of those constraints means that you're aware when it changes because your strategy is going to need to change as your context and your constraints change.

[01:05:36] Dave Adsit: Yeah. I was going to say you have to be reactive when constraints change over time. We've talked about that multiple times, how different constraints apply at different points and how you have to be adaptive to that. And I mean, we've all worked in places where someone has ignored the constraints that exist, right? One example from my history is picking a programming language because it was kind of fashionable and not just fashionable, but also it has some technical merit to it. Like, oh, this is the fastest programming language in the market right now. And we could say it was C++ or Go or Rust because all of them have the same problem, right? The problem is there aren't any developers who know those languages. And so you put your most important thing in your company into that tech stack. And now you have to hire people to build a team. And you might find a couple of people who have spent time learning those languages and are okay at them or even pretty good. But what do you do if you need a bigger team than that? Now you've got to hire people and train them. And that makes everything slower. And do you want your flagship product, even if it's written in the best language, Do you want it to be written by novices to that tech stack? There's a story from the small talk era of one of the big small talk projects that at the beginning they estimated they needed about a dozen small talk experts to pull this thing off. And the management and the HR came back and said, what if we gave you a hundred novices instead? And they said, we would rather have the 12 experts. And they're like, I don't think you understand what we're saying. We have a hundred novices and zero experts. And so over the course of the project, of course the project was over budget and over time and all of the things that happen when you don't have the right people to do it. But at the end when they did their post-mortem and they said, hey, what did we learn? They said, we probably could have pulled this off within the time constraints and the budget constraints if we'd had a dozen experts from the beginning. Now we've got a few dozen experts because we all worked on this project for the last couple of years, but they didn't have them available at the time. And I mean, I've worked on these projects where someone made a technical choice without considering the greater constraints in the market. We picked Go instead of C-sharp because Go is the hot new sexy and C-sharp was originally built by Microsoft and must therefore be suspect at all turns. It turns out there's hundreds of thousands of C-sharp developers on the market and we could have filled a team almost immediately with very high skilled and very qualified developers. But instead we're struggling to find the handful of Go developers who aren't already happy working at one of the few big Go shops. In fact, in one example that I'm thinking of, we accidentally hired about eight to 10 expert level C-sharp developers and then taught them Go so that they could work on the Go project. And the question then becomes, are we fighting against sunk costs? Should we just start over and rewrite it in a language and a toolkit that we all know really well or not? I don't know. But there are definitely costs when you ignore your constraints. And so I would encourage everyone to understand the constraints that you're operating in and don't necessarily fight against them. Embrace them and leverage them to fly to higher heights.