Crafting Code Podcast

[00:00:15] Allan Stewart: Welcome to the Crafting Code Podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately I've been considering the utility of running local LLMs and embedding models.

[00:00:30] Dave Adsit: I'm Dave Adsit, an engineering leader, and I have recently been thinking about the challenges of scaling both features and customers at the same time.

[00:00:40] Allan Stewart: Our topic for this episode is volatility. Something volatile is inconsistent, ephemeral, usually changing rapidly, possibly explosive.

[00:00:51] Dave Adsit: Yeah, I looked up a few definitions of volatile just to make sure that I knew all of them, and the first one is easily evaporated at normal temperatures, like solvents or like rubbing alcohol. Another one, the one that normally comes to mind for me is liable to change rapidly and unpredictably, especially for the worse. That's the one that comes to mind first for me. And the third one is retaining data only as long as there is a power supply connected, which is very relevant to computers. And possibly LLMs.

[00:01:29] Allan Stewart: Right. And then the other thing that we've thought about as we discuss this topic in preparation for this episode is that volatility means risk for all those aforementioned reasons. And so volatility is something that comes up a lot, both in our business and our product, also in code. So where does volatility show up in a business?

[00:01:57] Dave Adsit: The first thing that came to mind for me is volatility in business or in your product. And when I think about that, I think either your product is rapidly changing or the market where you sell your product is rapidly changing. Your customers are changing. Something is changing very quickly. And if you want to get ahead of that or take advantage of that, you've got to change along with it very quickly. And so that's volatility in business and in a market. And so I think that's a very important thing. And I think that's a very important thing. The transaction costs are going to be very high or impossible to overcome. And so you miss out on the whole concept of creative destruction, like the economic concept of creative destruction, where things are created and therefore change, and then some things are going to go away. And so what you want to do in your business is be one of the change agents, not one of the creatively destroyed victims of a changing market.

[00:03:26] Allan Stewart: Right. Yeah. This makes me think about the worldly mapping that we talked about in a prior episode. In the far right of the evolution of a worldly map is where you find a lot of companies in kind of the commodity space. Commodities tend to be non-volatile, right? You know exactly what they are, what they're supposed to do, all the specifications for them, right? So I can buy batteries anywhere, right? If I buy anybody's batteries from anybody, I can buy batteries from Energizer or Duracell or Costco or Amazon Basics or whatever it is. They'll all fit. They'll all work. I mean, some of them might be better than others. They definitely claim to be better than others. But in that commodity space, right, I can get them anywhere. I can go down to the corner store and they'll have batteries. So existing in that space is very different from where a lot of SaaS companies live. In a product space, which is further to the left, closer to Genesis of ideas where things are very volatile and you don't know how it's going to be. And you can make a business out of commodities, but how you scale it and how you operate is very different in that space than in a lot of the the companies that I have worked in, at least. A lot of software companies tend to be more volatile.

[00:04:58] Dave Adsit: Definitely the case for me as well. And so much so that we just have a common phrase in the industry, right? We talk about moving fast and breaking things. And the questions that you have to ask yourself is how fast is fast enough and possibly how fast is too fast? And how many things are you willing to break? If I were Energizer or Duracell, I would not ship a battery called a two and a half, like, there's two, a, and there's three, a, nobody has anything that you can put a two and a half, a, or a, 4, a, into, uh, actually, I think I may have seen a 4, a, once, uh, But, uh, it's, there's, there's, there's, there's, there's, there's, there's, there's, there's, things that you need to think about, you need to think about things differently, right? You are willing to break some things in order to move faster. And that affects how you make decisions as a business, as an organization. And one of the things that I think about a lot in that space is how reversible is this decision? If I make a bad decision, are we committed and we have to keep going this path, even though we know it leads to our destruction? Or did we create a way to back out and reverse and do something else instead? Yeah. And that reversibility becomes more and

[00:06:31] Allan Stewart: more important as the volatility increases, right? So if you know that you're going to be building some commodity and, hey, I'm going to make this decision and it's going to affect the next million batteries that we make, it's probably not that big of a deal, right? Because the batteries are going to happen, right? Yeah. Yeah. Yeah. You've already got a lot of those things in place. There's not as much that can shift around. It's not foolproof, but low volatility means that you're more likely to have a better understanding of that decision space. And if it's not reversible, that often doesn't matter quite so dramatically as when things are volatile, when the market conditions are changing very rapidly, when the technologies that you're using to build the thing are changing very rapidly. That affects our decision-making. And if we choose something that's not reversible or there's a very long tail, well, there's a lot of opportunity for the situation to change before we can actually

[00:07:42] Dave Adsit: deliver on the thing that we were doing. For example, if I know that 20% of the U.S. battery market is double A's, and it's so many billions of dollars per year, But I can afford to spend several years and tens of millions of dollars building a battery assembly line that will be more efficient than my competitor's battery assembly line, because then I will get the profits and they won't. I can do that if we're in a non-volatile product space. However, I wouldn't want to do the same thing in a SaaS product if I hadn't even validated that anybody would buy it. if I'm trying to set up a new SaaS product and I don't know if there is a market for it yet, then I probably shouldn't invest tens of millions of dollars before I validate that there is a market. And I probably shouldn't have product releases that cost millions of dollars. I should have product releases that are much more, much less expensive and much more reversible.

[00:08:47] Allan Stewart: I remember recently you sent a link to a website that was something about, it was very tongue in cheek, but is my company a markdown file is basically the idea of it. Because with AI assisted coding now, there's a lot that can be done with, hey, here's a markdown file, please create a product like this, right? And so it's a very volatile space for a lot of SaaS companies right now because there are a lot of opportunities for people to create, something custom build their own thing that doesn't have to meet all the requirements of a multi-tenant, multiple people, multiple business spanning product if it just meets their own one need that they have. On the other hand, the tools to create those things are extremely volatile at the moment as well. And so you can't necessarily bank on having them forever either. And so you gotta be careful and smart about making those decisions. And if it were to change, right? So like one of the things that I've been thinking about a lot in that specific volatile space right now is that I wanna make sure that I am keeping, not letting my coding skill completely atrophy just because a robot can write code a lot faster than me because I don't know from day to day what's gonna happen in the space. And if in a month, I have to completely change my tool set because everything's changed or one of these companies falls apart, I don't wanna be locked in on a sinking ship.

[00:10:31] Dave Adsit: Definitely the case for me as well. We all know that when you're scaling from zero users to one user, that's different than when you're scaling from one user to 100,000 concurrent users, right? And so if I'm building something just for my own consumption, I can take a lot more liberties. It can be a lot more volatile because the user, and the developer are the same person. And I know what changes I'm putting in. And also I'm the one who requested them and then executed them. So it can be super volatile. When I'm working on LLM skills for my own local consumption, they change really fast. And when I see something that I don't like, I just change it and then move forward. If I were shipping those to customers, the change cadence would be different. It would probably be a little bit lower. I'd probably do a little more validating before I just make a change and see what happens. Right. So that speaks to several things then, not just the business and the product volatility, but also the project structure and maybe even your team structure. We've talked before about how we like having stable teams that we feed a series of projects or products through. That's as opposed to the project management strategy of the developer, define a project and then build a team around it. We've talked about our preference for being on teams that are stable, longer term, build more of that, not just camaraderie between members of the team, but that high trust of, I know if Allan says he's going to work on the back end, he's going to do that. And then that means I can go work on the front end and we can come back together tomorrow morning and see where we are. And all of those things happen as we build relationships, rapport over time on a stable product team. We also know where our weaknesses are and what types of projects we should ask for additional resources on because of things that we may not be good at. So having a stable team and running a series of projects through it is one strategy. And forming a new team for every project is a different strategy, one that I don't prefer.

[00:12:45] Allan Stewart: And it depends on the nature of the kinds of work that you're doing too, right? Yeah. I think that that team per project works really well when you have short-term projects. When you're not looking at a long tail of maintenance, you're not looking at, which could be a couple different ways, right? Like it could be, this is something we're going to build and it's going to ship like a video game, right? And we're going to support it for a couple of years and then after that, it gets into maintenance mode at best, right? Which is very different from we're building a product, and we're expecting over the next five to 10 years, we're going to be working on this same product. And it's just the context matters quite a bit.

[00:13:31] Dave Adsit: Yeah. So it's just another dimension that we need to be considering when we talk about the concept of volatility. How volatile is our team and our team composition? So one of the things that we've done is we've seen departments or organizations that mix different types of volatility together. So you might have one team that is working on the long-term stable monolithic product that supports the whole business. And then another team might be working on the disruptor for that. And they might be working in different ways, right? Maybe they even have different metrics or they prioritize metrics differently. For example, if you are on that monolith team and you're sustaining that monolith for a long time, the long haul, maybe when you go grab your Doro metrics, you're looking at meantime between failure and meantime to recovery. And you might be super risk averse when it comes to anything that could disrupt the product in production for the customer base that's using it right now. And you might be willing to sacrifice lead time, longer lead times in order to create lower risk for taking that product offline or... for, you know, anything that would cause it to go offline and stay offline for any period of time. Right. Meanwhile, if you're working on the disruptor team, you're like, oh, well, we can do this whole thing with AI and we're going to replace the main product. Let's go see if we can do that. If you're on that disruptor team, maybe you don't care so much about meantime between failure and meantime to recovery because you don't actually have any customers yet. You're still early enough in the process that you're figuring things out. And so maybe you, you are willing to take on more risk in order to have lower lead times and try things faster and iterate and experiment faster than you would have if you were making changes to, you know, the 100,000 active user monolith.

[00:15:38] Allan Stewart: Personally, I tend to like to focus on the meantime to recovery over between failures because failures happen. It's hard to avoid them happening in some way. Some form or another. There's definitely places that I've worked where the meantime between failures was very important. Educational software is a good example of this. There's a lot of school districts who do not want, you know, the teachers, they're so busy, they don't want all of the changes. Please don't change anything until the summer. Right, until we're between terms or, you know, between school years. But even so, in a case like that, personally, I like to look at how can I keep that lead time really low? So if something does go wrong, we can fix it really fast. And maybe we just have some long-lived feature flags before we roll out some changes that are going to impact people. But again, it depends on that context. And that also can drive your system architecture. Right? The kind of business volatility that you are seeing might change what practices you use. Right? So we've talked about, you know, before that during the early startup phase, maybe you're doing a lot of rapid prototyping and there are some forms of testing that you skip completely. Or like the kind of the tiger team that you were talking about, you know, the team that is experimenting heavily and they don't really have customers yet. There's a lot of opportunity to go in and say, hey, we can save some time by just trying the concept in a prototype that we're going to throw away. Before we actually deliver it at scale.

[00:17:24] Dave Adsit: That's right. And that means that leads to the idea of having different types of teams doing different types of work. Right? We've talked previously about the concept of team topologies where there are four different categories of teams. You'll have your stream aligned teams, which are building features along a roadmap. And those could be either the high volatility or the low volatility teams. Then you also have your enabling teams and those teams are going to be pretty volatile because one of their responsibilities is go fix problems for other teams when they come up. You know, those are like your firefighters who are running around fixing problems and then putting in place sprinkler systems and fireproof materials so that fires don't come about again. Then you've got your platform teams that are, you know, they need to be building, they need to be building in a low volatility way. They need to be building things that are stable for a while so that your stream aligned teams can build on, can rely on them and build on top of them. Right? Those teams are going to maybe be building things like your design system. Here is version two of our UI designs system. You can use all these components and you'll be able to use all these components for the next two years. And then the fourth category in team topologies is the complicated subsystem team. And this one's interesting because when I've actually worked on teams that have one of these, what I've seen is the people who are members of that complicated subsystem team become very risk averse and the team itself becomes very non-volatile. We tend to have a lot of stability on the membership of that team because it's hard to come up to speed on the complicated subsystem and changes to the complicated subsystem can be very risky. And so that team doesn't want new members coming around and breaking things and causing them to work through the weekends. So, you know, those different categories of teams, those different purposes that teams fill can also speak to how volatile their work is or should be.

[00:19:36] Allan Stewart: And again, going to the architecture idea that can also depend on those kinds of work. So if we're working on, you know, a team that is working on a specific stream aligned thing, and they're innovating, they might need to, to use an architecture that's different, right? So, we, we haven't needed any Python in our system before, but now we do, because we want to use, you know, some machine learning library, or we want to take advantage of an AI tool, and, so, we're, we're, we're, we're, we're, we're, we're, we're, we're, we, we, we, we, we, we, we, we, we, accomplish a specific goal. But that doesn't necessarily mean, because that's high volatility, you might see a different, some different patterns there. But down at the platform team, they're going to be wanting to do things very in a very non volatile way. Like how can we, how can we adjust the architecture, so that we can support these teams doing lots of different kinds of work, but in a kind of a unified way that works well across the different, you know, substrates of volatility, right? There's, there's some things that need to be more open, right? More loosey goosey, where you can make changes. And then there's other places where if, you know, if you let some of those key architectural decisions, just be, you know, whatever anybody wants to do, you Speaker 2 can have a bad time.

[00:21:08] Dave Adsit: Yeah. Well, and you might intentionally decide, Hey, we're going to build this as a monolith. We're going to use these specific tools that we know very well. We're going to not let our team change or get too large. We're going to kind of hold some of those organizational things static. And some of those architectural decisions we're going to make in a way that allows us to iterate really quickly. Yeah. Where once we start shipping APIs and once we start decomposing a monolith into a bunch of microservices or following a service oriented architecture, we're going to start having more challenges in the changes that we make. Right. Right. Even I may not even want to use or want to extract a design library until I've shipped the first iteration or second iteration of my product to actual customers. Yeah. Yeah. Yeah. But I'm going to have to change those designs really quickly. And if I just added a little bit of friction of I'm going to put this in NPM and then pull it back to, you know, all of that, which isn't a lot of friction, but it can be enough that it slows you down enough that it can impact your ability to make change at the cadence of the business. Right. And so you and I have both experienced that most of the startups that we've worked on when they start getting to that early scale up phase, they've got a monolith. Yeah. And it's probably a mess. And it seems to be a commonality of companies that have survived past that early, very volatile space is that they made the simplest decisions they possibly could at the time.

[00:22:50] Allan Stewart: Because to do otherwise is often a recipe for failure because you don't know how it's going to change. And if you add a lot of overhead where change means that you're going to have to edit a lot of things in order to stay consistent or that you've made your architecture too strict, then yeah, you might not make it. That's right.

[00:23:14] Dave Adsit: Okay. So what practices are, how do practices and volatility intersect? There are some practices that allow support volatility, higher volatility, some practices that impede volatility, just like architectures.

[00:23:27] Allan Stewart: Yeah. And I think about it with just the variability of work, even the volatility of what am I doing today? I haven't done it before. I'm writing my tests differently or not writing tests at all as contrasted with your regular production code. So the kind of work that you're doing matters, but then the practices themselves have different levels of volatility. So like source control,

[00:24:09] Dave Adsit: probably one of your more stable practices. That's a stable practice. That was, that's the one that you and I together identified as basically non-negotiable. We could not come up with a legitimate reason not to use source control if you know that it exists and have

[00:24:24] Allan Stewart: access to download Git. Right. Yeah. So like there's a lot of projects where I have no intention to ever push it to GitHub or any of the other places. I'm not sharing it with my team. I'm not, I don't need the distributed part of the distributed source control, but I would like some, some source control because as I make, make changes and I break it, I want to be able to go backwards. And so if you're using Git already, it's just so easy to say, well, Git init. Okay, great. Here's a commit. Now let me try something else. Oh, that worked. Awesome. Let me commit and try something else. And so, yeah, that's a very low volatile, you know, very low volatility. It's very stable. The use of source control in general

[00:25:12] Dave Adsit: is a very stable practice that you and I would use on every project always. Right. Regardless of other characteristics of the project. Um, the adjacent to that, but not quite in the same spot is test driven development. I would use tester driven development on almost any project or product that I thought was going to, I was going to keep around even for a little while. I think the, the few examples or exclusions to that, the exceptions, uh, include maybe I'm doing a research spike. Maybe it's a new library that I don't understand. Well, I might not write tests on that until I do understand. Um, so source control, I would use always test driven development. I would use almost always.

[00:25:57] Allan Stewart: And then more volatile than that are methodologies, right? So are you a scrum team that has a two week cadence? Are you trying to do lean and Kanban? Are you doing some kind of agile methodology? Like those things can be somewhat interchangeable. Right? Like no matter which one of those my team is doing, I'm probably still getting to use source control and test driven development because those are more stable, but I can be a little bit more flexible in these work methodologies. And they might even shift on me as I'm working on a team, as I'm working in a company. And that's okay because they tend to be, those practices can be more volatile and we can, we can change up things like the cadence of our retrospective. Or we can change up whether we're going to have a daily standup or some other kind of meeting. And that doesn't impact us. Like there is an impact, right? Like it, it changes how your day feels, but it's also more volatile than some of these other things.

[00:27:07] Dave Adsit: And if you're doing any of the agile methodologies properly, retrospect and adapt are part of it. And you will change your methodology as you are doing it. Right.

[00:27:18] Allan Stewart: Otherwise you're. You're arguably not doing that in the first place.

[00:27:21] Dave Adsit: Right. It's definitional. And we should also be learning better ways to do it and learning better ways to do software development means that we are at the very least experimenting or someone's experimenting and we're copying them, thus running our own validation of that experiment.

[00:27:39] Allan Stewart: Right.

[00:27:40] Dave Adsit: So, but, so yes, those are going to change as you use them. The, the next one that I think about as being even a little bit more volatile than that, is the collaboration styles. Sometimes we're working solo. Sometimes we're pairing, sometimes we're mobbing. Sometimes we're doing all three in the same day. Sometimes we're doing all three twice before noon. It, it can be that the collaboration style is going to, that you're using for each individual feature or part of your product or whatever you're working on that can change much faster than even your software development. Yeah. Methodology can be very volatile, especially on teams that are very comfortable with collaboration. You can move in and out of high collaboration all day without any real impact or without any adverse impact.

[00:28:33] Allan Stewart: Yeah. And then the example that we thought that was kind of on the most extreme is your AI tools usage. Yeah. We kind of alluded it alluded to this before, but how you work with your AI tools, that might change even, more frequently.

[00:28:51] Dave Adsit: A team that I worked with very recently in six months, we went through three different strategies for how to use AI. We picked one tool and everybody used it. And then we picked a variety of tools that you could pick off of. Everybody could change and use off of menu. And then we went back to one tool again, but it was a different tool than the first time. And so the, that, and that all happened over the course of six months.

[00:29:14] Allan Stewart: JavaScript frameworks, or maybe another one that has that level of high volatility. Oh my gosh. Exactly. Everybody's using this thing now. Oh, no, we're all changing. We're moving again.

[00:29:25] Dave Adsit: It used to be the joke that by the time you finished the conversation about which JavaScript framework to use for the project, three more had been launched.

[00:29:34] Allan Stewart: And so then the important thing here is recognizing which things are more volatile, which ones are not and adapting appropriately. You know, if you're changing, like if you're you know, um, software that you're using to do the source control every few months, that's going to be terrible. And you're going to have a really bad time. But if you're changing your JavaScript stuff every few months, well, that's kind of par for the course and you plan for it. You expect it. You change your behaviors to allow for that high volatility and for it to be okay. Right? Like you have to, you have to make sure that you've set yourself

[00:30:19] Dave Adsit: up for that kind of, kind of thing. Right. And just to simplify that and say that as concisely as possible, understand where volatility is going to be in your code. And then don't mix the highly volatile and the low volatility stuff together. Don't cross the streams. If that means anything to anyone anymore, you've got to have your, your high volatility stuff over here and your low volatility stuff over there. That's part of the reason why we tend to use things like specific architectural patterns, why we like to separate your UI framework from your display code, from your UI code, you know, separate those things out in a way that you are not going to have to rewrite an entire layer of your software when the next version of the framework gets

[00:31:10] Allan Stewart: released by the vendor. Yeah. So when we get down to the code level, I think about volatility, meaning change, and it comes back to coupling and cohesion. That's exactly right. Are they compatible? The things that change together, are they placed together in the code? Do we have it in a cohesive way so that we can make a shift? For example, if you've got a new and unproven feature and you've coupled it too closely to something, your existing code base, then you've made it more difficult to make changes to the existing thing. And then you aren't able to move as quickly on the new thing because of those constraints that you inherited by, by grabbing on. So like, I see this all the time with entities, business entities inside a lot of software have this problem because you'll say something to yourself like, oh, well, for this new feature, we, we need to access the user object. Okay, great. Well, we already have the user object and we're just going to pull that in. But now we need to add a new field and we're already using the user. So we're just going to add this one thing onto the user table and we've changed the user entity and now we're highly coupled there. And then somebody else needs to change a user for a different reason. And now it's more complicated. And some of the other, I think like we might not be able to move very quickly on my new feature because there's other constraints that are in the system around the user. It's like, oh, well you forgot that the user has to be able to be serialized in a particular way because that's how we deliver messages to another part of the system. And now we've got to go and make changes to that. So those are all examples of, of ways that I see code commonly not thinking about volatility or the, the developers, they weren't thinking about the volatility of certain aspects of code and they just go ahead and couple everything together. Even things that don't belong together. But there are easy alternate alternatives. You can create like, it's a, it's a bit of a pain upfront to say, it's like, oh, well this, we're not going to couple to the user. We're going to create a new table called, you know, user feature preference or, you know, whatever it is. And so now we have to have a new table. We have to do some new stuff. We have to make sure that, you know, our, our relationships with the new users are handled correctly. But taking that small investment means that I didn't have to take all that other coupling. And so if the user object changes, that's okay. Or if my feature changes dramatically, as we release this new thing and we're, we're learning, Hey, I can change this new table as much as I want. Cause it's specific to this feature. It's not. It's not coupled together and we can move fast.

[00:34:16] Dave Adsit: When I think about that along those same lines, I think about strategies that we've tried to decouple things, right? If I've got a volatile front end, there are at least three standard strategies for decoupling that volatile front end from my less volatile backend. One of them is restful APIs. I'll just make the front end, figure out my schema and pull the data that it wants out of my restful endpoints. And that means I protected my backend from the front end. That can actually slow down front end development because now I have to navigate a whole bunch of restful endpoints and make a whole bunch of HTTP requests. Another solution that we've tried as an industry is GraphQL, which is like, Oh, well, I don't want to make a bunch of requests. I'll just make one request for all the things at once. And now I've got another layer in my API. That's got to navigate all those things and be configurable and all that, which again, make that might. Okay. The volatility on the front end easier, but it makes volatility on the backend a little harder because now I've got to ma or manage all those mappings. One that I like is backends for front ends. It's another strategy, another architecture for solving the same problem, which is I'm instead of having generic API endpoints that you can call from anywhere. I'm going to make a new API endpoint for every one of my front end pages, and it'll just make a single request and I'll put everything in the response. And, and I know that I know that this API endpoints only used in one place because it's only authorized to be used in one place. And now when I create another page that needs different data, even if it's 90% the same, I'm just going to create a new endpoint that pulls all the data together. And so I've decoupled some of that volatility by saying, Hey, if you've got a bunch of changes that you're making on the front end, well, good news. You can have as many changes as you want. Without breaking the API, because this endpoint is just for you. And now I can probably do move some of the, some of that work back and say, okay, I've got a stable domain model, stable ish domain model. Obviously it's going to be evolving as I implement new features. So I've got this stable ish domain name domain model. And when I make changes, I'm going to know if I broke something because I've compiled and unit tested that seam. As opposed to. Hoping that I didn't remove something that was used by one of the AI or one of the, not AI, one of the front ends that's written in a different code base in a different language, right? If my backend is.net and my front end is react, there's a seam in there where things could break. But we can use strategies that actually make it easier to identify and fix those breaks before they have an adverse effect on customers.

[00:37:10] Allan Stewart: And those strategies. Don't have to. Only exist. Along those technology lines, right? Like we use them. So frequently and things like, oh, well, this is going to be my mobile front end. And this is going to be my server side API backend. But some of those just kind of feel natural, but you can, there are plenty of patterns. For doing similar things like all within your front end or all within the backend. Right. So you might have facades and adapters and you might use like. So gonal or architectures. You might, you might use some of like the micro front end patterns, even if you're not doing like a whole micro front end concept, just to help you get through some of those, those issues. The book clean architecture had a concept in there related to volatility that I really liked where it talks about the degree of abstraction and stability. So stability is a measure. Of like your incoming and outgoing dependencies. And generally. I liked this idea that. That generally you want to look at things. In how concrete. And how abstract they are. So if I'm going to, in my code, mention anything by name. It better either be very abstract. Right? Like this is just, it's an interface, for example, or. Or it's an abstract class. But something that's concrete. Like string. That can be okay to. Mention by name. Because it's stable. I know that they're not going to ship dramatic changes to string from day to day. Right. But the, but the things that in my, you know, other concrete things inside of my code base, like a specific class that might change more rapidly because it's in our evolving code base rather than in a. A library that is tested and trusted. Or a framework or, you know, or, or even just the standard. Stuff that comes right out of your language. You know, those things don't change very often. And so you, because they're stable. You can build upon them. If you build upon a lot of things that are unstable. It's like the house of cards and it's going to fall over.

[00:39:35] Dave Adsit: Yeah. Well, and that's especially important if your product is ever evolving. Right. You might. You would be better off. As an example. To depend on. An I send messages interface or an I message sender interface, depending on your preference for. Style. You would be better off to depend on that. Then. A rabbit MQ. Message broker. Or a Kafka message broker. And that would allow you to have. To isolate those changes to one part of the code. If you made a decision to. Upgrade your rabbit library or replace rabbit entirely with. A different. Queue. You could isolate those changes to one part of the code. And the volatility would not trickle through the entire code base. And so that's one of the things that I like to think about in terms of designing software for. The future. For I don't like to say to future proof it or make it extensible because I'm not really making it extensible. I'm making it so that I can change it. I don't necessarily want to extend it. I want to replace components. So. When we talk about clean architecture, I. I go back. Over and over and over to the four rules of simple design. And. And how those help with volatility. The. The. So. Volatility in terms of simple design. One of the things that comes to mind is the concept of do not repeat yourself or the dry principle, right? When we try to dry up our code. One of the things that we may do inadvertently. Is to. Sent the question, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, database as a DTO and then do something on it, do something with it in my domain, and then push it out the API into the front end as a view model. And so I look at those two things and they may have a very similar shape initially. And if I'm trying too hard to make the code dry, I might combine those together, even though they're going to have a very different volatility characteristic. The database interactions are probably going to be pretty stable. I put a lot of thought into the table. It has all the data I need. There's no more data, but the things I'm doing in the front end, oh, maybe I'm going to be doing localization or I'm going to be doing, I don't even know what kinds of changes I might make. Maybe I'm going to format something on the server because I need server time instead of client time for some reason. And so the view model might change at a different cadence, is likely to change at a higher cadence than the data transfer object. If I've looked at those and said, they are the same, I'm going to combine them together. Now I'm bringing all of that volatility into both spaces. And that actually reveals something, which is an interesting characteristic of your domain model, which is that you have some false cognates, meaning they're not the same. They just kind of look the same. And volatility is the key difference between them. And over time, they will reveal themselves to be different. But in their initial, you might say, oh, this is duplicate. I'm going to create one object that represents both concepts.

[00:43:16] Allan Stewart: Yeah, there have definitely been times where I'm working on a new feature and it feels very similar to an existing feature. And so there's a temptation there to say, oh, well, let's just reuse some code because we already have code that deals with sorting this, or we already have code that can render something like this, or even that if we just tweak the shape of the new thing, a little bit, then it fits the shape of an old thing. And then we don't have to rewrite any code. But the problem is that the new thing tends to be volatile because it's new. It's not as well understood yet. It's not tested. And so then it wants to change at a different rate. And if you're not careful, you might go changing a bunch of things that actually were fine, but shouldn't have been changing. And you really should have, you know, made some duplication or some apparent duplication. It felt like duplication at first, but then as you move along and you see how it changes differently from the other parts of, you know, the other entities or concepts in the system, then it's better to have them separate and they can change independently and you can find out, which is why we've talked about a lot of times the, you know, the rule of threes, you know, don't, don't collapse the duplication until you, you've done it a few times and you're sure. Yep. These are the things that actually go together. This is, this is what's real duplication versus the false cognate. Or we've also talked about this idea of harvesting frameworks. Don't, don't start off creating the framework that you think is going to power everything because you don't know yet. Go build a few things and then see if there's some commonality that makes sense to extract into one reusable piece rather than, than trying to make it reusable from the get-go.

[00:45:13] Dave Adsit: Yeah. We've talked a lot about the rule of three and designing for use or building for use rather than reuse. And I think that it's important to remember that that only works if you have a strong habit and practice of doing refactoring as you go. If you are, if you are operating in a space where it's, you're just like, you never go back and refine the things that you've been working on. First of all, you're going to make a huge mess. But secondly, if you're going to make a huge mess, you're going to have to refactor. And second of all, some of these practices don't work very well for you. You should probably put a little bit more thought into it if you're not going to refactor later, which is, you know, I think probably the wrong strategy. I mean, do maybe put a little thought in, but then also make sure that you are continuously refactoring your code. Some percentage of your time is refactoring to remove duplication, actual duplication, make concepts crisper and easier to understand. So I like to think about volatility from the perspective of architecture, like physical building architecture. When you are building a physical building, there's several layers that are each, each additional layer is more volatile than the one underneath it. And I think if I remember correctly, the, this, the classic layers are structural, mechanical, electrical, and the facade. Like one of the things that if you lease a building for your business, one of the things that often comes with it is tenant improvements. And you'll get some kind of a budget for replacing the carpet and repainting the walls and moving walls around to be more appropriate for your business versus the previous business that was here. All of that is part of the facade, right?

[00:46:56] Allan Stewart: Right. As long as they're not load-bearing walls.

[00:46:58] Dave Adsit: You're not going to move a load-bearing wall, at least not more than once. And if you are, you know, inside of your space, but you might erect some impacting the mechanical and the structural. You're putting in new walls. You want to have outlets in them. You're going to run some additional electrical lines under the floors or through the ceilings and drop those into those walls. And that's all possible when you're redoing the facade. You're going to have a lot harder time changing the electrical without breaking the facade. Right. And the same is true for software projects. There's, I think I tend to identify a few more layers than for physical architecture. You've got your language and your runtime and your framework. Those are your more stable things. Those are probably more akin to the structural layer of the architecture. You can sometimes change your language within a framework or a runtime, within a runtime at least. Like the .NET runtime supports C-sharp and Visual Basic. And F-sharp. The JVM, Java Virtual Machine supports, I don't even know how many languages, Java and Scala and Clojure and whatever. Kotlin. Then you've got things that are a lot more volatile, like your domain. You're building your domain on top of the framework, on top of the runtime. And then you've got individual workflows. You probably can create a bunch of new workflows, new jobs that can be done by your application without making too many changes to the domain. And then the user interface, of course, is the most volatile of all. You can change the user interface without changing any of the functionality of the actual software. Like we want to update the colors. We want to move the buttons around. We want to use windows instead of tabs. I don't know. There's a lot of things that you can do in the interface that don't impact any of those other layers.

[00:49:35] Allan Stewart: Yeah. Yeah. work for us. And one of the ways that I've seen that done is by saying, yes, the LLM is volatile, but we're going to create certain tools or skills or other things that interact with the LLM or the LLM can interact with that are deterministic. And so we know what's going to happen when a particular tool call is made, for example, even though we don't know exactly how the LLM is going to behave in a particular way at a particular time. There's also statistical things that we can do to say that we have certain likelihoods for non-deterministic outcomes and how we're going to handle them differently. Even sometimes I think about that, even just in error monitoring for my APIs, it's hard to make sure that you've covered every single kind of error. And so if I look at it in terms of percentiles, that can be very useful. What's the average case look like? What's the median case look like? What's the 95th percentile look like? What's the max? What's the longest running query? Is that something I should mess with? Or do I need to look at the query that runs all the time and it's faster, but not fast enough? And the longest running query actually only gets run once a month and so on. It's fine.

[00:51:33] Dave Adsit: Yeah, that is definitely the case. I was just thinking about a personal experience where I was using an LLM and one of the things I was doing is converting certain Markdown files to PDF. And I had been doing this successfully for weeks. In fact, I had created a Python skill that does that because there's a couple of tools that are really good in the Python ecosystem for converting Markdown files to PDFs. And then one time I ran it, I said, you know, I used kind of the same, command that I had for all of the other things. At least it looked the same to me as a person. And the LLM is like, hey, I have to go brew install this PDF generator. Can I do that? And my answer was no, absolutely not. In fact, I'm going to give you another instruction. Anytime I ask you to do a PDF, use this existing skill that you and I wrote together weeks ago. It's like, got it. I'll always use the Python skill instead of trying to install new things on your computer. Until the next time. Until the next time I decide you need new things installed on your computer.

[00:52:42] Allan Stewart: So as we wrap this discussion up, I think the important thing to consider is in the work of software, there's always going to be things that have different levels of volatility. Recognizing that and planning for it can help us. We can look at what things are going to be more volatile in our business and make coding decisions. to allow for some things to be more volatile, that we are okay with that, that we are, even how we work can be more agile and more able to deal with change, to respond to change. And in our code base, we can find those places where volatility matters and adjust how we write our code to compensate for it.