Crafting Code Podcast

[00:00:16] Allan Stewart: Welcome to the Crafting Code Podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately I've been thinking about the opportunity costs of consulting work.

[00:00:30] Dave Adsit: I'm Dave Adsit, a VP of engineering, and recently I've been thinking about science fiction as a driver of technology development.

[00:00:38] Allan Stewart: Our topic for this episode is trunk-based development. This is a practice that's based on a single, simple idea. Always commit your changes to the mainline, or as they say on the website trunkbaseddevelopment.com, a source control branching model where developers collaborate on code. In a single branch called trunk, resist any pressure to create other long-lived development branches by employing documented techniques. They therefore avoid merge hell and do not break the build and live happily ever after.

[00:01:15] Dave Adsit: Well, that sounds an awful lot like another technique that we've discussed, continuous integration. So I would say that trunk-based development and continuous integration are very closely related. In continuous integration, we want to make sure that we're not just doing the same thing We want to make sure that all of our code works together on a regular cadence, ideally daily, right? We want to avoid the techniques we've used in the past where people would work for weeks or months on a custom branch and then try to integrate all of that code right at the end. Continuous integration, the definition from the article of that same name is continuous integration is a software development practice where each member of the team, whether they're with their colleagues' changes at least daily. So I guess trunk-based development encourages regular integration. Continuous integration encourages or requires daily integration. So how often should we integrate our code? Pretty soon, I guess. Yeah, pretty soon. I would say at least once daily, especially if you want to say that you are doing one of these practices. Continuous integration or trunk-based development, you have to integrate every single day. And that means everyone on the team is integrating every single day so that all of the changes that are being made, whether you're working in solo or in pairs or as a team, all of the code gets integrated together all the time. So I guess the immediate question is, can we still use pull requests if we're doing this? And I would say, I guess so. You can. You can. I have teams that use pull requests and integrate to main every single day. And so they are technically meeting the requirements of trunk-based development. I would say that in general, teams that adopt trunk-based development tend to drop some of the formality and rigor around branching and PRs and things like that. And so it just kind of feels overburdensome. It becomes heavyweight. If a branch lasts less than a day. Then going through a whole PR process, et cetera, et cetera, feels unnecessary.

[00:03:34] Allan Stewart: Yeah. I think, I think it can definitely work, right? Like you can keep to that definition, but the creation of the branch, like you say, it adds some overhead, the asynchronous nature of, Hey, now we have a pull request that needs to be merged also can add delay. Right. And, and the more often that there's delay in that process, you're going to be able to be tempted to continue to work on the branch. Or just natural variation is going to delay the other developers who need to be reviewing the pull request. And so you might not be able to actually meet that timeline to really be calling it continuous integration. And that's going to depend on a lot of factors just based around how the team works together. But a lot of the teams that I have worked with that are doing trunk-based development have also been doing some kind of collaboration. Yeah.

[00:04:58] Dave Adsit: code before it gets deployed to production. And I don't think anybody who advocates trunk-based development suggests that that's the ideal way that you work. So what is the point then? If we are doing this daily check-in to trunk, and I don't know, probably all of our listeners know this, but trunk is what it used to be called before Git. And Git called it master for a while and then converted to main. So when we say trunk, we mean main in Git.

[00:05:29] Allan Stewart: Exactly. The trunk is representing like the trunk of a tree, right? Like it's the main part of the tree. And exactly, the branches branch off from the trunk. And so that's the original interpretation there. And if you have too many branches, then it's more like a bush than a tree, I guess.

[00:05:52] Dave Adsit: Right. Well, and if you're using a tool like Git, you'll have local main and probably remote main. While it is not required when using a distributed version control system like Git, it is very, very common. In fact, I would say probably every team, not every team, almost every team that is using Git has a server that hosts their canonical main branch. And I would say you need to merge your main into canonical main. Right. And then you want to do it daily in order to meet the criteria for trunk-based development or continuous integration. Yeah. So if we say that you have to merge at least once a day, we have an upper bound on what a batch size can be. What is the lower bound on the batch size?

[00:06:37] Allan Stewart: Yeah. And I think this gets into the kind of the question about why you would be doing this in the first place, right? Like when you were reading about the continuous integration definition, it reminded me of things I've learned in the past. And it's been a while since I was in college. Right. And even then it seemed like there was this old practice that everybody would write a bunch of code and they'd spend days, weeks, months on writing code. And then there would be an integration phase, right? At one point that was considered the normal way of writing software. So everybody would write individual pieces of code and then at the end we'll bring it all together. And it causes a lot of problems because Right. there was so much drift that reconciling the differences between all of the different people's code or the different branches was very difficult. And so by reconciling the differences early and often, it is easy and cheap. And so once daily seems pretty good, but then like you were suggesting, we can take sort of take the limit of that. And probably at some point you're going to hit a limit. Right. And then you're going to hit a limit that's too small. Right. If every character that you type gets merged immediately with the mainline, well, those are going to be broken changes most of the time, a lot of the time. And that's not healthy. Right. We're going to kill the trunk of the tree by doing that. And so I think that there is a lower bound and it's probably somewhere in the small still. It's not individual characters, but it could be individual forms. Right. files, it could be individual, you know, functions within a file or something, so long as those changes are small and safe. Right. And at some point, you're going to have to look at, well, what does merging look like, and how much time and cost and pain is there to merge. And if the overhead of getting it merged onto the mainline is more expensive, then, you know, you have to find that U shaped optimization curve and say, Okay, well, we're not going to do it. Every two minutes, but we might do it, you know, several times a day.

[00:08:57] Dave Adsit: Right. Well, and I would say there is a potential for it to be as short as every few minutes. If you are working in a TDD cycle. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right Right Right Right Right Right Right Right Right Right Right Right Right that people had with merging. Yeah. Part of the reason why merging was always the thing that you pushed off until the very end of the process is because we would build such big batches that were so hard to put together. I think that one of the mistakes we make as software developers is that we learn patterns that work well for machines and not so well for humans. And specifically, I think of some of those early processes as being like executing a scatter gather routine on delivering software, but using humans as processors and forgetting that one of the critical components of scatter gather is that every machine is doing the same exact type of work and they're all completely independent units. And then when we try to gather back all of those disparate branches, we learn that everything is actually a little bit more coupled than we wished. And all the work is just a little bit different than we wished. And so we run into some problems, right?

[00:11:09] Allan Stewart: Yeah. Yeah. I think that there can definitely be fear there that if you're used to the merge step being painful, that you're not going to want to do that, right? You're going to try to avoid that. But continuous integration and trunk-based development are really about pushing that pain forward and saying, hey, if it hurts, let's do it more often. Let's get better at it. But the interesting thing is that what I have found by doing trunk-based development is that it is much less of a problem. It's less common that I will have to merge. And if I am running into merge conflicts, it probably means I'm working on the same code as somebody. Like most likely, I'm doing something that is very close to something that somebody else is doing on the team. And we need to communicate because we're going to be stepping on our toes no matter what, on each other's toes, no matter what.

[00:12:06] Dave Adsit: Right. I would say if you adopt this type of a workflow, it is going to encourage you to push first because you don't want to be the one who pulls and deals with the merge chain, the merge conflicts. And I would say you're right. The trunk-based development paradox or continuous integration paradox is that despite merging more often, we suffer fewer merge conflicts. And a merge conflict is a very, very important signal that you need to be talking to somebody else on the team. Modern merge tools have gotten so good that they can work out most of the issues that we used to get tripped up, And so if you're getting a merge conflict, I think you're right. You are working on the exact same thing someone else is. And maybe you're trying to do two different things with it.

[00:12:51] Allan Stewart: Right. And hopefully the batches are so small that if you do have a merge conflict, they're a lot easier to resolve than if you were trying to resolve the merge conflict of a sprawling 70 file change that has, you know, 60 of the files have merge conflicts. Like, but that's just no fun. And it's not fun. It's going to take forever to try and work through it.

[00:13:13] Dave Adsit: I think it's amazing that you think that a 70 file commit is a sprawling commit. I think that says a lot to the batch size that you are working in, Allan, and I applaud you for it. But yeah, the more often we merge, the less likely we are to have conflicts. And that means we're working in smaller batches and smaller batches are healthier because if we have small batches of working software that we can deliver, we have the opportunity to create value on a continuous basis as well. So I would say, even though you can use PRs, you shouldn't have really any feature branches the way we've thought of them as an industry for a while. Right? If a branch only lives for a couple of hours, you don't care what it's named. You don't need to think too much about that aspect of it. You're just going to get rid of it anyway. You're going to delete that branch because now you've got, and re-branch from main, you know, once you've pulled your coworkers, updates, so you can all stay working there together.

[00:14:16] Allan Stewart: And a lot of features don't last that long, right? At least in my experience, by the time that you're really ready to ship a feature, it often takes days, right? So like these timescales aren't the same. So if you're going to be continuously integrating and your branch is going to be merging into master continuously, then yeah, it might not matter what the name is because you've got to figure out, out ways to work in smaller chunks. Because by the time that you've done some development and you've had some back and forth with other developers, you've had some back and forth with your user experience people and stakeholders and whatever, features can change. And that typically doesn't happen within a day. And so you're going to have to figure out smaller ways to work and smaller pieces. And so no, it's not the typical feature branch that we're used to thinking of that this branch is going to live indefinitely until the feature is done and then

[00:15:16] Dave Adsit: it's going to get merged into the trunk. Right. And I think we all have experiences where even in a pairing or a more collaborative environment, one or two developers have gone off and worked on some big rewrite or refactor or brand new feature. And they've been working independently for weeks, if not longer. And when the code comes back, you're like, oh man, this doesn't even match our current state of the code base or our current understanding of the problem. And it's a problem. So by integrating every day, we can avoid some of that drift. We can enable our teams to deliver smaller batches more often, which creates a lot of other benefits that we've talked about in the past. But I would say the biggest reaction, the biggest drawback, the most visceral response to, you want me to merge every single day is going to be, but the feature I'm working on is not possibly going to be done today. It's going to take me a week or two, possibly even a month to get this entire feature built and delivered in a way that meets the customer's need. Yeah, I might have an MVP in two or three weeks, but I won't have the full thing done. So now what? Gotcha, trunk-based development, you're impossible. So what do we do about that?

[00:16:42] Allan Stewart: Well, we have to get better at carving things up. And I think the first step is getting better at breaking down the work, our user stories or the cards on your Kanban board or however you want to think about your work tasks that you're doing. Because there's usually ways that we can break those up. Yeah. And we have to start getting good at writing dark code. So being able to work towards a goal without actually having that code negatively impact users until we're ready to actually release it to them. And being able to do it in a way that's not going to be disruptive to people. The other developers that you're working on are working with. And so writing just new classes is one of the ways that I will often tackle this and say, Hey, this is a new feature. It needs a new database table. It needs a new set of files within the system. And if you are building those out and test driving them, but they don't touch any other code yet, there's nothing that's calling those and activating them. then it's pretty safe. You can take something, even though it's not quite done, or you don't know if it is going to operate correctly at scale or some of these other problems, you can just go ahead and put that into the code and it's not being executed yet. And that of course leads into the next thing, because you can't just leave it off on an island by itself forever. And so the next thing that you're going to want to start looking at is feature flags.

[00:18:34] Dave Adsit: Right. So writing new code, writing parallel code as you develop a new feature or change the behavior of an existing feature is a very powerful way of implementing code or committing and sharing code that you don't want people to execute yet. And feature flags are a great way to control the flow of your application as you build out and get And so if you have a feature flag system and you're using feature flags in your code base, you can have a code base that is constantly changing, constantly improving, constantly being enhanced by your developers, but also is continuously ready to ship to production. And for any feature that we're not ready to give to customers, we can just turn it off with a feature flag. Right. So the basic concept here, the basic goal of using a feature flag tool is to decouple your deployment step from your release step. So I will, I can build code. I could write code. I can test code. I can deploy code to our staging and production environments. But if I'm not ready for people to use it yet, I don't have to release it to them. And this can be new features. Pretty easy for new features. In fact, you just don't even let people have access to them. A little bit more complicated if you are enhancing or changing the behavior of an existing feature that people have access to. Now you're going to have to do a little bit more work to control when the new code is executed and when the new UI is shown versus when the old code is executed and when the old code is shown. One of the big benefits of using a feature flag system deploy to a production environment and potentially just do a limited release of the software to my own user or my dev team's user or my product manager or QA team. And we can validate new features, new functionality of a system in a production environment while still keeping that code isolated from or unreleased to our paying customers. We don't put the customer's experience at risk, but we do enable ourselves to validate the quality and behavior of new code in that production environment every day, right? Yeah. Yeah. And one of the things I was

[00:21:10] Allan Stewart: thinking about even just today is that very small changes don't necessarily need a feature flag, but my own rule of thumb is that I want feature flags to be so easy to use that there's not much Right. It's not a big deal to just use them every time. Because I hit one of those instances today where although our feature flags are very easy to use in my system, I got lazy. And I said to myself, I don't really need it for this particular change. And so then I was just about ready to be done. And I needed some feedback from another person. And then I found myself suddenly in this stuck position where it's like, Oh, well, you know, this is done. It's working. But I don't want to push it to the trunk and merge it. Because then if I do, it will automatically get deployed to staging. And if it's on staging, somebody might release it into production. And until I have the answer to this question, I'm not 100% confident that it should go to production. And so I waited. And luckily, in this case, it was only like 30 minutes. And then I got the answer. And then I got back into the flow. But I could have avoided that it would have taken me less than a minute to do it. Greg Isokaw, CNN Greg Isokaw, CNN Greg Isokaw, CNN Greg Isokaw, CNN Greg Isokaw, CNN Greg Isokaw, CNN Greg Isokaw, CNN Greg Isokaw, CNN Greg Isokaw, CNN Greg Isokaw, CNN Greg Isokaw, CNN Greg Isokaw, CNN Greg Isokaw, CNN Greg Isokaw, CNN Greg Isokaw, CNN Greg Isokaw, CNN

[00:22:50] Dave Adsit: Greg Isokaw, CNN Greg Isokaw, CNN fits of feature flags is that even if I've released something and given it to customers, if it's not working properly, I now have a, a rip cord to pull to rescue the code base from my bad change. Yeah. Right. I can go flip that flag off and that feature disappears until I am able to resolve whatever the problem was that it created. I think it is important as you're

[00:23:18] Allan Stewart: thinking about feature flags that there, there are a lot of toggling behaviors in code, right? You know, branching if statements are kind of one of the lowest level common things that we see in code. And so you do need to be sure that you understand what is a feature flag versus what are the longer term configuration setting variables that you're working with, because they look very similar, right? From, from a code standpoint, it's just if condition, then do this. Do that. But feature flags, as we're discussing here now in the, in the context of you're trying to do trunk-based development are typically transient. They are not meant to last a long time and the safety that you got from using them quickly disappears. If you let them persist for a long time, there are definitely horror stories about when feature flags get messed up or they, something was configured wrong and, and companies do a whole bunch of bad trades on the stock market based on that misconfiguration or things like that, because they didn't go through and clean up old behaviors, old flags. And that's, that's a different problem than saying that some of our customers are paying for the upgraded plan. And so they have access to things that other customers don't have access to because they are on the lower tier or the free tier or whatever. It's important to keep those two concepts separate.

[00:24:54] Dave Adsit: Yes, definitely. On my team, we refer to them by different names. So the feature toggle or feature flag that is used for release, we often refer to as a release control flag or a release flag, because that is one of the primary reasons why we use them. And then once something has been released to GA, we want to go clean up the flag that controlled that release. And when we go clean it up, we've got code that says, we're going to use this feature flag to clean up the code that was in the if branch or the else branch or whatever. Like if flag enabled, do new, do the new thing else do the old thing. Well, now that it's everybody is on the flag, the flag is GA generally available. Now I don't need that else case anymore. So I've got a bunch of code to clean up. So my teams will go in when they create a new feature flag, they'll create a card in their backlog or in their to do to go clean up that code and make sure that it's clean up. And that flag, once it's been released to GA and it generally it's available to all customers. And sometimes we refer to the feature bundling flags as just that feature bundling flags, but sometimes we just call them feature flags, right? You've got a bunch of conditions inside the code. Basically, like you said, the gold plan has these five features. The silver plan has these four and the bronze has these three. Well, we need a way to distinguish those. And that is a long term consideration.

[00:26:54] Allan Stewart: like pulling in a third party library or a SAS product or something like that. Although there are a lot of them out there that can, that can help. And I think if the main thing that you're looking at is trunk-based development, and you're trying to get away from all of the, the merge troubles, and you want to actually be integrated with everybody else's code, you can start really simple. I mean, even just simple if statements are functionally feature flags, right? Like every feature flag is tied to that condition. So the poorest man's feature flag can just be if false, or if user ID equals my user's ID, or something like that, that you can rest assured is not going to get out to other people. But there are a lot of advantages to using something that is a little bit more full featured, right? Because if you In your code, well, it works, it's safe to deploy. But once you turn it on, and you deploy the turned on version, then it's on. And it's, you're gonna have to do another deploy to turn it back off. And so having it having it pull from a database, or from a product that you're using, definitely has some advantages there that you can just quickly turn things on and off. Um, once you can start like segregating to different, different customers, or even like percentage based rollouts, where you say, Hey, we want to do 10% of the customer base today and have them try it. And then tomorrow, we're going to increase it to 20%, or things like that, so that you can you can learn. They're definitely nice things to have, as as you advance and grow with your feature flags. So if you want to build it yourself, and you have a table in the database that you can control and, and be able to turn, things off without having to deploy, that's not a bad middle ground. If you're just getting started, that's a pretty good way to start learning about feature flags and get them working.

[00:29:04] Dave Adsit: That's definitely the case. In fact, I've gone so simply as to, I try to avoid using if user ID equals my user ID, because probably my user ID is different in different environments. And so my testing gets harder. Also, that's not very descriptive of what it is I'm checking for. Right? I would like I typically, like to have a name for a flag that describes the feature that it would be enabled if this is on. But I've gone so simply as to just use like environment variables. So launch the app with this environment variable, you know, golden flute equals true. And now I've got the opera, now I can use the golden flute in this in this version in this instance, or not, right. But, but you're right, there's a lot of power that comes from building a more robust system. Including one of my favorite features of the SAS products that I've used in the in the past is the ability to put release control in the hands of the product manager instead of the developers. You know, in the past, deploy and release have always been the same thing. And that feels scary sometimes. But once we separate deploy from release, now I can deploy all the time. In fact, my team deployed something like 1500 times to production last year. You know that you'll notice that that number is bigger than 365 or bigger than the 250 working days, right? We deployed a lot. We didn't release code to customers nearly that often or that many times. Because we have adopted a tool that allows us to give each product manager a login and they can go and find that well-named feature flag and they can decide, I'm giving it to the alpha customers that are in this named group or the beta customers that are in this named group. Or I'm going to start rolling it out to 10% of all requests or 50, you know, as the product becomes more and more ready for all customers and we validated that. The PM gets to control that and the PM gets to control turning it off. Though the, you know, the tech lead or the developers on the team typically also have access to that tool so they can A, create new flags and B, pull the emergency stop if something's going wrong. So definitely worth having one of those tools if you have a mature practice and your team has the budget to pay for one.

[00:31:30] Allan Stewart: And as far as cost and budget goes, some of the tools, some of the products can be. Can feel pricey. But as soon as you start thinking about what would it take for me to have a team that builds equivalent functionality just in-house, I think in a lot of cases it quickly becomes apparent that, oh, well, it is cheaper to use the feature flag as a service than it is to try and build it and maintain it yourself. And hopefully what you're finding as you're doing trunk-based development. As you're getting the value out of feature flags is that it's worth having. You don't want to go back to where your release and your deploy were coupled. And by the way, you're merging in a bunch of people's changes all at the same time and hoping that it works. And if it fails, well, we're rolling back that entire thing and try to fix it. And then fixing it just makes the batch bigger. And you're just trying over and over to ram this thing through. And hoping. That it's going to work this time.

[00:32:37] Dave Adsit: Yeah. I, I'm not certain, but some of that feels like a personal attack, Allan. Have I worked in environments where we did weekly releases and if your code was the one that screwed up the release, everybody got to roll back and you all got to make your one week batch into a two week batch and maybe a three week batch. Yes. Yes, I have.

[00:33:00] Allan Stewart: It's only a personal attack if you were the one who shared the broken code. Right?

[00:33:04] Dave Adsit: Yeah. That's. I think. I said what I said. So, right. Feature flags are an enable and enabling technology that allows us to do trunk based development and trunk based development is an enabling technology that allows us to deliver working software reliably, consistently and often.

[00:33:28] Allan Stewart: So as I think about trunk based development and some of the benefits. My mind comes back to. Uh, can I. three parts. And we're trying to figure out how long is something going to take? Well, it's going to depend on how much your utilization is, what's your variation like, and what's the tact time. And the tact time is from when you started to when it finishes. So in software development, that's typically when did we start the feature and when did we release it? Now, trunk-based development doesn't necessarily have much to say about utilization as far as the people on the team and how busy they are and how close to 100% pegged they are. But it does tell us that variation makes things take longer and tact time, the longer that those are, it makes everything take longer. And so trunk-based development helps with our variation because because we have fewer merge conflicts, we're working in smaller branches, there's less opportunity for everything to shift. And then it can also help us with our tact time because we're getting things out the door potentially quicker. It's deployed if not released. And so I like that as a way of describing from a mathematical standpoint what the benefits are. There's other benefits too, right? It's easier to review changes when they're small. If you only have a if you only have a few things, you can review them. And so if you only have a few things, you can review them. And so if you only have a few things, you can review them. And so if you only have a few things, you can review them. And so Because they didn't have that opportunity to share.

[00:36:01] Dave Adsit: Right. And there's additional benefits, such as eliminating the need to do a code freeze. If we are very good and rigorous about doing our releases, or rather our deploys behind a feature flag, we don't have to ever stop deploying code in order to create a safe space where the code is unlikely to have an adverse impact on customers. Right? We want to have positive impacts as often as possible. If we're changing and committing and merging together and collaborating together on code every day, we don't really need things like change advisory boards as much. We also gain increased alignment between our developers because everybody sees the changes everybody else is making on a consistent basis. Right? As soon as if you write code and push it, and I write code and try to push it, and you've already pushed code, now I have to pull code. Now I see what's changed, and then I can push my code. Right? I'm going to pull, merge, rerun all the tests, and then push again. And we're good to go. We get the opportunity to deploy to production faster and more often. Right? Before we would have to wait until an entire feature was functional and well-tested before we could ever look at it in a production environment and see how do the queries perform? How does the UI look with the production data? You know, that kind of thing. But now, with feature flags and the ability to control who has access to, who the code is released to, we have a lot more ability to deploy sooner, test sooner, in a real environment. And also, our code is always release ready, because anything that's not ready is flagged off. And that, to me, is fantastic. That is one of the promises from early Agile, is that we want to always be working. We want our code to always be ready. be ready to deploy, deploy at any time. And using tools like trunk-based development and feature flags, we can actually accomplish that almost regardless of our team size. We build a monolith and we have multiple teams working on the same monolith and they use feature flags to enable them to work safely together despite working on different features that are going to be released

[00:38:23] Allan Stewart: to customers at a different cadence. Talking about this subject with people in the past, I've definitely encountered some who are skeptical. They don't know how this could work. It's hard. It can be hard to see how it works. It can be hard to believe that it will work sort of a, well, in my scenario, I don't know that that would work. But this has been a practice that's been around long enough now that I think it's pretty well battle tested. And And it works very well in most scenarios that I'm aware of. So there might be some that it won't work as well on. If you're sending out a probe to space or something like that, then maybe you have some additional constraints that's not typical for a software developer. But there are many large companies that use these techniques from small code bases, lots of individual teams, working on their individual branches to big mono repos where everybody in the company is working on the same code, the same main line every day. And it works. So I'd recommend that anybody who hasn't tried it, dig in, do a little bit of research and give it a try, because there are a lot of these benefits that you can get by doing so.