Crafting Code Podcast

[00:00:16] Allan Stewart: Welcome to the Crafting Code Podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately I've been thinking about the opportunity costs of consulting work.

[00:00:30] Dave Adsit: I'm Dave Adsit, a VP of Engineering, and recently I've been thinking about science fiction as a driver of technology development.

[00:00:38] Allan Stewart: Our topic for this episode is Trunk-Based Development. This is a practice that's based on a single, simple idea. Always commit your changes to the mainline. Or, as they say on the website trunkbaseddevelopment.com, "a source control branching model where developers collaborate on code in a single branch called trunk, resist any pressure to create other long-lived development branches by employing documented techniques. They therefore avoid merge hell and do not break the build and live happily ever after."

[00:01:14] Dave Adsit: Well, that sounds an awful lot like another technique that we've discussed, continuous integration. So I would say that, you know, Trunk-Based Development and continuous integration are very closely related. In continuous integration, we want to make sure that all of our code works together on a regular cadence, ideally daily, right? We want to avoid the techniques we've used in the past where people would work for weeks or months on a custom branch and then try to integrate all of that code right at the end. And continuous integration, the definition from the article of that same name is, "continuous integration is a software development practice where each member of the team merges their changes into a code base together with their colleagues' changes at least daily." So I guess Trunk-Based Development encourages regular integration. Continuous integration encourages or requires daily integration. So how often should we integrate our code? All of the code gets integrated together all the time. So I guess the immediate question is, can we still use pull requests if we're doing this? And I would say, I guess so. You can. I have teams that use pull requests and integrate to main every single day. And so they are technically meeting the requirements of Trunk-Based Development. I would say that in general, teams that adopt Trunk-Based Development tend to drop some of the formality and rigor around branching and PRs and things like that. And so it just kind of feels over burdensome. It becomes heavyweight. If a branch lasts less than a day, then going through a whole PR process, et cetera, et cetera, feels unnecessary.

[00:03:34] Allan Stewart: Yeah, I think it can definitely work. Right? Like you can keep to that definition, but the creation of the branch, like you say, it adds some overhead, the asynchronous nature of, "Hey, now we have a pull request that needs to be merged," also can add delay, right? And, and the more often that there's delay in that process, you're going to be tempted to continue to work on the branch, or just natural variation is going to to delay the other developers who need to be reviewing the pull request. And so you might not be able to actually meet that timeline to really be calling it continuous integration. And that's going to depend on a lot of factors just based around how the team works together. But a lot of the teams that I have worked with that are doing Trunk-Based Development have also been doing some kind of collaboration, pair programming or mob programming techniques that have allowed them to to get those code reviews done and get their code merged in without having to resort to the asynchronous nature of the pull request.

[00:04:42] Dave Adsit: Yeah, certainly not our topic of discussion today, but I would say continuous integration or Trunk-Based Development will lead to a process of continual review. We want to continuously be reviewing the code. Certainly, neither of us is advocating that only one developer ever look at code before it gets deployed to production. Right. And I don't think anybody who advocates Trunk-Based Development suggests that that's the ideal way that you work. So what is the point then? If we are doing this daily check-in to trunk, and I don't know, probably all of our listeners know this, but trunk is what it used to be called before Git. And Git called it master for a while and then converted to main. So when we say trunk, we mean main in Git.

[00:05:29] Allan Stewart: Exactly. The trunk is representing like the trunk of a tree, right? Like it's the main part of the tree. And exactly, the branches branch off from the trunk. And so that's the original interpretation there. And if you have too many branches, then it's more like a bush than a tree, I guess.

[00:05:53] Dave Adsit: Right. Well, and if you're using a tool like Git, you'll have local main and probably remote main. While it is not required when using a distributed version control system like Git, it is very, very common. In fact, I would say probably every team, not every team, almost every team that is using Git has a server that hosts their canonical main branch. And I would say you need to merge your main into canonical main daily in order to meet the criteria for Trunk-Based Development or continuous integration. Yeah. Yeah. So if we say that you have to merge at least once a day, we have an upper bound on what a batch size can be. What is the lower bound on the batch size?

[00:06:37] Allan Stewart: Yeah. And I think this gets into the kind of the question about why you would be doing this in the first place. Right. Like when you were reading about the continuous integration definition, it reminded me of things I've learned in the past. And it's been a while since I was in college. And even then it seemed like there was this old practice that everybody would write a bunch of code and they'd spend days, weeks, months on writing code. And then there would be an integration phase. At one point that was considered the normal way of writing software. So everybody would write individual pieces of code and then at the end we'll bring it all together. Other. And it causes, it caused a lot of problems because there was so much drift that reconciling the differences between all of the different people's code or the different branches was, was very difficult. And so by reconciling the differences early and often, it is easy and cheap. And so once daily seems pretty good, but then like you're, you were suggesting, we can take sort of take the limit of that. And probably at some point, you're going to hit a limit that's too small, right? If every character that you type gets merged immediately with the main line, well, those are going to be broken changes most of the time, a lot of the time. And that's not healthy, right? We're going to kill the trunk of the tree by doing that. And so I think that there is a lower bound and it's probably somewhere, somewhere in the small. Still, it's not individual characters, but it could be individual files. It could be individual, you know, functions within a file or something. So long as those changes are small and safe. Right. And at some point, you're going to have to look at, well, what does merging look like and how much time and cost and pain is there to merge? And if the overhead of getting it merged onto the mainline is more expensive, then you have to find that U-shaped optimization curve and say, "okay, well, we're not going to do it every two minutes, but we might do it several times a day."

[00:08:57] Dave Adsit: Well, and I would say there is a potential for it to be as short as every few minutes if you are working in a TDD cycle. If your TDD cycle is extended to red, green, refactors, in other words, write the test, write the code, refactor the code, commit the code, push the code. If you moved into a cycle at that scale, that would be probably the ideal smallest state, right? Because at that point, if your tests all pass, then you know the code you have works for the definition of works that you have automated. And so now is an opportunity to commit that or to share that code with the rest of the team. And I would say that is probably not a terrible place to be. You might want to work, you might want to have code that has created new value before you push, but I would say that at the smallest, it would be this code passes all the tests. This code is not worse than the code you have now. And if you are doing that, you are going to be working in very, very small steps. Very, very, you are going to avoid most of the challenges that people had with merging. Yeah. Part of the reason why merging was always the thing that you pushed off until the very end of the process is because we would build such big batches that were were so hard to put together. I think that one of the mistakes we make as software developers is that we learn patterns that work well for machines and not so well for humans. And specifically, I think of some of those early processes as being like executing a scatter-gather routine on delivering software, but using humans as processors and forgetting that one of the critical components of scatter-gather is that every machine is doing the same exact type of work and they're all completely independent units. And then when we try to gather back all of those disparate branches, we learn that everything is actually a little bit more coupled than we wished. And all the work is just a little bit different than we wished. And so we run into some problems, right?

[00:11:09] Allan Stewart: Yeah. Yeah, I think that there can definitely be fear there that if you're used to the merge step being painful, that you're not going to want to do that, right? You're going to try to avoid that. But continuous integration and Trunk-Based Development are really about pushing that pain forward and saying, "hey, if it hurts, let's do it more often." Let's get better at it. But the interesting thing is that what I have found by doing Trunk-Based Development is that it is much less common that I will have to merge. And if I am running into merge conflicts, it probably means I'm working on the same code as somebody. Like, most likely, I'm doing something that is very close to something that somebody else is doing on the team. And we need to communicate because we're going to be stepping on our toes no matter what, on each other's toes, no matter what.

[00:12:05] Dave Adsit: Right. I would say if you adopt this type of a workflow, it is going to encourage you to push first because you don't want to be the one who pulls and deals with the merge chain, the merge conflicts. And I would say you're right. The Trunk-Based Development paradox or continuous integration paradox is that despite merging more often, we suffer fewer merge conflicts. And a merge conflict is a very, very important signal that you need to be talking to somebody else on the team. Yeah. Modern merge tools have gotten so good that they can work out most of the issues that we used to get tripped up by. And so if you're getting a merge conflict, I think you're right. You are working on the exact same thing someone else is. And maybe you're trying to do two different things

[00:12:51] Allan Stewart: with it. Right. And hopefully the batches are so small that if you do have a merge conflict, they're a lot easier to resolve than if you're trying to resolve the merge conflict of a a sprawling 70 file change that has, you know, 60 of the files have merge conflicts. But that's just no fun. And it's going to take forever to try and work through it.

[00:13:13] Dave Adsit: I think it's amazing that you think that a 70 file commit is a sprawling commit. I think that says a lot to the batch size that you are working in, Allan, and I applaud you for it. But yeah, the more often we merge, the less likely we are to have conflicts. And that means we're working in smaller batches and smaller batches are healthier because if we have small batches of working software that we can deliver, we have the opportunity to create value on a continuous basis as well. So I would say, even though you can use PRs, you shouldn't have really any feature branches the way we've thought of them as an industry for a while, right? If a branch only lives for a couple of hours, you don't care what it's named. You don't need to think too much about that aspect of it. You're just going to get rid of it anyway. You're going to delete that branch because now you've got, and re-branch from main, you know, once you've pulled your coworkers updates, so you can all stay working there together.

[00:14:16] Allan Stewart: And a lot of features don't last that long, right? At least in my experience, by the time that you're really ready to ship a feature, it often takes days, right? So like these timescales aren't the same. So if you're going to be continuously integrating and your branch is going to be merging into master continuously, then, yeah, it might not matter what the name is. Because you've got to figure out ways to work in smaller chunks. Because by the time that you've done some development and you've had some back and forth with other developers, you've had some back and forth with your user experience people and stakeholders and whatever, features can change. And that typically doesn't happen within a day. And so you're going to have to figure out smaller ways to work and smaller pieces. And so, no, it's not the typical feature branch that we're used to thinking of, that this branch is going to live indefinitely until the feature is done, and then it's going to get merged into the trunk.

[00:15:18] Dave Adsit: Right. And I think we all have experiences where even in a pairing or more collaborative environment, one or two developers have gone off and worked on some big rewrite or refactor or brand new feature, and they've been working independently for weeks, if not longer. And when the code comes back, you're like, "oh man, this doesn't even match our current state of the code base or our current understanding of the problem." And it's a problem. So by integrating every day, we can avoid some of that drift. We can enable our teams to deliver smaller batches more often, which creates a lot of other benefits that we've talked about in the past. But I would say the biggest reaction, the biggest drawback, the most visceral response to, "you want me to merge every single day," is going to be, "but the feature I'm working on is not possibly going to be done today. It's going to take me a week or two, possibly even a month to get this entire feature built and delivered in a way that meets the customer's need. Yeah, I might have an MVP in two or three weeks, but I won't have the full thing done for a month. So now what?" Yeah. Gotcha, Trunk-Based Development. You're impossible. So what do we do about that?

[00:16:42] Allan Stewart: Well, we have to get better at carving things up. And I think the first step is getting better at breaking down the work, you know, our user stories or, you know, the cards on your Kanban board or however you want to think about your work tasks that you're doing. Because there's usually ways that we can break those up smaller. But then there's still the problem that even small tasks might take more than a day. And we have to start getting good at writing dark code. Right. So being able to work towards a goal without actually having that code negatively impact users until we're ready to actually release it to them and being able to do it in a way that's not going to be disruptive to the other developers that you're working with. Working with. And so writing just new classes is one of the ways that I will often tackle this and say, "Hey, this is a new feature. It needs a new database table. It needs a new set of files within the system." And if you are building those out and test driving them, but they don't touch any other code yet, there there's nothing that's calling those and and activating them, then it's pretty safe. You can take something, even though it's not quite done, or you don't know if it is going to operate correctly at scale or some of these other problems, you can just go ahead and put that into the code, and it's not being executed yet. And that, of course, leads into the next thing, because you can't just leave it off on an island by itself forever. Ever. And so the next thing that you're going to want to start looking at is feature flags.

[00:18:34] Dave Adsit: Right. So writing new code, writing parallel code as you develop a new feature or change the behavior of an existing feature is a very powerful way of implementing code or committing and sharing code that you don't want people to execute yet. And feature flags are a a great way to control the flow of your application as you build out and get closer and closer to having a releasable, complete feature. So if you have a feature flag system and you're using feature flags in your code base, you can have a code base that is constantly changing, constantly improving, constantly being enhanced by your developers, but also is continuously ready to ship to production. And for any feature that we're not ready to give to customers, we can just turn it off with a feature flag, right? So the basic concept here, the basic goal of using a feature flag tool is to decouple your deployment step from your release step. So I can build code, I can write code, I can test code, I can deploy code to our staging and production environments. But if I'm not ready for people to use it yet, I don't have to release it to them. And this can be done for new features. Pretty easy for new features, in fact. You just don't even let people have access to them. A little bit more complicated if you are enhancing or changing the behavior of an existing feature that people have access to. Now you're going to to have to do a little bit more work to control when the new code is executed and when the new UI is shown versus when the old code is executed and when the old code is shown. One of the big benefits of using a future flag system is that I can deploy to a production environment and potentially just do a limited release of the software to my own user or my dev team's user or or my product manager or QA team. And we can validate new features, new functionality of a system in a production environment while still keeping that code isolated from or unreleased to our paying customers. We don't put the customer's experience at risk, but we do enable ourselves to validate the quality and behavior of new code in that production environment. Every day, right? Yeah.

[00:21:08] Allan Stewart: Yeah. And one of the things I was thinking about even just today is that very small changes don't necessarily need a feature flag, but my own rule of thumb is that I want feature flags to be so easy to use that there's not much of a reason not to, right? It's not a big deal to just use them every time. Cause I hit, I hit one of those instances today day where although our feature flags are very easy to use in my system, I got lazy and I said to myself, "eh, I don't really need it for this particular change." And so then I was just about ready to be done and I needed some feedback from another person. And then I found myself suddenly in this stuck position where it's like, "oh, well, you know, this is done, it's working, but I don't want to push it to the trunk and merge it because then if I do, it will automatically get deployed to staging." And if it's on staging, somebody might release it into production. And until I have the answer to this question, I'm not a hundred percent confident that it should go to production. And so I waited. And luckily in this case, it was only like 30 minutes and then I got the answer and then I got back into the flow, but I could have avoided that. It would would have taken me less than 30 minutes to have the feature flag in there. And, and if you put that in at the get go, then you can deploy your changes all the way throughout. You have stuff that is half working, half broken, new UI that shows up on the, on the screen, on the website, whatever. That's okay. Cause it won't show until you turn it on and it makes it just a lot safer to, to get things out there.

[00:22:50] Dave Adsit: Well, and safety is one of the key benefits of feature flags is that even if I've released something and given it to customers, if it's not working properly, I now have a ripcord to pull to rescue the code base from my bad change. I can go flip that flag off and that feature disappears until I am able to resolve whatever the problem was that it created.

[00:23:15] Allan Stewart: I think it is important as you're thinking about feature flags that there are a lot of toggling behaviors in code, right? You know, branching if statements are kind of one of the lowest level common things that we see in code. And so you do need to be sure that you understand what is a feature flag versus what are the longer term configuration setting variables that you're working with, because they look very similar, right? From a code standpoint, it's just if condition, then do this, else do that. But feature flags, as we're discussing here now, in the context of you're trying to do Trunk-Based Development, are typically transient. They are not meant to last a long time. And the safety that you got from using them quickly disappears if you let them persist for a long time. There are definitely horror stories about when feature flags get messed up or something was configured wrong, and companies do a whole bunch of bad trades on the stock market based on that misconfiguration or things like that, because they didn't go through and clean up old behaviors, old flags. And that's a different problem than saying that some of our customers are paying for the upgraded plan. And so they have access to things that other customers don't have access to because they are on the lower tier or the free tier or whatever. It's important to keep those two concepts separate.

[00:24:55] Dave Adsit: Yes, definitely. On my team, we refer to them by different names. So the feature toggle or feature flag that is used for release we often refer to as a release control flag or a release flag, because that is one of the primary reasons why we use them. And then once something has been released to GA, we want to go clean up the flag that controlled that release. And when we go clean it up, we've got code that was in the if branch or the else branch or whatever. Like, if flag enabled, do new, do the new thing, else do the old thing. Well, now that it's everybody is on the flag, the flag is GA, generally available. Now I don't need that else case anymore. So I've got a bunch of code to clean up. So my teams will go in, when they create a new feature flag, they'll create a card in their backlog or in there to do to go clean up that that code and that flag once it's been released to GA. And it generally, it's available to all customers. And sometimes we refer to the feature bundling flags as just that feature bundling flags, but sometimes we just call them feature flags, right? You've got a bunch of conditions inside the code, basically, like you said, the gold plan has these five features, the silver plan has these four and the bronze has these three. Well, we need a way to distinguish those and that is a long-term consideration or concern of the code. Exactly. So if we want to use feature flags, how does one get started with this concept of feature flags? Say I've bought in, I at least want to experiment with Trunk-Based Development, but now I need some feature flags because most of the stuff I work on takes me two to three days to get finished. And I now know how I have to merge every day. How do I get started with feature flags?

[00:26:49] Allan Stewart: I don't think that you have to jump straight to like pulling in a third party library or a SAS product or something like that. Although there are a lot of them out there that can that can help. And I think if the main thing that you're looking at is Trunk-Based Development and you're trying to get away from all of the merge troubles and you want to actually be integrated with everybody else's code, you can start really simple. I mean, even just simple if statements are functionally feature flags, right? Like every feature flag is tied to that condition. So the poorest man's feature flag can just be if false, or if user ID equals my user's ID, or something like that, that you can rest assured is not going to get out to other people. But there are a lot of advantages to to using something that is a little bit more full-featured. Because if you have some hard-coded feature flag in your code, well, it works, it's safe to deploy, but once you turn it on and you deploy the turned on version, then it's on and you're gonna have to do another deploy to turn it back off. And so having it pull from a database or from a product that you're using definitely has some advantages there, that you can just quickly turn things on and off. Once you can start like segregating to different customers or even like percentage-based rollouts where you say, "hey, we want to do 10% of the customer base today and have them try it. And then tomorrow we're going to increase it to 20%" or things like that so that you can learn, they're definitely nice things to have as you advance and grow with your feature flags. So if you want to build it yourself and you have a table in the database that you can control and be able to turn things off without having to deploy, that's not a bad middle ground. If you're just getting started, that's a pretty good way to start learning about feature flags and get them working.

[00:29:04] Dave Adsit: That's definitely the case. In fact, I've gone so simply as to, I try to avoid using if user ID equals my user ID, because probably my user ID is different in different environments. And so my testing gets harder. Also, that's not very descriptive of what it is I'm checking for, right? I would like, I typically like to have a name for a flag that describes the feature that it would be enabled if this is on. But I've gone so simply as to just use like environment variables so launch the app with this environment variable, you know, golden flute equals true, and now I've got the opera, now I can use the golden flute in this, in this version, in this instance, um, or not right? But but you're right, there's a lot of power that comes from building a more robust system, uh, including one of my favorite features of the SAS products that I've used in the pro in the past is the ability to put release control in the hands of the product manager instead of the developers. In the past, deploy and release have always been the same thing, and that feels scary sometimes. But once we separate deploy from release, now I can deploy all the time. In fact, my team deployed something like 1,500 times to production last year. You'll notice that that that number is bigger than 365 or bigger than the 250 working days, right? We deployed a lot. We didn't release code to customers nearly that often or that many times because we have adopted a tool that allows us to give each product manager a login and they can go and find that well-named feature flag and they can decide, "I'm giving it to the alpha customers that are in this named group or the beta customers that are in this named group or I'm going to start rolling it out to 10% of all requests or 50," you know, as things, as the product becomes more and more ready for all customers, and we validated that, the PM gets to control that. And the PM gets to control turning it off. Though the, you know, the tech lead or the developers on the team typically also have access to that tool. So they can A, create new flags and B, pull the emergency stop if something's going wrong. So definitely worth having one of those tools if you have a mature practice and your team has the budget to pay for one.

[00:31:30] Allan Stewart: And as far as cost and budget goes, some of the tools, some of the products can feel pricey. But as soon as you start thinking about what would it take for me to have a team that builds equivalent functionality just in-house, I think in a lot of cases it quickly becomes apparent that, oh, well, it is cheaper to use the feature flag as a service than it is to try and build it and maintain it yourself. And hopefully what you're finding as you're doing Trunk-Based Development, as you're getting the value out of feature flags is that it's worth having. You don't wanna go back to where your release and your deploy were coupled. And by the way, you're merging in a bunch of people's changes all at the same time and hoping that it works. And if it fails, well, we're rolling back that entire thing and try to fix it. And then fixing it just makes the batch bigger. And you're just trying over and over to ram this thing through and hoping that it's going to work this time.

[00:32:38] Dave Adsit: Yeah, I'm not certain, but some of that feels like a personal attack, Allan. Have I worked in environments where we did weekly releases? And if your code was the one that screwed up the release, everybody got to roll back and you all got to make your one week batch into a two-week batch and maybe a three-week batch? Yes, yes, I have.

[00:33:00] Allan Stewart: It's only a personal attack if you were the one who shared the broken code, right?

[00:33:04] Dave Adsit: Yeah, I think I said what I said. So, right. Feature flags are an enabling technology that allows us to do Trunk-Based Development. And Trunk-Based Development is an enabling technology that allows us to deliver working software reliably, consistently, and often.

[00:33:28] Allan Stewart: So as I think about Trunk-Based Development and some of the benefits, my mind comes back to queuing theory, because I always love to put in a little queuing theory whenever I can get the chance.

[00:33:41] Dave Adsit: Always queuing theory.

[00:33:43] Allan Stewart: And in queuing theory, there is a formula called the Kingman approximation. And this I feel like describes pretty well why you want to use Trunk-Based Development. You can go and look up the formula online, but it basically boils down to the three parts. And we're trying to figure out how long is something going to take? Well, it's going to depend on how much your utilization is, what's your variation like, and what's the tact time. And the tack time is from when you started to when it finishes. So in software development, that's typically when did we start the feature and when did we release it? Now, Trunk-Based Development doesn't necessarily have much to say about utilization as far as the people on the team and how busy they are and how close to 100% pegged they are. But it does tell us that variation makes things take longer and tack time, the longer that those are, it makes everything take longer. And so Trunk-Based Development helps with our variation because we have fewer merge conflicts. We're working in smaller branches. There's less opportunity for everything to shift. And then it can also help us with our tack time because we're getting things out the door potentially quicker it's, it's deployed if not released, and so I like that as a way of describing from a mathematical standpoint what the benefits are. There's other benefits too, right? It's easier to review changes when they're small. If you only have a few things to code review, well, that's easier. It's also less risky because those changes have less of an effect. There's less of a scope for them to mess something up. And there's less duplication because the longer that you go without getting everybody merged to the main line, the more often you end up having multiple developers doing the same thing a little bit differently in similar parts of the code because they didn't have that opportunity to share.

[00:36:01] Dave Adsit: Right, and there's additional benefits such as eliminating the need to do a code freeze. If we are very good and rigorous about doing our releases or rather our deploys behind a feature flag, we don't have to ever stop deploying code in order to create a safe space where the code is unlikely to have an adverse impact on customers. We want to have positive impacts as often as possible. If we're changing and committing and merging together and collaborating together on code every day, we don't really need things like change advisory boards. Passwords as much. We also gain increased alignment between our developers because everybody sees the changes everybody else is making on a consistent basis, right? As soon as if you write code and push it and I write code and try to push it, and you've already pushed code, now I have to pull code. Now I see what's changed and then I can push my code, right? I'm going to to pull, merge, rerun all the tests, and then push again. And we're good to go. We get the opportunity to deploy to production faster and more often, right? Before we would have to wait until an entire feature was functional and well-tested before we could ever look at it in a production environment and see, how do the queries perform? How does the UI look with the production data? Data, you know, that kind of thing. But now with feature flags and the ability to control who has access to who the code is released to, we have a lot more ability to deploy sooner, test sooner in a real environment. And also our code is always release ready because anything that's not ready is flagged off. And that to me is fantastic. That is one of the promises from early Agile is that that we want to always be working. We want our code to always be ready to deploy, deploy at any time. And using tools like Trunk-Based Development and feature flags, we can actually accomplish that almost regardless of our team size. You know, we use, we build a monolith and we have multiple teams working on the same monolith and they use feature flags to enable them to work safely together despite working on different features that are going to be released to customers at a different cadence.

[00:38:25] Allan Stewart: talking about this subject with people in the past, I've definitely encountered some who are skeptical. They don't know how this could work. It can be hard to see how it works. It can be hard to believe that it will work. Sort of a, "well, in my scenario, I don't know that that would work." But this has been a practice that's been around long enough now that I think it's pretty well well battle tested. And it works very well in most scenarios that I'm aware of. So there might be some that it won't work as well on. If you're sending out a probe to space or something like that, then maybe you have some additional constraints that's not typical for a software developer, but there are many large companies that use these techniques, from small code bases, you know, lots of individual teams working on their individual branches, to big mono repos where everybody in the company is working on the same code, the same main line every day, and it works works. So I'd recommend that anybody who hasn't tried it, dig in, do a little bit of research and give it a try because there are a lot of these benefits that you can get by doing some.