Crafting Code Podcast

[00:00:15] Allan Stewart: Welcome to the Crafting Code Podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately, I've been thinking about the many evolutions and revisions that a novel goes through before being published, like Hobbits, such as Bingo and Trotter.

[00:00:36] Dave Adsit: I'm Dave Adsit, a VP of Engineering, and I have recently been thinking about the importance of iterative character development in both storytelling and personally.

[00:00:46] Allan Stewart: Our topic for this episode is culture and architecture. In our last episode, we talked about company cultures and how they affect us as people. Now we're going to talk about how culture affects our software architecture.

[00:01:00] Dave Adsit: So one of the first things we have to do then is decide what definition of software architecture we're going to be using, because there are a lot and we use a lot of different ones. And if somebody asks, I'll just throw up one of the very simple ones like, "oh, architecture is the hard things to change." But I know you've put a lot of thought into what would be a more effective effective definition for us to work from in this case?

[00:01:25] Allan Stewart: I don't know about effective, but for me personally, I like to think about it as the structure of your software system, right? So how does it work? How is it built? So we call it software architecture because we're drawing from the physical architecture, like physical buildings, right? It's a metaphor. And that that metaphor breaks down horrifically in some places, but, but generally you can look at a structure like a cathedral and say, "oh, well, it's a Gothic style." Right. And so, you know, it's got Gothic architecture and you'll point out different pieces of the structure. And I think you can do the same thing with, with code. So like, how do you deploy it? How do the components interact? You know, what kind of coupling it has inside internally? Like there's a lot of dimensions to it. But essentially for me, it's that overall idea of what the structure is, which going to your definition is hard to change.

[00:02:24] Dave Adsit: It is, right? And honestly, when I think about building architecture, I tend to go deeper. I think of the things that we see like the style as just being a facade. And so in some of the studying that I've done, I've encountered things like you've got the the structural layer. You've got multiple layers in a building, right? The structural layer, the mechanical layer, the electrical layer, the facade. There's probably others I don't know because I'm not either a building architect or a building engineer. I work in software, squishier things. But yeah, I think that that's a good definition for us to work with is the the overall structure of your system, which does in fact, tend to be hard to change. So that just means we need to get it right the first time, right? And then we'll have a perfect architecture and a perfect culture.

[00:03:18] Allan Stewart: That'd be nice. If, uh, if only there were a perfect architecture and then that would be great. We could be done with this, wrap up this recording and everybody goes home and we're done with software engineering. There you go. Problem problem solved. Yeah, no, unfortunately, it's, it's, it's not that easy, right? And, and it's hard, because there's so many things that are in flux, right? Every organization is different. Every, it's just like what we were talking about with culture. In that last episode, every company is different. They have different objectives, they have different needs. Technology changes over time, like all of these things impact what we want to do with software architecture. And so, yeah, I think you inevitably end up needing to change your software architecture regardless, just because of entropy.

[00:04:11] Dave Adsit: Yeah, that's definitely the case. I often tell people that the only time you can stop maintaining your software is when you have completed its end-of-life life plan and shut off the servers, right? Up until that point, you're constantly making changes to things to keep things up to date to, like, there's always security patches. There's always something, there's always something you have to be doing. So it's, it's like tending an organic thing, a garden or a fish tank. You've got to always be adapting and adjusting. You can't just just put things in place and expect them to stay that way forever. Right.

[00:04:53] Allan Stewart: So for me, one of the big ways that software architecture and culture are connected is summed up through Conway's Law. And I know we've talked about this in the past, but Conway's Law says that "organizations which design systems are constrained to produce designs, which are copies of the communication structures of these organizations." And I look at that quote, that famous quote, and I think about that. It's like, okay, well, system designs, to me, that's another way of saying software architecture, at least when you're working in software systems. And then the communication structure of organizations is highly related, in my mind, to culture. Because culture is effectively the result of the interaction between people and we communicate, right? And communication is a big part of our interaction. And so how we interact with each other is our culture. It's also represented in our communication systems and our communication structures and therefore in our code systems.

[00:06:01] Dave Adsit: I completely agree with that. In fact, independently, I made a note that how we communicate is one of the biggest aspects of our culture, or at least has one of the largest outsized impact in the culture that we create, right? We talked previously about how culture is an emergent characteristic or an emergent phenomena from the various things that we do. And communication is one of the most important things we do as people. In fact, I've read in some places that that's one of the things that sets us apart from other mammals, other great apes, whatever is communication. So when it comes to communication, it can be intentional. We can design it, right? And we talk about those sometimes as communication flows, information flows, and communication can definitely be accidental. And sometimes I hear those referred to as communication drafts, like a draft, like air going in a place you didn't expect it to. When I was a kid, we would throw these little sand, these long sandbag worms on the floor of our bedrooms to keep the air from the rest of the house from blowing under the door and cooling us off or whatever, right? This is a common thing years and years and years ago is to try to cut down on the drafts. First of all, it makes your home more energy efficient. It makes your team more efficient or effective as well. If we can cut down on communication drafts, accidental communication. One example that I see is lack of clarity around how to communicate about a problem in the system. We use tools, chat tools like Slack, and you may find that you have dozens or hundreds of channels in your company's Slack. And people will just post stuff wherever, or more likely than not, once they've found an engineer that they like, they'll just take all of their questions directly to that person.

[00:07:57] Allan Stewart: Just go straight to the direct message.

[00:08:00] Dave Adsit: Yeah, and now instead of having an intentional communication flow, you've got a communication draft. And that can be, you know, part of that is architecture. Part of it is just overall system culture. But I would say, you know, communication patterns have a very strong impact on both our people culture, the culture of the people in our socio-technical system, but we talk about them intentionally in software systems as well, right? In fact, I would say that we're much more intentional about how we communicate in the software hardware side than we are in the

[00:08:32] Allan Stewart: people side. Yeah. Yeah. There's some very explicit protocols that you need to use to be able to communicate, whether that's because you compiled it together and that, you know, it got linked together or dynamically linked, or because you have to use like HTTP or there's like a message broker or something like there are very specific ways that you can communicate between systems. So yeah, we, we tend to think more, I think more explicitly about that because as a human, you're just like talking and texting and sending DMS over Slack. Like all of that feels a little bit more natural.

[00:09:08] Dave Adsit: Right. I mean, and you and I, we have a story that we've probably told multiple times, which is when I said something to the effect of, "hey, prefer asynchronous messaging over synchronous calls." And you were working on a part of the system and you were trying to follow the guidance that I had given. And I had likely miscommunicated it as "stop doing APIs and start doing messages." And so you were trying to follow my instruction and the system was getting more more and more and more complex. And eventually you're like, "help me understand how to do this while following the architectural guideline that you've laid out." At which point I realized that I had been very unclear in my communication. And in your attempt to do asynchronous messaging, you were reinventing some of the HTTP protocols, which is really cool. And also Also wholly unnecessary when we have things like HTTP request response.

[00:10:05] Allan Stewart: Yep.

[00:10:05] Dave Adsit: And so we upgraded our communication and we enhanced our system architecture and we simplified things substantially by using a more appropriate tool for the context that you were in.

[00:10:19] Allan Stewart: Right. And so this is a good example. You know, here we have a company culture, right? And that culture was such that you were in an architecture position. You had influence over the different teams. And we had had meetings and discussions about, hey, this is directionally where we want to go. And so we're building up a culture of how we want to interact. And there's cultural norms around, are you going to listen to those who are in authority or things like that? And that leads people to action, right? And in this case, we were taking actions that was contrary to what you had desired. Like the actual system architecture that you wanted to see was not the one getting built because we were doing something far more convoluted through a misunderstanding that was based on communication or perhaps the failure of communication.

[00:11:20] Dave Adsit: Communication failure on my part, I am certain. In retrospect, it seems obvious. I would say that when we have, so it's kind of more than that, right? Like the communication between the people and the communication between the systems and the architecture, if the communication patterns of the people are aligned with the communication in the software system, then things can be easy. Crazy. And when we have done that poorly, when they're misaligned, then we can make challenges that are unnecessary. And in this case, I mean, this is an example of that happening, right? But I think honestly, I think that what I've just explained and said is just Conway's Law with extra steps. Like I've just, I mean, it's the same thing, right? Like if the communication patterns are good, you're going to get a system that, well, whether the communication patterns are good or or not, you're going to get a system that kind of mirrors them. And so you want to align your communication patterns with your desired outcomes for the software system, the software architecture.

[00:12:22] Allan Stewart: Right. Yeah. I think that that's really, really important. And we can talk about it a little bit more, maybe after a few examples. A bit ago, you mentioned that culture is an emergent property. And we were talking about what even is software architecture. I think that there's a lot of emergent property behaviors there as well, right? Like, well, this is what it turned out to be. We would, we would love to make it intentional oftentimes, but, uh, but it's difficult because and, and part of that is this linking. So, so one example just to kick off some more, some additional examples, if you have a culture that is risk averse, then oftentimes those, those companies will put put checks or gates into place to prevent change from happening because we don't want bad change, right? We want to have everything approved. We want to make sure it's good. I remember a time that I was working at a company where I had a change that I needed to make. And so I had to go to the CAB meeting because at a certain time of year, you could only deploy things if you went to the CAB meeting because of the seasonality of the business. And so I went to the CAB meeting and it was very busy and they did not have time to hear my case. My plea went, uh, ignored, deferred until the next week. Well, what happens there?

[00:13:45] Dave Adsit: Well, inevitably you stopped working for a week because nothing you were going to do was approved yet. Right?

[00:13:52] Allan Stewart: Of course not. Um, especially for our, you know, lovely, uh, resource efficiency minded, uh, company, we had to stay busy. Um, Um, so we just kept making more and more changes. And so by the time that next week rolled around, we had twice as much change in, in the system that wanted to get deployed. Um, and so, so I think that, you know, this risk aversion, or, or at least the, that particular solution, right. A cultural solution. Hey, this is what we, as people, are going to do to put checks and balances into the system can lead to, like, bigger batches, which leads to now you have to have coordinated deployees. And it starts affecting your architecture because you're going to want to have fewer deployees, right? If you're working in a microservices environment and you have to get CAB approval for all of the microservices that you want to deploy, and maybe your team owns a dozen of them, it is unlikely that you're going to want to do that each and every time. And so you're going to just pull it all together. It gets approved in a big batch. So we're going to deploy it in a big batch.

[00:15:03] Dave Adsit: Yeah, definitely the case, right? And I think a lot about risk and what things we do to take on risk, what things we do to mitigate risk, what things, you know, there's, I'm always reminded of the H. L. Mencken quote, "for every complex problem, there is a solution, which is simple, obvious, and wrong." And I think that when it comes to avoiding risk in software, one of the things that we we try to do is stop deploying it or don't deploy it as often. And in my experience, that's exactly the wrong thing to do. It's like, "I feel weak, so I should probably not lift any weights because they make my arms hurt." Turns out if you go the opposite direction, you can actually meet the needs of the business. So this is an area where something that I would consider part of the the software architecture is how you deploy it. And so if we go from manual deploys with runbooks to automated deploys with tools, configurable source controlled tools, we've reduced the risk substantially of doing software deploys, especially if we've also built in rollbacks, right? So in this case, the deploy process can meet the needs and improve the overall culture, allowing us to reduce batch size, deliver more often, and still reduce the risk that we're worried about. I've worked at companies where we only did one release a year because we were worried about the risk. And we would basically every year provision a whole new set of hardware and stand up the system from scratch again in its entirety on the new system until we had thoroughly tested it for a couple of months. And then we would do effectively a DNS cutover for everyone, which is the biggest possible batch and the highest possible risk. We could cut back quickly once we identified a risk. And then all we would have to do is try to reconcile the millions of transactions that had occurred on the new system with the old system so that people could actually use it. But I mean, now we release multiple times a day using automated tools so that we can keep those batches very small so that the impact of any change is also very small, positive or negative.

[00:17:20] Allan Stewart: Right. And these are just like different, it is very cultural, right? Like this is how, this is the culture of the people and how they want to address a problem, right? So risk aversion might mean that you say, "oh, well, let's just, let's don't take risks. Let's not make changes until you've ultimately you're forced to do the big batch deploy." Or you can say, well, well, let's take a lot of really small changes that are very small risks, and we'll do them all the time and lower that batch size and say, oh, well, we're going to accept that problems happen, and we'd rather have them happen maybe more frequently, but they're going to be smaller.

[00:18:00] Dave Adsit: Right. And so the aversion to risks, it can be part of your people culture, but the software solution that you approach, that you use to approach it can be different based on your overall architecture.

[00:18:15] Allan Stewart: Right.

[00:18:16] Dave Adsit: And yeah, so I think, I mean, I think about this one a lot because there's a lot of things that we do that increase the risk of the system. So like that, you know, often risk aversion is a top down thing. Like the CEO doesn't like risk. The CTO doesn't like risk. The VP of engineering doesn't like risk. The head of product or accounting or sales doesn't like risk. So that's kind of top-down driven into the culture of the company. And related to that is your decision-making process. We talk a lot about how decisions are made. You mentioned the CAB, the Change Advisory Board, which is not actually an advisory board at all, but rather the Change Gateway Decision Board, who doesn't really like engineers anyway. Something to that effect. With veto powers. With veto powers, right? So if we talk about centralized decision-making, if we have a culture that values the "one throat to choke" perspective or mentality, one person makes the decisions for everyone. That leads to a certain type of software architecture and it's probably not one that has very much variety or nuance because it has to be simple enough for one person to keep it in their head as not even their full-time job, right? So we would, we would tend towards, what, standardization, common frameworks, common languages, you, a single language for the back end, single language for the front end, single toolkit for all deploys, all infrastructure is the same. Um, the types of things that facilitate making centralized decision making

[00:19:52] Allan Stewart: making easy. Yeah. Yeah. And I think it's like with software architectures, it's, there's not like a right or wrong answer here. Sometimes centralization can be very powerful. It can give you some scaling effects, especially people scaling effects where you know that there's better mobility across teams within your organization and things like that. But, but there's just, there's trade-offs, right? So if you're more, if you are more standardized, and you have a lot of central decision-making, you might not have that variety when you need it. On the other hand, if you go too far into decentralized territory and your decision-making is all decentralized, well, then it might just be a thinly-veiled anarchy where everything is just constantly in flux and you don't, and you never know. You know, a security patch comes along. Every security vulnerability you are going to have to patch because you've got every tech stack in the world on your boxes. So there are tradeoffs. And again, that goes back to this idea that software architecture is the hard things to change, but also difficult to choose because there isn't a right answer.

[00:21:13] Dave Adsit: Right. Right. As you were talking about this, I'm thinking about the book, The Cathedral and the Bazaar, and how, you know, even in the open source world, you'll have like a core team of the core maintainers of something who are making a lot of the hard decisions. Even when you have a bunch of people contributing around the edges, you may have, you know, 10 people in the core team and a thousand total contributors. Um, the core team is going to decide what languages and frameworks and toolkits you're using and everybody will just either get along with that or not. Um, but I think that there, there tends to be some level of centralized decision making, but depending on your culture, you may want to push as much decision making, authority, responsibility, and guidance as you can out to the edges so that that you can go fast. If you are, I mean, one of the challenges of having a single central authority for decision-making, you talked about it with the CAB, we've seen it with architects, right? If I don't know the answer, I have to go ask somebody and that person becomes a bottleneck. And if they've never thought about the problem that I'm asking them, they may have to go do research and then I can't get an answer. And then I'm blocked on taking the next step on my part of the system.

[00:22:30] Allan Stewart: Unless you go with the tried and true method. "This is how we've always done it. This is part of our culture. We know that this is already approved." Yes. And so you end up with an unintentional architecture.

[00:22:45] Dave Adsit: Right. Well, and even in that case, I've had problems in the past where I've said, "hey, we need a relational database for this system. What one should we use?" Because we could use, you know, any number, server, Oracle, MySQL, Postgres, SQL Server, I don't care. Just tell me which one to use. And the answer is like, "I don't know. Let me go do some research." I'm like, "well, we have some in production already, right? Which ones are they? And are we happy with the ones that are in production? Should we keep using the same ones?" Right? So then you end up with things like menus and those constrain your architecture. And sometimes your architecture is so chaotic that centralized decision-making does become the right answer to reduce the chaos. Well, I would say also one of the contexts in which I would think, let's have decentralized decision-making is if we're leaning heavily into training a bunch of novices. My understanding is this is one of the things that was at play when the mob programming practice was developed is there were a very small handful of people who'd been around and knew the system and a large number of novices or whether software novices in general, or just novices to the company. They came in and they needed a lot of guidance. And so we couldn't just say, "Hey, you guys have both been here for two weeks and both been programming for two years a piece. Why don't you go build this new invoicing system for us, right?" That's not likely to succeed. Anyway, there's a lot that I think about when it comes to decision-making and how that affects architecture and how the architecture affects what we desire in our decision-making process.

[00:24:33] Allan Stewart: Yeah. Another one that I think about is your culture around how do you work together? How do you collaborate, right? So going along along with that idea of mobbing or just, you know, in, in the traditional Agile parlance, are you cross-functional teams? Or do you have different silos of disciplines? So this over here, we've got the testers and over here, we've got the ops and over here, we've got the developers and, you know,

[00:25:01] Dave Adsit: everybody's segregated and backend.

[00:25:03] Allan Stewart: And yeah, exactly. Right. Like that, that's a, that's another great access, right? Are you a, Are you a front-end developer? Are you a back-end developer? Are you the DBA? If that's how you separate those things, or if you combine them into a cross-functional team, those are going to have different results in your software architecture. Because when you're cross-functional, then you're going to want to, it's going to incentivize team independence. "Well, we've already got all the people that we need on the team. We've got the DBA and we've got the DevOps person and we've got the UX designer. And so let's just go and build the thing and deploy it and be independent and not have to wait on everybody else." And that has a different architectural result than the other side of the spectrum where everybody is separated and you're working in silos and you want to collaborate in a very different way. Which often has a lot of throwing work over the wall to the next team.

[00:26:03] Dave Adsit: Right. Is our culture one where we collaborate and align along specialties or focuses, or is it one where we align around products? Are we building a product together, or are we a team of front-end developers that receive work from project managers and do work and then deliver it and blah, blah, blah? We don't know where it goes, nor do we care. Right. I think there's a, that affects both the architecture and the, I mean, classic example, front-end back-end. If you're going to separate your front-end and your back-end teams, you need a tool to allow them to talk to each other when they obviously don't want to. What is that tool? Everybody say it with me. GraphQL. Because GraphQL allows the back-end developers to build APIs without any care for how they're are consumed and GraphQL allows the front end developers to build UIs without any care for how the backend is built. And right. So you introduce a tool like that, which at scale requires you to run a bunch of extra infrastructure and get some expertise around that. You better have your Apollo Server up and running. All of those things have to happen. And that's a big impact on the architecture of the system in order to align it with the communication patterns and the structure of the people and the culture of the company, or at least the engineering department. One of the other things we, I mean, it's hard not to think specifically about the culture of engineering, but there's also like the culture of the company as a whole. One of the things that you find is that if you are creating silos like that, you can silo your data. And data is one of the most valuable things that modern companies have, right? Basically all your software can be copied, but nobody can necessarily get your data unless unless you unfortunately have a breach. But if you have your data and I need the data to build a good product, but you need the data to do business operations, business analytics for the company and run the company, well, now what do we do? What architecture have we built around data that either facilitates good biz ops or impedes biz ops? What have we done that facilitates good product development versus impede product development? And I've worked at a company where we literally had two people with the same title, head of data, and they worked in completely different operations. One was part of the product org and one was part of the finance org. And they were kind of at odds with one another. Like they both had different agendas and it created a lot of friction between departments because we didn't have a unified data architecture that worked well for all aspects of the company. And that's not to say that you should have one data pipeline. I would typically advocate against that. I don't want all of the data fed into a unified pipeline that is then used for steering the company and serving the customer needs. I would do probably different things than that.

[00:29:11] Allan Stewart: Yeah. Yeah. I've definitely seen outages when the analytics query hits the transactional database and then you have a bad time. And I've also seen problems where because report X that is owned by a different part of the company relies on this data, I'm sorry, you cannot change your database schema, even though it's not scaling. Right.

[00:29:36] Dave Adsit: Yeah. We've run into problems with the product, but we can't fix them because that would require a different team under a different leader in a different part of the country to make a change that they don't want to do.

[00:29:49] Allan Stewart: So one more example that I think about is the culture around how do you do testing, right? If you have a culture where you tell your developers that they have to test it, if they have to own their code in production, you're going to get a very different architectural result than the culture where you say developers develop code and then they hand it over to a testing team and probably start on the next thing while the testing team does their thing and you're throwing stuff back and forth over the wall.

[00:30:24] Dave Adsit: Well, and once the testing team has finally certified a build, they hand it to the ops team who actually runs it and knows nothing about the internals. And really at the end of the day, "hey, all the ops team can do is reboot it or roll it back. Don't know what else to do for you because we were not involved at any way, shape or form in the creation or maintenance or debugging or any understanding of this code." So yeah, that definitely, again, that is kind of a special case of the specialized teams, right? Across different skill sets, not just within, you know, all of the things necessary to build a software product, but the testing and running a software product. Testing is one of those things that I think has a huge impact on the culture of your development team. I think if you can get people into a mode of test first or better yet test driven, you're going to get a much better outcome when it comes to software. Agreed. Test first, because if I write my test, then I'll know when the code actually does what it's supposed to do. And test-driven, if I write the code, if I write the test, it will actually tell me what code I needed anyway.

[00:31:37] Allan Stewart: Yeah. And the cultural aspect of how do your humans act, if they're doing test-driven development, then it often leads to looser coupling and things like dependency injection, which affect the architecture of your system.

[00:31:53] Dave Adsit: Right. Well, and when you start having your developers do your testing, they do take more care and responsibility for what it is they're building, I think. Nobody wants to get woken up by an outage in the middle of the night. If you call a software developer when they have that kind of, devs hand it to QA, QA hand it to ops, if there's a problem then, don't even bother calling a developer. They're just going to tell you the same thing you were going to do anyway, which is reboot it because, I don't know. I don't know anything about the server that you're running it on. QA said it worked. That's their problem. Right? So, I mean, that improves the overall quality of your engineering culture when people are, you know, building more of a cross-functional approach to things, I think. But it also affects the things that you do within the code.

[00:32:46] Allan Stewart: I mean, I just put it all in a big in-memory table. So just throw some more RAM at it. But that's not my problem. That's your problem. There you go. Problem solved. Add more RAM. It's going to be super fast, you guys.

[00:32:58] Dave Adsit: I mean, we jest, but isn't that kind of sort of what that one database company did? They're like, "what if we put the entire database in memory? We'll just put racks and racks of DRAM in." And so instead of reading from disk, you've got everything right there in high-speed RAM.

[00:33:17] Allan Stewart: So one thing I think is really important to understand about this relationship between culture and architecture is that it is bidirectional. Right. So we've talked a lot about, well, the culture of how you do things influences the architecture that you build. Right. And that's that's following Conway's Law. But I've seen it work in reverse as well. Right. So if we if we bring out a couple of our standard examples, if you've got a monolithic code base, that's going to impact your culture. Right. If you've got multiple teams all in shared code and they can break or delay each other's releases, just because that's how it is architected, that is going to influence how people behave within the company. Same thing with the tech stack and the code. We talked about it as being a conscious decision, and sometimes that's true, but sometimes it's just, this is what we have we have a shared tech stack, and so we use that tech stack. We have some shared code, so we've affected our team structures around shared code. And I know, I know that's one thing that you've been working a lot, struggling with, with one of you, with your job right now. You've got some shared code in a monolithic system, but you want to figure out the right way to divide that up into teams.

[00:34:41] Dave Adsit: It's true. Yeah, and there's a lot of things that you can do there around, and we talk a lot about ownership models. Do you have a collaborative ownership model? Do you have a loose ownership model? Do you have a strong ownership model? All of these things have a big impact on how the system is built and the culture of the people building the system. One of the things I think about, because I've worked at a lot of companies that have done acquisitions or been acquired is sometimes you accidentally end up with a bunch of different tech stacks that are only loosely compatible. You can only integrate them through APIs, for example. I mean, I suppose Python and .NET and Node could all share the same database, but that's just madness. If you've already got three systems with three databases, have them communicate through APIs, right? So that is one of the things that you end up doing. And then you say, "okay, well, we've done this. What does that do to our culture?" Well, now we have the Python team and the .NET developers can't really work on the Python team because the .NET developers don't know Python. They might go over there and they might start to learn or they'd be slower at first. And the Python team might just need somebody who already knows Python and could go fast now. So now we're going to hire more people into the Python team. And now we've got kind of multiple subcultures within our culture.

[00:36:03] Allan Stewart: Right.

[00:36:04] Dave Adsit: And we've created communication barriers. And yeah, we have a Python guild where we talk about the Python things and we have a .NET guild where we talk about the .NET things. And we have a Node guild where we talk about mostly front-end things, but sometimes we remember Node. And all of these guilds are continuously isolating of the developers and creating more and more fractured islands lens within your overall engineering culture. And so now you have to make some intentional decisions. How are we okay with this? Is this a culture that we're comfortable with? Or is this a culture we want to change? And what are we going to do if we don't like this culture? What are we going to do to change the culture? Is the right place to start with making cultural changes? Or is the right place to start with making architectural changes that facilitate the cultural changes? And I mean, again, the answer is going to be, it depends because it always does.

[00:36:56] Allan Stewart: And always done.

[00:37:26] Dave Adsit: Right. So all of these things kind of interact with one another, and they, you are right, it's a bi-directional swirl, there's kind of, there's feedback cycles over and over and over where culture drives architecture, which drives culture, which drives architecture. Yeah, when you have a

[00:37:43] Allan Stewart: separated system or a distributed system like that, um, yeah, it changes how you look at things, right? If you're building, if you're intentionally building microservices because that's your architecture, well, you're going to be incentivized toward things like DevOps and making sure that it's easy to run these things, operationalizing them. But it also might mean that you collaborate with other teams by contract. Whereas before you might've gone and just sat down and talked with the other teams like, "hey, we have this bit of shared code, is it okay if I change this thing for what I'm doing?" Well, now we have to do it by contract and we say, "hey, we're changing our API contract. And so you're going to have to, you're going to have to adopt the new version because you have six months or whatever, you know, whatever your policies are." And, and, and those policies are human interaction things, right? The computer doesn't care what, what your policy is around when you do your API versioning, but the people do.

[00:38:45] Dave Adsit: Right. And that affects what things you have to put into your architecture to allow you to adapt to that kind of a changing landscape. How are we architecting in the small? Are we putting all of the APIs behind a gateway or an adapter so that when it changes, I only have to change one part of my code base? Or did I accidentally take a tight coupling to that remote API and I call it all over my code base directly? I've got curl commands everywhere. And if that's the case, well, now I'm going to have a worse time if that team changes. And so I'm going to use every cultural tool I have at my disposal to apply pressure to them not to change any of their code, which means that the overall architecture would tend to stagnate. Because I made a poor architectural choice, then I used cultural leverage to cause that bad architecture choice to permeate the system. And on the counterpoint, you can do the opposite, right? You can start making good architectural choices or good cultural choices. You can say, "hey, I'm going to shut down communication drafts in this area of my system so that the developers in this part of the system can spend more time focusing on architectural improvements that enable us to meet customer needs more quickly or deliver value sooner."

[00:40:15] Allan Stewart: Yeah. Yeah. Another example that I thought about is where do you focus? It's kind of related to what you're saying, right? If you're in a big monolithic structure, you might be more incentivized to think about the system as a whole. But then people tend to generalize and they might not get really good at a specific thing. But if you've got a distributed system where there's a lot of independence between teams, I've seen problems where teams will go off and they're like, "this is our thing, right?" We've established our identity around this microservice that we own. And so that's it. That's the thing that we're doing, even beyond the utility of that for the business. It's like, "well, but that's who we are." And so we're just going to keep on doing it instead of moving on to something else. Yeah. So, so thinking about how, how you set up that architecture can, can influence that culture, like you're saying. And, and if you, if you apply a new architecture, intentionally pick a new architecture because you want it to change the culture. Well, then they just call that the reverse Conway maneuver.

[00:41:21] Dave Adsit: That's right. That's right. And it's fantastic when you can get that right.

[00:41:26] Allan Stewart: It does require change though, right? And that cultural change might also include organizational change around things like how teams are organized and how teams are organized might impact how managers are organized and how departments are organized. And so like it can, it can be a big change. And that brings us, I think, to what you were talking about earlier about alignment. What happens when our culture and our architecture are misaligned?

[00:41:56] Dave Adsit: Well, not good things, typically. I mean, I would say typically that type of friction, it's like the impedance mismatch between object-oriented code and a relational database. Base. Like if we, if we are not intentional about, about it, then it can cause the kind of friction that causes more and more problems in the code. Because usually if the culture and the software are not aligned, there's a reason something has changed. You got, you got to the situation that you're in somehow, and then something changed and now you're out of alignment. Maybe you have, maybe you are trying to do an Agile transformation and everybody's gone through your Agile training and you're all like, "we're going to do this. We're going to get these one week iterations. We're going to do sprints and we're going to get stuff done." And, and then you realize that your code is actually very, very, very complicated. The mess that you've made, it doesn't, doesn't facilitate rapid iteration because you're testing, maybe your, your iteration cycle is one week and your testing cycle is two. That might cause a problem. On the other hand, you can take that that as an opportunity to make improvements, right? So the overall system might, the complexity of the system might drive you to pull out a small microservice and start altering culture in one area first. You know, we're going to keep the system and the culture as it is for most people, but we're going to start making changes and improvements over here in this area, right? There's all the, what was the Linda Rising books around Fearless Change. All of those things start to come into play when your culture and your architecture are out of alignment. Also, a lot of the concepts from Working Effectively with Legacy Code, right? You're like, "okay, well, assuming that the culture that we're trying for is the one we want, what do we need to do to our overall architecture to align with it?" Over the last two years, I've done this in two directions with my current company, my current team. Two years ago, we had a goal to grow the organization and to continue to deliver product at a faster and faster rate. And so we intentionally adopted a plan to move towards a microservices strategy, a microservices architecture. And as part of that, we intended to grow the team, split things up and have small teams working on dedicated components of the overall product. The last two years have been kind of rough overall. We're not where we thought we would be organizationally. And we've actually shrunk instead of growing. And so in order to keep delivering value at a rapid clip, we've reversed course. Probably six, eight months ago, we reversed course and said, "hey, our new strategy is simplicity. Our old strategy was microservices and our new strategy is simplicity." And we're going to align the system architecture to the needs of the business and the the architecture we have here or the culture that we have here. The culture requires that we continue delivering so that we can continue meeting customer needs and growing, but we don't have the staff or the capacity to invest in decomposing this monolith. In fact, we're going to fold in some of the things that we had previously extracted. We've got a couple of Node services that are being folded back into our PHP monolith. We, in fact, even stood up a Go microservice that we are going to be folding back into our PHP monolith so that we can leverage simplicity on the architecture and deliver value for the company, which is very much aligned with our overall culture as a company and in engineering.

[00:45:57] Allan Stewart: I think there's plenty of stories of companies wanting to adopt some kind of a change. Especially big architectural change, right? So I want to say like 10 years ago, lots of people, lots of hype around microservices and people would try it out. It's like, "oh yeah, we're changing. We're changing our architecture," but they didn't change any of the organizational structure. They didn't change their culture of how they do things between the people. And then the transition fails, right? Right. It's the same kind of thing like the like the Agile transformation failure that you were mentioning before, because these things, they don't happen in isolation. If you if you have an architecture and a culture that are at odds with each other, then in the best case, you might end up with a distributed monolith. And that's not a very good best case. But in the worst case, it's just, it doesn't work. And one of the things is going to fail.

[00:46:55] Dave Adsit: Well, and I often think that a microservices transition is a technical solution to a people problem. We're having trouble communicating. Our culture is not working effectively for our goals. So we're having trouble communicating. So we're going to create smaller groups that can be more focused and communicate with each other through the contracts that we talked about earlier. And so that is, I mean, again, that is very much the thesis of this conversation, of this podcast episode, is culture and architecture are intrinsically linked. And you have to figure out how they drive each other and what you want, what you want from both. What do you want from your software architecture? What do you want from your engineering culture, your company culture? And how can you align them in such a way that they cause improvement across that boundary versus just kind of falling into some of the common traps that we see?