Crafting Code Podcast

[00:00:16] Allan Stewart: Welcome to episode 16 of the Crafting Code podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately I've been thinking about re-watching Star Trek The Next Generation.

[00:00:31] Dave Adsit: I'm Dave Adsit, a technologist, and recently I have been thinking about effective product development teams and startup cultures.

[00:00:41] Matt Baker: My name is Matt Baker. These days I just think of myself as someone who tries to write computer code.

[00:00:48] Allan Stewart: Before we get into our topic today, we wanted to give a big thanks to Todd Fisher for our new intro music, which debuted in our previous episode. Our topic today is architecture examples. It's really easy to talk about architecture in very vague terms in order to be general enough to make it broadly applicable. We do this sometimes in this podcast, it happens in conference talks, etc. But unfortunately, that's a lossy process. Without understanding the context of a system, the people involved, a lot of those important details, the architectures, are lost. So in this episode, we're going to talk about some specific architecture examples, situations that we've experienced where we've had fit and unfit for purpose architectures. We'll try to provide some context on why it worked out or why it didn't. So Dave, do you want to kick us off?

[00:01:42] Dave Adsit: Yeah. So the first one that I want to talk about is back in 2003, I worked on a software application for managing inventory in small restaurants. This was an add-on service that we gave to our customers that allowed them to keep better track of food costs, etc. et cetera, and do a better job ordering. Of course, the company I worked for, our interest was in getting people to order a lot of things from us. The system that I built ran on a desktop computer and it ran on a Handspring Visor. And that may or may not mean anything to you. The system that we had communicated with our systems through FTP. So the VB6 application would dial up possibly to the internet and upload files to an FTP site, download files from an FTP site, etc. et cetera. And then the user in the restaurant would plug in the Visor and it would synchronize things to the handheld. And then they could go around the restaurant and scan in their inventory and use their current inventory as input into creating a new order to place with us as a food provider. All of that was pretty revolutionary for the time and has all been superseded by the amazing cell phones that we all carry around these days, even in restaurants. The thing that was interesting about this project, the thing that was a constraint that led to an architectural decision is the fact that in 2003, most restaurants of the scale that we were serving only had one computer. And that computer typically was purchased to run the point of sale system. And the point of sale systems that most of these small restaurants were using were running on Microsoft SQL Server Developer Edition or Micro Edition. I can't remember the exact name of the edition, but it was a little tiny SQL Server. And it was single instance, distance, which meant that you couldn't run a second one on the same machine and you couldn't run a second database in the same database engine. And you couldn't use this database because if you were to touch it and break the point of sale system, that could cost the entire account. So we made an architectural decision early on to use Microsoft Access databases or Jet databases or whatever you want to call them so that we could actually have something resembling a relational database to store all of this inventory information. And that database ended up being a single flat file on the system. And basically that was the decision that we made that allowed us to move forward with developing the product. I'm sure that a lot of people have a lot of opinions about using Access Databases, Jet Databases, or whatever to store data locally. A lot of those opinions are probably right. But if you think about the constraints that we had on on our system, using an Access database was certainly a better option than maintaining a whole bunch of flat files or XML files, which the JET engine actually did that for you. It would manage all those things for you. And then it would allow you to write actual SQL statements to query that data, which became critical when we started creating custom reporting inside the app for the the users. That is an example from my perspective of making an architectural trade-off based on the constraints that otherwise would have prevented you from building the system at all. There were some knock-on effects, some positive consequences to this that were helpful. One of which is that if a user ran into trouble with their system, we were able to implement a a menu item in the app that would take a copy of their database and upload it to our FTP server, pause their app until I, as the sole developer, had been able to download it, run the app on their version in debug mode and find exactly why it was breaking, fix the file, fix the data in their database, upload a new copy back to the FTP server, at which point they could click the resume button and it would download the file and they'd be like, "oh, you fixed it. Everything's magical. This is great." And it was certainly a lot of work, but it actually was a nice unintended consequence of that architectural decision.

[00:06:31] Allan Stewart: I hope that that was before SLAs became all the rage.

[00:06:35] Dave Adsit: Yeah. Usually these were like live phone calls with the single end user at the small restaurant where they had been panicking about their inventory and being unable to deliver an accurate order to their salesperson. And so they would sit on the phone with me while I was in VB, in debug mode, fixing their database and possibly fixing the code and preparing a patch to upload that would then fix the problem for them. So the SLA was not a consideration. Also, Also, this isn't necessarily a strategy that scales beyond a few hundred active customers.

[00:07:15] Allan Stewart: So one architecture that I participated in that's a little bit more modern was a microservices architecture where we were publishing events. We were using RabbitMQ in order to signal other services when there were changes in our system. This worked out really well, but one of the things that we didn't really understand or or observe early on in the system was the different ways that people can handle events. This came up in a couple of different ways, but particularly there's an article by Martin Fowler on his website. It's entitled, "What Do You Mean by Event-Driven?" And in that, he distinguishes between event notification, where you send an event and say, "something happened." Here is a minimal amount of data, generally just some identifiers to let you know which thing it happened to in the system, which entity had changed or was added or deleted or whatever. And then the other type was event carried state transfer. And in that you send more than the identifiers, you're sending all of the related data about the change. So if there's a new user added to your system, you don't just say new user with this ID, you say, "here's the new user and And here's their name and here's their email address. And here's whatever else that is relevant to share with the rest of the system." And overall, I felt like this messaging architecture worked really well for us. But then we started running into problems where some teams were having item potency issues because they were listening to event carried state, taking that data and saving it into their database. And that was really convenient because they didn't have to make any API calls. Everything could be handled out of band, but they ran into item potency issues where messages would get replayed, or if they came out of order for some reason. If there were a lot of high traffic messages, it could really cause some problems. And meanwhile, there were other teams that were having a different set of problems because they were just looking at the IDs of things and making API calls. This incongruency between the two ways of doing message notification really caused some confusion in various parts of the system, depending on which team you're working with and which pattern that they were attempting to use. And the patterns that the other teams that they were interacting with happened to be using.

[00:09:37] Dave Adsit: I remember working in this system and I was one of the biggest champions of leaning hard into event-carried state transfer. It certainly does work in theory, provided that you have perfect hardware and no message failures. Even out of orders can be handled by things like timestamps or offsets, or if you get more detailed, each message can have its own monotonically increasing ID. So you know the messages. If you get messages out of order, you can know to ignore them. But man, it certainly does cause problems if you are working on different paradigms. First of all, if you've got different expectations about what messages are doing, it causes even more problems when you have to deploy one of these systems onto actual physical hardware somewhere. And you realize that not all of the systems are up all the time and not all of the messages even get received every time. And there's all kinds of data transfer issues.

[00:10:43] Allan Stewart: Yeah. And ultimately, I feel like either pattern could be very successful. The architectural issue that we had was kind of an accidental architecture. We talked a lot about how we were going to do events, but we just didn't know how important this distinction was. And so, so many teams were just doing it in different ways. And it really exploded the number of patterns that were going on inside the system and limited our ability to share strategies across teams. Until we realized what the problem was and could start thinking about how we wanted to address it.

[00:11:19] Dave Adsit: One of the things that this makes me think about is just the general concept of a data-intensive application where you have multiple data sources and providers working together with transfer mechanisms, et cetera. And just what a complex problem that becomes in production or with any Any kind of scale or load or concurrency.

[00:11:45] Matt Baker: Okay. So an architecture I want to talk about is one that maybe doesn't get as much airtime in the thick O'Reilly books, but still I think super valuable. I don't know the name for it. I'm going to call it trying to find product market fit architecture. This is the type of architecture I reached for when helping a startup who was trying to find product market fit, had a very short runway, and needed to deliver quickly and often in order to test ideas. They had a few pilot customers, and they had constant access to them. They anticipated that they needed to iterate on a mobile app they were building for their customers quickly. So the situation, they would make a change, go out into the field with the customer and then see how it worked. And they wanted to do that a lot. They didn't have any investment, bootstrapped and very thin. So I'm saying that just to underscore that there was no fluff. Every dollar needed to be spent well. So we looked at the situation, and there was nothing on day one, and they needed a mobile app. We quickly converged on Kotlin for the language, just an Android mobile app. And we on day one also built a CICD pipeline, and it took maybe be six or seven hours to build that CICD pipeline. We did it in Jenkins just on a hosted instance. And then we got to work. So right away, we abandoned a ton of practices that I would do at a more established company. We didn't do TDD. We never talked about a branching strategy until we had to release multiple versions of the app to different contexts, like this one's for test, and this one's a production release. And then we started branching to handle that. We didn't write a team agreement. We didn't do a lot of things the more established company you would do, right? And that was okay. We quickly keyed in on the fact that we needed to minimize any non-differentiating work. And so I want to give a few examples of how we did that in order to really respect the startup's bank account, which was again, just super slim. And if they had a chance, we wanted as much time as we could to test that idea. So we chose Firebase to deploy our Android app to the app store. It made interfacing with Google just easy. So we reached for Firebase. And at some point in the app process, we decided we needed analytics. And it wasn't a conversation of like, "okay, what analytics provider we should use?" The conversation went like this. "Does Firebase offer analytics? Yes, use it." And it wasn't like a use it in like a being told what to do kind of way. But yes, we all agree. Let's use the analytics that's closest to where we're currently at because it's the best way to minimize non-differentiating work. And that theme just threaded through all of our choices. So at some point we needed an identity solution. We needed to add user accounts to the application. So we used the identity solution that Firebase offered. We didn't go out and compare other tools and offerings. We didn't even consider other things. We just said, "no, we know our strategy here. Our architecture is one of the key tenants of it is minimizing non-differentiating work. And that drives our tech solution or our tech choices." So after that one, at some point we realized we needed a state store. We kind of hit the point in the app process where we needed persist state, and same thing, just reach for Firebase. This sounds like an advertisement for Firebase. It's not, but it happened to work in this case. But the thing I want to highlight is it was an architecture. We were very intentional in what we did. We didn't just accidentally make choices. We understood that the money was tight. Now was not a time to flex interesting tech or try and reach for a new tool that might be fun to play with. Now was the time to ship and ship a lot. And so we chose tools that optimized for that and nothing else. And I would call that an architecture. And I think it's, like I said, the trying to find a product market fit architecture. And the reason I led with this particular example is I think it kicks the door open a little bit on what an architecture is. And when you talk about, "well, what's this company's architecture?" Sometimes we might say, "well, that company doesn't really have an architecture. It's just nonsense." And I kind of take issue with it. I'm not saying that any of you have said that, but I kind of take issue with that because it's there. Care. No, they've made choices just like I talked about. That has occurred a lot of times at that company to get to where they're at. The architecture is there. It may be unclear. It may have drifted, or I don't know. I guess we could theorize why some places look like they have architectures and some don't. But sometimes an architecture looks like just trying to save as much money and time as you can.

[00:16:05] Allan Stewart: Yeah. I like that concept about the multiple facets of architecture. One of the facets is managing change and helping understand what things are going to change and and how you're going to segregate different parts of the app based on change. And so CI, CD becomes part of your architecture. We don't often think about that. It often is thought about as like, "oh, well, that's just part of our deployment pipeline." Sometimes we'll think about deployment architectures in terms of like, "oh, is it a monolith? Is it a microservices? Is it something out some end tier application?" But really that concept of how quickly can you get changes out into production informs other choices that you make within the system. It affects how you're going to write your code. It affects how you're going to manage releases and deployments and such.

[00:16:54] Dave Adsit: You got me thinking a lot about the rapid application development toolkits that used to be super, super popular and now may not be, I don't know. Sometimes they get lumped into the demo ware type of toolkit. Kit. And I have often been at the forefront of the conversation, bashing on anything that allows you to build quickly and maintain poorly. But what do we always say? "Context is king," right? And the context that you are sharing, it really does matter that you can iterate quickly. You probably ran into some spaces where you're like, "okay, throw that away, rebuild it because it's It's easier to build it again than it is to maintain it or iterate it or refactor it from where it is." I think we've all experienced that. I worked with one QA automation tester who every iteration would go in and rerecord all of his tests using the test recording tool versus maintaining the scripts that were used. And he's like, "well, I can record all the tests in about two hours and it would take take me all week to make these changes manually to the existing test code. Plus I can see where the app is failing when the tests don't pass. And I am now familiar with it." I think that's something critical for us to remember is that in certain contexts, certain practices are essential. And in other contexts, those practices become an anchor that sinks the entire team or company or product or whatever.

[00:18:35] Matt Baker: Yeah. So to speak concretely here, I am definitely a bit of a TDD zealot. That's because I've just seen what it's done to the way I deliver code. And I assume that it would do the same thing for everyone else. Probably not true, but, like, it was so profound for me, the way it changed that I, the way I write, write code changed after adopting TDD. It's almost like, "when would you not do this? This is crazy." It's great, but it does slow you down. And I think that even the biggest advocates say that, that yes, you're trading in some delivery up front for sustained delivery in the long haul. And I would just call out going back to what you were saying, Dave, we didn't want sustained delivery in the long haul. We were, the idea for the app we were writing was changing every day. So I was like, well, the last one, and Dave's right, we would throw away iterations of the app, easy peasy. And we could do that because we were trying to minimize the choices that married us to that particular revision. And so we didn't reach for TDD. And did we suffer some quality? Absolutely. Like we had some bugs, we had some production bugs that we had to deal with, but the customer count was so low, you know, it was okay. And at the end of the day, we accomplished our ultimate goal, which was delivery. So I guess in summary, Dave, I'm just trying to offer a concrete with what you're saying. You know, you should typically do TDD. If anyone asked me, "should I do TDD?" I'll say yes, without even asking like, "well, it depends." Like I'll just say yes. No context needed.

[00:19:55] Dave Adsit: Yes. This is the context free answer. Yes, you do need to do TDD. What if I'm throwing away my entire app every two hours and starting from scratch? Well, okay. Maybe not in that case.

[00:20:11] Allan Stewart: Going back in time for my next example, about, I don't know, 2010, 2011, I was working, doing a lot of rich internet applications, which meant writing ActionScript code for Flash and Flex. And I was working on a project where we were delivering educational templates, so we wanted to have a lot of interactivity with different kinds of templates, and so there was like a player object object that would manage basically a lesson. We're going to teach US history, or we're going to teach math or English or a social studies, something. And then the player could load up a whole bunch of different templates. And some templates were really simple. And all they did was show text with a picture. There were other templates that were multiple choice or drag and drop or other kinds of interactivity that you could do to try and improve the engagement and improve the learning outcomes. When I was first working in the code base, I had come on to an existing code base. It was kind of a mess. There was a lot of code just splattered all over the place. Templates intermixed all of their concerns. State management was intermixed with the user interface, was intermixed with receiving events from the mouse or keyboard. It was just hard to get anything working. And every time that they did a new course, they wanted to reskin things slightly, basically use the same engine, but now with some new changes. Given the state of the code, it was really hard to make that happen. But through a number of circumstances, I ended up being the sole developer on the project. There had been like maybe four developers originally, originally, and I was the one who got the honor of continuing to work on it. One of the things that I was learning at that time was unit testing. And I was also learning a lot about patterns. And so I decided to make some architectural choices around applying the MVC pattern to break up these different concerns. So we'll have more of a model to represent here is the data that matters for this template. Here's a controller. It interacts with the user. Mouse events, keyboard events, whatever, they go to this controller and it will make the appropriate change to the model. And then the view will just display things. It made my unit testing go a lot better. I was still learning how to unit test. I didn't know how to write very good unit tests, but unit testing the models and the controllers became very easy because it just had to do a particular function based on whatever the current state was. And then the views were basically impossible to test and I just ignored that. And so as they started having all these different templates, I felt like this architecture became really powerful. I was able to work very quickly, get a lot of work done, made the product owners very happy with what we were doing because Because each new template, especially when they would make minor view revisions on the template, I would just keep 90% of the controller and model intact and just replace the view. And so very quickly, things were getting done. I may have gone a little bit overboard, though. I think, Matt, you've talked about in the past about how when you learned about the decorator pattern, it's just like, "now everything is decorated." Well, I kind of did the same thing here with Model-View-Controller. It's just like, "okay, every template has Model-View-Controller." And it's like fractal because the player itself has Model-View-Controller and all these different components that go on the player, like in order to do like page navigation, they were all using Model-View-Controller, and they were bubbling up events through, not the DOM because it wasn't HTML, but essentially the same kind of a concept. I thought it was great. And we were doing unit testing and things were really good. But underscoring the importance of people, we got to a point where they wanted me to transition to a different product. And now I had to teach another person how to maintain this code that I had written. And I was like, "Oh, well, this is awesome. And it's easy and it's simple. It's so flexible. You can do all these things." And the other developer just was totally lost, had no, it was so foreign to how they were used to writing code that the architecture, which was in the context of Allan is writing this code all by himself. Great. Really fit for purpose. You make that one human element change. And now all of a sudden it was not fit for purpose anymore.

[00:24:48] Matt Baker: What would you call it? Allan, like, the hard to transfer architecture, like, hard to leave your job architecture.

[00:24:55] Allan Stewart: I don't know, but I'm really excited to hear in a conference talk from somebody, the trying to find market product fit, product market for architecture. Once I start hearing that in the wild, then I know that you will.

[00:25:08] Matt Baker: Toss credit back my way. If you hear it, don't let them steal it. Dave, I feel like hits this point a lot about architecture being a people problem. And I don't mean to say that it's in its entirety, it's a people problem, but it's highlighted for me in the story you just told, Allan. And it definitely reminds me of similar similar missteps I've made. I have to imagine when you were writing it, the points of extensibility, I'm assuming the points of extensibility made it a little more complex than it needed to be, or like the generalizing, you know?

[00:25:45] Allan Stewart: Yeah. Is that right? Yeah, I think it probably did.

[00:25:48] Matt Baker: I assume when you were writing those, the intention was to add value to the company. Like, I think you saw, "hey, this thing's going to change in these ways." So I'm going to add these points of extensibility here or maybe even just leave the scaffolding for it, you know, if they need it. But for sure, and I know when I've made those choices in hindsight, I look back and I just say it was just noise. Like, I, I get why I did it, but I think it, like, given the chance to do it again, I would say just do it the, the dead simple way. And then if it sucks enough to do it the dead simple way, you'll probably know where you should make it extensible or configurable. Like it'll it'll announce itself. It's a hard thing to guess where it's going to show up.

[00:26:29] Allan Stewart: Yeah. And I think there was definitely some of that going on in that code base, but also there was just a kind of disconnect between what was important to me and how I was working and how the other developers around me in the same company were working. I think going back to that idea of the socio-technical system, what I had created wasn't bad. I think that there's a a lot of similarities when we look at some modern, like, especially like front end architectures today, you might have like a React app with Redux. We don't think about everything being in terms of model view controller, because it's not exactly the pattern that we're using, especially when you go back and, like, look at the Gang of Four book, or you're doing like a UML diagram, but there's a lot of similarities there as far as like, "Oh, well, we've got a store. That's kind of like where your models live." You've got, you know, you might be doing things more functional now instead of really state-based, but you've got state in there somewhere. How do you interact with the different events that are happening on the screen and so forth? There's a lot of similarities there. And so I don't think it was a, it wasn't a bad architecture as far as like creating all kinds of things that were unnecessary, but just the pattern was different.

[00:27:42] Dave Adsit: You didn't dump out the whole Gang of Four books on top of it.

[00:27:45] Allan Stewart: Well, you know, like I said, fractal MVC for the win, but it was, it was so different from how the other developers thought about working that we were just basically incompatible. I couldn't, couldn't, wouldn't work on their projects. And if, if I had to, I would rewrite huge chunks of it because I was working in a different kind of paradigm than they were. And if they worked on my projects, then they got lost because they're like, "well, where is everything? And I don't understand what's going on."

[00:28:13] Matt Baker: It strikes me that whether or not you were right and they were wrong, you're still going to have that same problem. Yeah, there's something there about, like, because I know you, you're smart, and chances are you probably had a better solution than that and you still had the problem. And so, yeah, I don't know when I say that, what does that say back to you? What does that highlight? Just the

[00:28:33] Allan Stewart: being right almost never mattered. You can be right about something and it doesn't make a difference to the outcome of the company. We also see lots of cases where people do things that we look at and like, "okay, well, when we really critique this, we're going to call this wrong." And yet the company is super successful. And that wasn't a differentiating thing as far as the outcomes for the company. So we have to really be careful about placing that value judgment.

[00:29:06] Dave Adsit: This reminds me of another project that you and I worked on together, Allan, where the two of us were doing something similar. We had a way that we wanted to develop this application, and we were introducing concepts like message bus and queuing as a way of managing out of band, out of process workloads that didn't need to be handled immediately in the request response flow for the website. And we were introducing all kinds of interesting tooling and patterns. And I remember looking at the, basically when we were drawing the diagram, the architectural diagram up on the whiteboard and talking to the developers we had on the team at the time. It was all going well over their heads. And that wasn't their problem. That wasn't their fault. That was us looking at interesting technology versus the essential socio-technical system, right? Right. You were looking at how we would have solved the problem, just the two of us, and not taking into account the fact that there were also five or six other developers who had no experience and no training, and honestly, no ability at that time to contribute to the system that we were building. We were basically unintentionally working them out of a job by making a system too complex for them to work on. And I think that that is a failure mode that I have fallen into multiple times as an architect, is expecting people to come along and keep up with ideas that I've struggled with or worked on or tried to learn about for five plus years. And now I'm ready to finally implement it in a system. I'm like, "hey, I know you've only got two years of experience since college, but let me tell you how we're going to do this event-driven microservices system with a Vint carried state transfer and data replication. And it's going to be awesome. You're going to love it." And they really no context, no concept of what you're doing. It reminds me of the first time I did a model view presenter app as a website, .NET, ASP.NET website. We did the model view presenter pattern so that we could introduce test-driven development into our presenter layer. Honestly, I had no idea what we were doing. For probably six months, people would say, "type this code here." And I'd be like, "okay." And I didn't know why. If we weren't pairing, I would have contributed nothing to the team during that period until those patterns actually finally started to click and inversion of control finally started to click and even interfaces. Like, honestly, I worked for a good eight years before any of the code that I worked on, any of the code bases I ever worked on even had a concept of an interface. We wrote a lot of code. We shipped a lot of value to a lot of people. I don't know. There's reasons to do different things. And the people have to be an essential concern for any skilled architect.

[00:32:20] Allan Stewart: detect. To give us some credit back, I will say that maybe we didn't fully recognize the extent of that happening. But we did recognize it a little bit, and we made adjustments to what we were doing. We simplified certain things. It's like, "well, if we were doing it, just the two of us, we would do it like this, but because of the developers that we had, we're going to do this other way instead." And we also tried to make changes around the environment too, instead of just dragging them along, help them understand the purpose of what we're trying to do and help level them up. How successful that was is a whole nother story, but I do believe that we recognized it at least partially.

[00:33:03] Dave Adsit: One of the failure modes for anyone skilled in a thing, and software architects being no exception, is underestimating the amount of work it takes to to level up your knowledge in esoteric concepts. It's hard. Learning is hard. How many of us have worked at a company that focuses on learning? Oh, I know all three of us have multiple times.

[00:33:34] Matt Baker: Don't out me. Don't out me. One day I hope to learn something. Oh, one thing that was coming to my mind as you were talking, um, and something that I honestly haven't appreciated until at this point in my career. I looked the other day, I've been paid to write software for 18 years. So to scope what I'm about to say. So it took me about this long, maybe a little bit. I'd like to claim it was a couple of years ago. Maybe it wasn't, I don't know. But it took me a while to understand that the level of expertise you hire is a business strategic choice, a strategic business choice. That's a better way to say that. It. And that has all the impact on what you were talking about, Dave, from where I'm sitting. Like, if you, if you're designing a system that is complex and hard, I was going to say hard to work and I don't think that's right because it's not hard to work. And if you understand, like, an eventing system, I don't think those are hard to work in myself. Some time ago at some point in my career, yes, they were, they were hard to work on for me. And so I guess what I'm inching towards is if it's hard for me to work on your system, I might not be the right hire for your company. And if your company is positioning themselves to hire people at my salary band and experience, because it's the right fit for what they're trying to build, you're going to run into some rough spots when the system can't be adopted and owned by these people that they want to hire. And so you're almost forcing the company to hire more expensive people. And so it just all comes comes back to, Dave in particular says this all the time, like, architecture is people problem. It just kind of comes back to that for me. Like, when you're designing this system, and this isn't something that I've ever heard an architect, like, sit down and think about, and that's kind of the nuance that I'm trying to highlight. It's like, it took me a long time to catch this point. Like, I don't see it talked about at conferences and, and other things, but it's real. You need to think about the people that are going to be working in the system you're designing, and you do need to consider their anticipated skill level. And like, what a, like, subtle nuance that comes out after you just, like, go at it a lot of times. I think you're not going to pick up on this. I don't think in the first one or two systems that you're the architect for, but in time, I think it hits you in the face.

[00:35:50] Dave Adsit: If you don't notice it beforehand, that is a very, very interesting point. I think that I've I've heard that referred to as the habitability of the system, that the people who live and work there have to be able to live and work there. And I've usually heard that in terms of using esoteric programming languages and extraordinary complex toolkits or patterns. But this applies at all levels of complexity for a system. If you know that you're going to be hiring a bunch of developers right out of school, and you're probably only going to keep them for a few years before they have enough skill that they go get a better, higher paying job on more complex problems or more interesting problems or whatever, then maybe you should be considering the advantages of having a monolithic system with maybe some well-defined layers or some rules in the system that you can use. You and I might chafe at with more experience, but they make the system habitable for the people who are there. I recently, speaking of architectural decisions, my youngest recently turned five and I went through my kitchen and aggressively removed all of the child locks from the cabinets after years of frustration, looking for that magnet that opens the cupboard that has my Tupperware in it. So why were you locking up your Tupperware?

[00:37:28] Matt Baker: Just every cabinet?

[00:37:29] Dave Adsit: Every cabinet had locks on it. Every lower cabinet had locks on it because the thing about two and three year olds is that they don't believe in cabinets. If you can't see it, it might not exist. So everything in every lower cabinet must be pulled out onto the floor on a a regular basis so that you can do inventory and make sure that you still have the pot that you can bang on and all of the lids are appropriately stored far, far away from the glass dishes that they belong to, etc. I can confirm this is a true thing, and at a certain age those things become significantly less interesting than playing Minecraft or whatever. So you don't need to lock up the plastic dishes.

[00:38:20] Allan Stewart: Going back to what you were saying, Dave, I worked in a system at one point where one of the architects was really trying to templatize a lot of stuff. "This is how you do it. You are going to work in this framework." And for me at the time, I really chafed at that. And I was like, "this is stupid. We should be building it up from first principles every time." And now I'm starting to realize, oh, given the context of that particular company, that actually might've been a really smart decision to push a lot of that stuff into templates. And this particular framework is just the way that we're going to do it because it's not the best, but it's consistent.

[00:38:58] Matt Baker: Yeah. It's almost like, look, is it delivering? Like then and stop the conversation kind of thing, right? Like that's almost how blunt it has to become. Like it's not that like we couldn't change it in fantastic ways. It's just that we don't need to. We're going to templatize this thing because, and I wasn't there, so I'm out on a limb, but we're going to templatize this thing because we don't care about it. Like we're not looking to innovate on it. It's just a necessary evil that we have to deal with in order to get to our ultimate goal.

[00:39:26] Dave Adsit: So Matt, you've often told me that sometimes the answer to a question in software is "go develop software for 20 years and then come back to me." I'm really excited to hear the answers to the software questions in a year or two.

[00:39:46] Matt Baker: The jig is up. The real, yeah. It'll probably be some answer about like the more you do something, the less, the more you learn, the more you learn about something, the less, the more you realize the less you know, or what, there's a saying there that I just can't get, as long as you don't come up and

[00:40:08] Dave Adsit: say "42" because that won't help me at all. What if it's the binary representation of 42? Oh, it'll take me a while to figure that out, and then I will send you a rickroll, or I'll feel as though you have sent me a rickroll. Okay, all right, so what's the next architectural experience that we have had?

[00:40:33] Matt Baker: Yeah, I've got one on deck here, um, I'm calling this one the smart-ish pipe architecture, the reference being to smart pipes, done pipes. I'm not going to go into that pattern, but there's lots of arguments about it on the internet, and this architecture I feel like cut up the middle in in an interesting way. And so I called it smart-ish pipe. The TLDR on the architecture, big monolith broken out into a distributed system. Surprise, surprise. Everyone's heard this, right? The interesting part though, is the integration between those subsystems that were broken out. In this particular case, they chose WCF. It's a Microsoft technology that allows you to write. I'm going to misrepresent WCF and I'm sure someone who knows it better is going to say, "You don't know what you're talking about." But what my experience was is that you create interfaces that are very easily exposed over TCP. And you can invoke them. Sometimes you can invoke them in a way that seems like you're invoking on your local machine, but you're not. It's a remote procedure call. And other times it's obvious that you're calling across the wire. But they handle things like I said. They'll expose the operations for you in a nice way in typed languages like C#. They also would handle GU that oftentimes you don't think about, like network retries. They just got it out of the box, three attempts over a TCP connection, and then it would fail, and that was configurable, and just stuff like that. When I got there, the system was already mature when I arrived, and I spent some time with the architect over the, I think I was there like a year or two in my time there and one of the things, one of the conversations I remember having with him, was about his choice of WCF, and I, I was harping on it. I told him, "you should have just paid HTTP as the lowest common denominator and what allowed for, like, uh, some, some variants when people wanted to integrate subsystems. But in this case we all have to use WCF, ergo we ought to use .NET." He, so I pissed him off when I said it. I didn't mean to, but it, you know how it goes. So he, he just kind of in a quick manner, like a bursty fire manner, said like, "well, here's all the reasons," and listed off all these reasons to back up his rationale. And then he was like, "do you have answers to those?" And it's like, "oh no, I don't." Um, and then we, he kind of walked off. But, uh, I appreciate what he did though because he, he highlighted for me, like, as I thought about it, that I didn't even, like, lead with, "why did you make these choices?" I just immediately condemned and went, you know, "whatever, you chose the wrong thing, HTTP, because dumb pipes, right?" So I thought that it almost like a, from a purist perspective, when you connect subsystems in a larger system, you do it with pipes that don't know anything about anything other than they are responsible for carrying data across the wire. That's what they know. They know where they receive the data. They know where they drop it off. And that's it. And so the smartish pipes is a little bit up from that, right? Like it, the pipes became a little bit more more interesting. They had some fault tolerance in them. They understood what a .NET interface was, and they knew how to take calls to that interface from one remote computer and patch it over to another. And it worked. And he was right. And they never had any problems. They're still a very successful company, growing quickly. People that have stock there are very happy with it. And they'll probably continue to be that way. And there's no reason for them to dumb down their pipes in this particular case. So I would call it a win. I call it an effective architecture choice. And like I was saying earlier, who cares what you think? It works and they're rich. And it's the smartest pipes architecture.

[00:44:16] Dave Adsit: I think one of the things that you mentioned there kind of in passing that I think is worth diving into is that by making that choice, a constraint was reified. Basically your system had a constraint added to it that you had to use .NET because .NET is the only thing that can effectively communicate through WCF, which for those who are unfamiliar is the Windows Communication Foundation, which is why you had to have .NET at the time. Sometimes a decision is made accidentally or incidentally to another decision. And sometimes a decision is made that that reinforces an existing decision. And it sounded like at the time, you probably were chafing at the fact that you had to use a .NET language in order to build components for that system.

[00:45:06] Matt Baker: That's right.

[00:45:07] Dave Adsit: And I suspect that that was intentional, at least on some level.

[00:45:12] Matt Baker: I think you're right.

[00:45:14] Dave Adsit: Reinforcing the requirement that components of this system are written in .NET. And that goes back to the sociotechnical system, right? That sometimes we want to put a constraint on because that actually allows us to facilitate other types of operations like moving people between teams or between projects because it's all .NET. I have personally worked in a system where we explicitly did not have a constraint around which programming language or toolkit was used to develop applications. And it created a lot of problems around flexibility in the system. If we only have one team that knows how to write F#, it's hard to hire for that team. We can't let anybody on that team go work on any other part of the system because we have to have the F# developers work on the F# component. If you have a system that's written in .NET, but that team is mostly working on Node.js, this may or may not have really happened. And that team consistently hires Node.js programmers and gets to the point where all all the .NET programmers have left, but now they're left maintaining a .NET app and they only know Node. And of course, they only know Node and they only have MacBooks. And so no one can even run framework, .NET framework. By not creating constraints, you create another type of constraint and another type of problem. So it goes back to the idea that there are no answers, only trade-offs.

[00:46:42] Matt Baker: Yeah. I liked what you said, going to the smartish pipe thing, about the choice to use WCF, you said it reified, is that what you said?

[00:46:51] Dave Adsit: Yeah. Reify is basically just an excessively fancy word for give a name to a thing that you are doing. So we do that with patterns a lot. You observe a pattern for a while and then you reify it by giving it a name.

[00:47:07] Matt Baker: I like that. Yeah. And it makes total sense. And I think it was like a smart move on their part to say, and it's It's interesting because thinking about it now, they never said, "we are a .NET shop." It was just baked in.

[00:47:23] Dave Adsit: You couldn't not do .NET because you couldn't communicate with the rest of the system if you weren't doing WCF.

[00:47:30] Matt Baker: Yeah. And even if you came in and said, well, let's say you became a bit of a smartass and thought, "well, I'll just write a .NET edge. And then I'll write an integration to that edge that you won't see and I'll write in whatever language I want." I feel like as an architect, you could just say something simple like, "well, okay, do you think that's the wise way to spend the company's money? Can you justify why you want to do that?" I like that no matter which way you spin it, you just baked in a requirement that you don't have to fight about anymore. You just can't work in the system without it.

[00:48:00] Dave Adsit: And the thing is, is that sometimes we make decisions that do that accidentally with good or bad consequences. And sometimes we do that intentionally with good or bad consequences. One of the hardest parts about being an architect is knowing the consequences of the decisions that you are making for and with the teams.

[00:48:21] Allan Stewart: Another example of trade-offs in a system that all three of us worked together on is a microservices system. And my team was involved with authentication into the system. There weren't a lot of the things that now exist for like API gateways, managing authentication tokens for you and whatnot. And so we created one ourselves. We use the JSON Web Token or JWT. And on the whole, I feel like it was pretty successful, but the requirements that we had changed over time. Early on, a big requirement for us was that it needed to be easy to use. And we wanted to make sure that all these different parts of our distributed system could independently verify the JWT, even if the authentication microservice was offline. So we decided to use a symmetrical signing key, and we would share the signing key across multiple teams. We also provided another way because it was early on and we didn't know what made sense or what was going to be most effective for teams. So we were also publishing messages that said, "hey, this token, if you see it, it's good until this time." And so teams were able to choose how they wanted to deal with authentication. And it was largely successful, as I said, but we ran into some problems because of those decisions. One problem that we ran into is that sometimes teams would want to abuse the shared secret. And I went back, I was like, "you know, really, really wish that I had made this a public private key kind of situation and that we would have the private key and everybody else would have the public key." But at the time, the public keys were much harder to use with a JWT and to verify, especially across multiple technology stacks. Elliptic curve signing hadn't really caught on yet. And so even just sharing the secret was more difficult because of the size of the secret and how many bits long it had to be. But I think the real thing that changed on us that impacted how I thought about the authentication was when we started having more and more enterprise customers who wanted features like single logout. We weren't really maintaining sessions. We were issuing tokens and saying, "this token is good until X time." Until that time, you were logged in. And if someone hijacks that token, then they're logged in as you until that time. Over time, it really changed my perspective of these architectural decisions that we had made around decentralization and ease of use. It was really successful for a while and it met those goals. But when we needed to change it, then it became much more difficult because it was baked into the whole system.

[00:51:09] Dave Adsit: One of the things that that reminds me of, Allan, is that when we were working on that same system and we were trying to extract the very first microservice, I had given very explicit instruction that we use events for communication between different services in the system. And you took that to heart and you were following that rule. And at one point you said, said, "come look at this flow and help me understand what I'm missing here." The whiteboard diagram that you had drawn up included messages flying back and forth. You basically replicated the TCP handoff or the TCP SYN-ACK cycle with messaging, one-way messaging. And I realized in that moment that my instruction had been far too rigid. And I changed it in the day. On the day, I said, "obviously, you should be using an API here." And so that day, I updated all of my architecture slides to say, "favor message-based communication, use API-based messaging or synchronous communication where appropriate." And it simplified the work that your team had to do by probably a week or a month, like maybe a year. I don't know.

[00:52:27] Allan Stewart: Yeah, for sure. We were reinventing some kind of saga and it was very complex, but it didn't need to be that complex. I remember you came over to the whiteboard and you're like, "here, one line." You grabbed a different color and it's just like, "make this one line and all this other complexity goes away."

[00:52:46] Dave Adsit: Make an API call. And that was my fault. I had intentionally put a constraint on you without considering the consequences. And when I saw the consequences, I realized that I was wrong.

[00:52:59] Matt Baker: Allan, what was one of the indicators that told you the JWT strategy you had devised was no longer the needed strategy?

[00:53:08] Allan Stewart: At some point, I realized that we had done a lot of extra work to make sure that there was very little centralization of anything. Than when these other requirements came in. More than anything, it was security requirements that I became aware of that drove that realization for me. So when we did a pen test and we got some bugs around logging out and then reusing, like, manually putting their JotPack into place in the cookie and then you were magically logged in again, some of the single logout requirements that enterprises wanted, or even just abuse of the shared key. There was a team or two who really wanted to manage things in a different way and say, "well, let us just sign some JWTs for a particular purpose." And that was breaking our security model. Those are the things that really came back to me and challenged my assumptions about why we had made the decisions that we had made.

[00:54:05] Matt Baker: It's interesting with those changing landscapes. It's often surprising where the pressure comes from that shows you your design is no longer going to work. I know in that case, there was an element of the customer there, a big customer had said, "we will cancel if you don't allow us to log out." And the JWT model wasn't allowing that right now. And it's just interesting because I suspect when you were designing the JWT of that whiteboard, you weren't thinking about this big customer that could come in and throw their weight around and give you a bad day. And I don't know what the, I don't know what the lesson is there other than the, I guess the things you take for static are, no, I don't know what the lesson is there. Well, if I could propose one, it's

[00:54:57] Allan Stewart: that your architectures aren't static. Things are going to change. The context of the business is going to change, and so that's one of the things that makes architecture important and difficult. There are some things that we have learned as an industry that generally work pretty well. Low coupling and high cohesion, as an example, that tends to allow you to organize your code in ways where you can make changes. Some of, like, the SOLID principles, open, closed, and single responsibility, dependency injection. These are things that we've generally learned. And then we find that as an industry, you get a new job and it hasn't been as learned as well as you had hoped. Generally, we know that there are some things like that, that most of the time work, but a lot of it is guesswork. A lot of it is, I understand what our business is like now, and I have no idea that we're going to make an acquisition later. And I have no idea that we were planning to grow. And so we were designing a system for growth and now we can't grow because the financials of the company won't allow it. And so it changes our decisions. It changes the context of where we were working. The lesson there is that you just have to watch out for those changes and be willing to go back and revisit your assumptions and say, "this worked fine, right?" So like that JWT, it really worked well for a long time in that system. And it allowed us to learn a lot. And it allowed us to break up our monolithic system and have teams that were able to work independently and much more rapidly releasing their products. So I think it was okay. I don't think it was a bad architecture. We just always have to be watching out that the future is going to change on us. The company is going to change or the world is going to change around us. And that will impact our architectures.

[00:56:49] Matt Baker: Yeah, that's one of my bigger takeaways from our discussion today. And it's something, it's one of those things that you know, but you, you know, you relearn all the time. And I guess this one for me is a reminder to not so quickly dismiss new and interesting information. You know, that's super general, but what I mean is I know when I design a system and I see it work for a year or two, it's hard to convince me that it's no longer working, like, I'll see the anomalies initially and I'll chalk them up to be just that, anomalies. Um, and I'm resistant to the pattern that's emerging sometimes when that occurs to me. And it takes me a little while, like, "crap, okay, this isn't working anymore." And, like, the bigger the investment in, sometimes the bigger the investment out on those for me, you know. And so if I'm really, if I really put a lot of work and, and really became confident that it was what was was needed, it's going to take me a minute to let go of it. That's the lesson I'm taking away is just remember that sometimes I can get a little fixated.

[00:57:53] Allan Stewart: Yeah, it's hard not to. Yeah. Well, I hope in this episode, we were able to provide some examples that shed a little bit more light on the context around some of the decisions that were made. There really aren't good or bad architectures. They're either fit for purpose or not. And like we were just saying, they can can change over time. As always, we recommend joining a community of professionals by attending a Software Crafters group or meet up near you. The Utah SC group at utahsc.org has a virtual meeting the first Wednesday of each month. Maybe we'll talk about our architecture war stories with you there.