Crafting Code Podcast

[00:00:16] Allan Stewart: Welcome to episode 16 of the Crafting Code podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately I've been thinking about re-watching Star Trek The Next Generation.

[00:00:31] Dave Adsit: I'm Dave Adsit, a technologist, and recently I have been thinking about effective product development teams and startup cultures.

[00:00:41] Matt Baker: My name is Matt Baker. These days, I just think of myself as someone who tries to write computer code.

[00:00:48] Allan Stewart: Before we get into our topic today, we wanted to give a big thanks to Todd Fisher for our new intro music, which debuted in our previous episode. Our topic today is architecture examples. It's really easy to talk about architecture in very vague terms in order to be general enough to make it broadly applicable. We do this sometimes in this podcast, it happens in conference talks, etc. But unfortunately, that's a lossy process. Without understanding the context of a system, the people involved, a lot of those important details, the architectures, are lost. We're going to talk about some specific architecture examples, situations that we've experienced where we've had fit and unfit for purpose architectures. We'll try to provide some context on why it worked out or why it didn't. So Dave, do you want to kick us off?

[00:01:42] Dave Adsit: Yeah. So the first one that I want to talk about is back in 2003, I worked on a software application for management. This was an add-on service that we gave to our customers that allowed them to keep better track of food costs, etc. And do a better job ordering. Of course, the company I worked for, our interest was in getting people to order a lot of things from us. The system that I built ran on a desktop computer. And it ran on a handspring Vita. And that may or may not mean anything to you. The system that we had communicated with our systems through FTP. So the VB6 application would dial up possibly to the internet and upload files to an FTP site, download files from an FTP site, etc. And then the user in the restaurant would plug in the visor and it would synchronize the system. And then they could go around the restaurant and scan in their inventory and use their current inventory as input into creating a new order to place with us as a food provider. All of that was pretty revolutionary for the time and has all been superseded by the amazing cell phones that we all carry around these days, even in restaurants. The thing that was interesting about this project. The thing that was a constraint that led to an architectural decision is the fact that in 2003, most restaurants only had most restaurants of the scale that we were serving only had one computer. And that computer typically was purchased to run the point of sale system. And the point of sale systems that most of these small restaurants were using were running on Microsoft SQL. So we had a little server developer edition or micro edition. I can't remember the exact name of the edition, but it was a little tiny SQL server and it was single instance, which meant that you couldn't run a second one on the same machine and you couldn't run a second database in the same database engine. And you couldn't use this database because if you were to touch it and break the point of sale system, that could cost the entire account. So we made an architectural decision early on to use. Microsoft Access databases. Or Jet databases or whatever you want to call them so that we could actually have something resembling a relational database to store all of this inventory information. And that database ended up being a single flat file on the system. And basically that was the decision that we made that allowed us to move forward with developing the product. I'm sure that a lot of people have a lot of opinions about using Access databases, Jet databases, or whatever to store data locally. A lot of those opinions are probably right. But if you think about the constraints that we had on our system, using an Access database was certainly a better option than maintaining a whole bunch of flat files or XML files, which, you know, the Jet engine actually did that for you. It would manage all those things for you. And then it would allow you to write actual SQL statements to query that data, which became critical when we started creating custom reporting inside the app. Yeah. So that's a good point. And then the other thing that we did was we added a new feature called the ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! lot of work, but it actually was a nice unintended consequence of that architectural decision.

[00:06:31] Allan Stewart: I hope that that was before SLAs became all the rage.

[00:06:35] Dave Adsit: Yeah. Usually these were like live phone calls with the single end user at the small restaurant where they had been panicking about their inventory and being unable to deliver an accurate order to their salesperson. And so they would sit on the phone with me while I was in VB, in debug mode, fixing their database and possibly fixing the code and preparing a patch to upload that would then fix the problem for them. So the SLA was not a consideration. Also, this isn't necessarily a strategy that scales beyond a few hundred active customers.

[00:07:15] Allan Stewart: So one architecture that I participated in that's a little bit more modern, like... It was a microservices architecture where we were publishing events. We were using RabbitMQ in order to signal other services when there were changes in our system. This worked out really well, but one of the things that we didn't really understand or observe early on in the system was the different ways that people can handle events. This came up in a couple of different ways, but particularly there's an article by Martin Fowler on his website, it's entitled, What Do You Mean by Event-Driven? And in that he distinguishes between event notification, where you send an event and say, something happened, here is a minimal amount of data, generally just some identifiers to let you know which thing it happened to in the system, which entity had changed or was added or deleted or whatever. And then the other type was event carried state transfer. And in that you send more than the identifiers, you're sending all of the related data about the change. So if there's a new user added to your system, you don't just say, new user with this ID, you say, here's the new user and here's their name and here's their email address and here's whatever else that is relevant to share with the rest of the system. And overall, I felt like this messaging architecture worked really well for us, but then we started running into problems where some teams were having item potency issues because they were listening to item potency issues where they were taking that data and saving it into their database. And that was really convenient because they didn't have to make any API calls. Everything could be handled out of band, but they ran into item potency issues where messages would get replayed, or if they came out of order for some reason, if there were a lot of high traffic messages, it could really cause some problems. And meanwhile, there were other teams that were having a different set of problems because they were just looking at the IDs of things and making API calls. This incongruency between the two ways of doing message notification really caused some confusion in various parts of the system, depending on which team you were working with and which pattern that they were attempting to use and the patterns that the other teams that they were interacting with happened to be using. I remember working in this system and I was

[00:09:41] Dave Adsit: one of the biggest champions of leaning hard into event carried state transfer. It certainly does In theory, provided that you have perfect hardware and no message failures, even out of orders can be handled by things like timestamps or offsets, or if you get more detailed, each message can have its own monotonically increasing ID so you know the messages. If you get messages out of order, you can know to ignore them. But man, it certainly does cause problems if you are Different paradigms. First of all, if you've got different expectations about what messages are doing, it causes even more problems when you have to deploy one of these systems onto actual physical hardware somewhere and you realize that not all of the systems are up all the time and not all of the messages even get received every time and there's all kinds of data transfer issues.

[00:10:43] Allan Stewart: Yeah. And ultimately, I feel like either pattern could be very successful. The architectural issue that we had was kind of an accidental architecture. We talked a lot about how we were going to do events, but we just didn't know how important this distinction was. And so, so many teams were just doing it in different ways. And it really exploded the number of patterns that were going on inside the system and limited our ability to share strategies across teams until we realized what the problem was and could start thinking about how we wanted to do it.

[00:11:19] Dave Adsit: One of the things that this makes me think about is just the general concept of a data intensive application where you have multiple data sources and providers working together with transfer mechanisms, et cetera. And just what a complex problem that becomes in production or with any

[00:11:40] Matt Baker: kind of scale or load or concurrency. Okay. So an architecture I want to talk about is one that I don't get as much airtime in the thick O'Reilly books, but still, I think it's super valuable. I don't know the name for it. I'm going to call it trying to find product market fit architecture. This is the type of architecture I reached for when helping a startup who was trying to find product market fit at a very short runway and needed to deliver quickly and often in order to test ideas. They had a few pilot customers and they had constant access to them. They anticipated that they needed to iterate on a mobile app, they were building for their customers quickly. So the situation, they would make a change, go out into the field with the customer and then see how it worked. And they wanted to do that a lot. They didn't have any investment bootstrapped and very thin. So I'm saying that just to underscore that there was no fluff. Every dollar needed to be spent well. So we looked at the situation and there was nothing on day one and they needed a mobile app. We quickly converged on they were opening котlin mobile app. And we, on day one also built the CI CD pipeline and it took maybe six or seven hours to build that CI CD pipeline. We did it in Jenkins just on a hosted instance. And then we got to work. So right away we abandoned a ton of practices that I would do at a more established company. We didn't do TDD. We never talked about a branching strategy until we had to release multiple versions of the app to different contexts. Like this one's for test and this one's a production release. And then we started branching котlin mobile app. branching to handle that. We didn't write a team agreement. We didn't do a lot of things, the more established company you would do. And that was okay. We quickly keyed in on the fact that we needed to minimize any non-differentiating work. And so I want to give a few examples of how we did that in order to really respect the startup's bank account, which was again, just super slim. And if they had a chance, we wanted as much time as we could to test that idea. So we chose Firebase to deploy our Android app to the app store. It made interfacing with Google just easy. So we reached for Firebase. And at some point in the app process, we decided we needed analytics. And it wasn't a conversation of like, okay, what analytics provider we should use? The conversation went like this. Does Firebase offer analytics? Yes, use it. And it wasn't like use it in like a being told what to do kind of way. But yes, we all agree. Let's use the analytics that's closest to where we're currently at because it's the best way to minimize non-differentiating work. And that theme just threaded through all of our choices. So at some point we needed an identity solution. We needed to add user accounts to the application. And so we used the identity solution that Firebase offered. We didn't go out and compare other tools and offerings. We didn't even consider other things. We just said, no, we know our strategy here. Our architecture is one of the key tenants of it is minimizing non-differentiating work. And that drives our tech choices. So after that one, at some point, needed a state store. We kind of hit the point in the app process where we needed to persist state and same thing, just reach for Firebase. This sounds like an advertisement for Firebase. It's not, but it happened to work in this case. But the thing I want to highlight is it wasn't architecture. We were very intentional in what we did. We didn't just accidentally make choices. We understood that the money was tight. Now is not a time to flex interesting tech or try and reach for a new tool that might be fun to play with. Now is the time to ship and like ship a lot. And so we chose tools that optimize for that and nothing else. And I would call that an architecture. And I think it's, like I said, the trying to find a product market fit architecture. And the reason I led with this particular example is I think it kicks the door open a little bit on what an architecture is. And when you talk about, well, what's this company's architecture? Sometimes we might say, well, that company doesn't really have an architecture. It's just nonsense. And I kind of take issue with it. I'm not saying that any of you have said that, but I kind of take issue with that because it's there. No, they've made choices just like I talked about. Like that has occurred like a lot of times that that company to get to where they're at, the architecture is there. It might, it may be unclear. It may have drifted or I don't know. I guess we could theorize why some places look like they have architectures and some don't, but sometimes an architecture looks like just trying to save as much money and time as you can.

[00:16:05] Allan Stewart: Yeah. I like that concept about the multiple facets of architecture. One of the facets is managing change. And helping understand what things are going to change and how you're going to segregate different parts of the app based on change. And so CI CD becomes part of your architecture. We don't often think about that. It often is thought about as like, oh, well, that's just part of our, you know, deployment pipeline. Sometimes we'll think about deployment architectures in terms of like, oh, is it a monolith? Is it a microservices? Is it something out some end tier application, but really that concept of how quickly can you get changes out into production? It informs other choices that you make within the system. It affects how you're going to write your code. It affects how you're going to manage releases and deployments and such.

[00:16:54] Dave Adsit: You got me thinking a lot about the rapid application development toolkits that used to be super, super popular and now may not be, I don't know. Sometimes they get lumped into the demo where type of toolkit. And I have often, then at the forefront of the conversation, bashing on anything that allows you to build quickly and maintain poorly. But what do we always say? Context is king, right? And the context that you were sharing, it really does matter that you can iterate quickly. You probably ran into some spaces where you're like, okay, throw that away, rebuild it because it's easier to build it again than it is to maintain it or iterate it or refactor it from where it is. I think we've all experienced that. I worked with one QA automation tester who every iteration would go in and rerecord all of his tests using the test recording tool versus maintaining the scripts that were used. And he's like, well, I can record all the tests in about two hours and it would take me all week to make these changes manually to the existing test code. The app is failing when the tests don't pass. And I am now familiar with it. I think that's something critical for us to remember is that in certain contexts, certain practices are essential. And in other contexts, those practices become an anchor that sinks the entire team or company or

[00:18:33] Matt Baker: product or whatever. Yeah. So to speak concretely here, I am definitely a bit of a TDD zealot. That's because I've just seen when it's done. So the way I deliver code and I assume that it would do the same thing for everyone else. Probably not true, but it was so profound for me, the way it changed that I, the way I write code changed after adopting TDD. It's almost like, when would you not do this? This is crazy. It's great, but it does slow you down. And I think that even the biggest advocates say that, that yes, you're trading in some delivery upfront for sustained delivery in the long haul. And I would just call out going back to what you were saying, Dave, we didn't want, we didn't want to sustain delivery in the long haul. We were the idea for the app we were writing was changing every day. So I was like, well, the last one and Dave's right. The last, we would throw away iterations of the app, easy peasy. And we could do that because we were trying to minimize the choices that married us to the, you know, that particular revision. And so we didn't reach for TDD and did we suffer some quality? Absolutely. Like we had some bugs, we had some production bugs that we, that we had to deal with, but the customer count was so low, you know, it was okay. And at the end of the day, we accomplished our ultimate goal, which was delivery. So I guess in summary, Dave, I'm just trying to offer a concrete with what you're saying, you know, yeah, you should typically do TDD. If anyone asked me, should I do TDD? I'll say yes, without even asking like, well, it depends. Like I'll just say yes.

[00:19:54] Dave Adsit: No context needed. Yes. This is the context free answer. Yes. You do need to do TDD. What if I'm throwing away my entire app every two hours and starting from scratch? Well, okay. Maybe not in that case.

[00:20:11] Allan Stewart: Going back in time from my next example about, I don't know, 2010, 2011, I was working doing a lot of rich internet applications, which meant writing action script code for flash and flex. And I was working on a project where we were delivering educational templates. So we wanted to have a lot of interactivity with different kinds of templates. And so there was like a player object that would manage, basically a lesson. We're going to teach US history, or we're going to teach math or English or a social studies something. And then the player could load up a whole bunch of different templates. And some templates were really simple. And all they did was show text with a picture. There were other templates that were multiple choice or drag and drop or other kinds of interactivity that you could do to try and improve the engagement and improve the learning outcomes. When I was first working in the code base, I had come on to an existing code base. It was kind of a mess. There was a lot of code just splattered all over the place. Templates intermixed all of their concerns. State management was intermixed with the user interface, was intermixed with receiving events from the mouse or keyboard. It was just hard to get anything working. And every time that they did a new course, they wanted to reskin things. Slightly basically use the same engine, but now with some new changes, given the state of the code, it was really hard to make that happen. But through a number of circumstances, I ended up being the sole developer on the project. There had been like maybe four developers originally. And I was the one who got the honor of continuing to work on it. One of the things that I was learning at that time was unit testing. And I was also learning a lot of patterns. And so I decided to make some architectural choices around applying the MVC pattern to break up these different concerns. So we'll have more of a model to represent, here is the data that matters for this template. Here's a controller. It interacts with the user. Mouse events, keyboard events, whatever, they go to this controller and it will make the appropriate change to the model. And then the view will just display things. My unit testing go a lot better. I was still learning how to unit test. I didn't know how to write very good unit tests, but unit testing the models and the controllers became very easy because it just had to do a particular function based on whatever the current state was. And then the views were basically impossible to test. And I just ignored that. And so as they started having all these different templates, I felt like this architecture became really powerful. I was able able to get a lot of work done, made the product owners very happy with what we were doing because each new template, especially when they would make minor view revisions on the template, I would just keep 90% of the controller and model intact and just replace the view. And so very quickly things were getting done. I may have gone a little bit overboard though. I think Matt, you've talked about in the past about how when you learned about the decorator pattern, it's like now did. Well, I kind of did the same thing here with model view controller. It's just like, okay, every template has model view controller. And it's like fractal because the player itself has model view controller and all these different components that go on the player, like in order to do like page navigation, they were all using model view controller and they were bubbling up events through not the DOM because it wasn't HTML, but essentially the same kind of a concept. I thought it was great. And we were doing unit testing and things were really good, but underscoring the importance of people, we got to a point where they wanted me to transition to a different product. And now I had to teach another person how to maintain this code that I had written. And I was like, oh, well, this is awesome. And it's easy and it's simple. It's so flexible. You can do all these things. And the other developer just was totally lost, had no, it was so foreign to how they were used to. Writing code that the architecture, which was in the context of Allan is writing this code all by himself. Great, really fit for purpose. You make that one human element change. And now all of a

[00:24:46] Matt Baker: sudden it was not fit for purpose anymore. What would you call it? Allan, like the hard to transfer architecture, like hard to, hard to leave your job architecture.

[00:24:55] Allan Stewart: I don't know, but I'm, I'm really excited to hear in a conference talk from somebody, the trying to find market. Product fit, product market fit architecture. Once I start hearing that in the wild, then I know that you will.

[00:25:08] Matt Baker: Toss credit back my way. If you hear it, don't let them steal it. Dave, I feel like hits this point a lot about architecture being a people problem. And I don't mean to say that it's in its entirety, it's a people problem, but it's highlighted for me in the story you just told Allan. And it definitely reminds me of similar missteps I've made. I have to imagine when you were writing it, the points of extensibility, I'm assuming the points of extensibility made it a little more complex than it needed to be, or like the general,

[00:25:44] Allan Stewart: generalizing, you know? Yeah. Is that right? I think it probably did.

[00:25:47] Matt Baker: I assume when you were writing those, the intention was to add value to the company. Like, I think you saw, Hey, this thing's going to change in these ways. So I'm going to add these points of extensibility here, or maybe even just leave the scaffolding for it. They need it. But for sure. And I know when I've made those choices in hindsight, I look back and I just say, it was just noise. Like I, I get why I did it, but I think like given the chance to do it again, I would say, just do it the dead simple way. And then if it sucks enough to do it the dead simple way, you'll probably know where you should make it extensible or configurable. Like it'll, it'll announce itself. You know, you're it's not, it's a hard thing to guess where it's going to

[00:26:28] Allan Stewart: show up. Yeah. Yeah. And I think there was definitely some of that going on. In that code base, but also there was just a kind of disconnect between what was important to me and how I was working and how the other developers around me in the same company were working. I think going back to that idea of the socio-technical system, what I had created wasn't, wasn't bad. I think that there's a lot of similarities when we look at some modern, like, especially like front end architectures today, you might have like a react app with Redux. We don't think about everything. Being in terms of model view controller, because it's not exactly the pattern that we're using, especially when you go back and like, look at the gang of four book, or you're doing like a UML diagram, but there's a lot of similarities there as far as like, oh, well, we've got a store. That's kind of like where your models live. You've got you know, you might be doing things more functional now instead of really state-based, but you've got state in there somewhere. You know, how do you interact with the different events that are happening on the screen and, and so forth? Like there's a lot of similarities. There. And so I don't think it was a, it wasn't a bad architecture as far as like creating all kinds of things that were unnecessary, but just the pattern was different.

[00:27:42] Dave Adsit: Dump out the whole gang of four books on top of it.

[00:27:45] Allan Stewart: Well, you know, like I said, fractal MVC for the win, but it was, it was so different from how the other developers thought about working that we were just basically incompatible. I couldn't, couldn't, wouldn't work on their projects. And if, if I had to, I would rewrite huge charts. I would rewrite chunks of it because I was working in a different kind of paradigm than they were. And if they worked on my projects, then they got lost because they're like, well, where is everything? And I don't understand what's going on.

[00:28:13] Matt Baker: It strikes me that whether or not you were right and they were wrong, you're still going to have that same problem. Yeah. There's something there about like, cause I know you you're smart and chances are you probably had a better solution than that. And you still had the problem. And so I don't know when I say that, what does that say back to you? What does that highlight?

[00:28:33] Allan Stewart: Oh, just the being right. Almost never mattered. You can be right about something and it doesn't make a difference to the outcome of the company. We also see lots of cases where, you know, people do things that we look at and like, okay, well, when we really critique this, we're going to call this wrong. And yet the company is super successful. And that wasn't a differentiating thing as far as the outcomes for the company. So. So we have to really be careful about placing that value judgment.

[00:29:06] Dave Adsit: This reminds me of a, another project that you and I worked on together, Allan, where the two of us were doing something similar. We had an, a way that we wanted to develop this application and we were introducing concepts like message bus and queuing as a way of managing out of band, out of process workloads. That didn't need to be handled immediately in the request response flow for the website. And, you know, we were introducing all kinds of interesting tooling and patterns. And, and I remember looking at the, basically when we were drawing the diagram, the architectural diagram up on the whiteboard and talking to the developers we had on the team at the time. It was all going well over their heads. And that wasn't their problem. wasn't their fault. That was us looking at interesting technology versus the essential socio-technical system, right? You were looking at how we would have solved the problem, just the two of us and not taking into account the fact that there were also five or six other developers who had no experience and no training and honestly, no ability at that time to contribute to the system that we were building. We were basically unintentionally working them out of a job by making a system too complex for them to work on. And I think that that is a failure mode that I have fallen into multiple times as an architect is expecting people to come along and keep up with ideas that I've struggled with or worked on or tried to learn about for five plus years. And now I'm ready to finally implement it in a system. I'm like, hey, I know you've only got I've had five years of experience since college, but let me tell you how we're going to do this event-driven microservices system with event-carried state transfer and data replication. And it's going to be awesome. You're going to love it. And they really, no context, no concept of what you're doing. It reminds me of the first time I did a ModelView presenter app as a website, a .NET, ASP.NET website. We did the ModelView presenter pattern so that we could into our presenter layer. Honestly, I had no idea what we were doing. For probably six months, people would say, type this code here. And I'd be like, okay. And I didn't know why. If we weren't pairing, I would have contributed nothing to the team during that period until those patterns actually finally started to click and inversion of control finally started to click and even interfaces. Like, honestly, I worked for, a good eight years before any of the code that I worked on, any of the code bases I ever worked on, even had a concept of an interface. We wrote a lot of code. We shipped a lot of value to a lot of people. I don't know. There's reasons to do different things. And the people have to be an

[00:32:17] Allan Stewart: essential concern for any skilled architect. To give us some credit back, I will say that maybe we didn't fully recognize the extent of that happening, but we did recognize it a little bit and we made adjustments to what we were doing. We simplified certain things. It's like, well, if we were doing it, just the two of us, we would do it like this, but because of the developers that we had, we're going to do this other way instead. And we also tried to make changes around the environment too, instead of just dragging them along, help them understand the purpose of what help level them up. How successful that was is a whole nother story, but I do believe that we

[00:33:00] Dave Adsit: recognized it at least partially. One of the failure modes for anyone skilled in a thing and software architects being no exception is underestimating the amount of work it takes to level up your knowledge in esoteric concepts. It's hard. Learning is hard. How many of us have worked at a company that focuses on learning? Oh, I know all three of us have multiple times.

[00:33:34] Matt Baker: Don't out me. Don't out me. One day I hope to learn something. One thing that was coming to my mind as you were talking and something that I honestly haven't appreciated until at this point in my career, I looked the other day, I've been paid to write software for 18 years. So to scope that, what I'm about to do. So it took me about this long, maybe a little bit. I'd like to claim it was a couple of years ago. Maybe it wasn't, I don't know, but it took me a while to understand that the level of expertise you hire is a business strategic choice, a strategic business choice. That's a better way to say that. And that has all the impact on what you were talking about, Dave, from where I'm sitting. If you're designing a system that is complex and hard, I was going to say hard to work in. I don't If you understand it, like an eventing system, I don't think those are hard to work in myself. Some time ago at some point in my career, yes, they were, they were hard to work on for me. And so I guess what I'm inching towards is if it's hard for me to work on your system, I might not be the right hire for your company. And if your company is positioning themselves to hire people at my salary band and experience, because it's the right fit for what they're trying to build, you're going to run into some rough spots. And the system can't be adopted and owned by these people that they want to hire. And so you're almost forcing the company to hire more expensive people. And so it just all comes back to Dave in particular says this all the time, like architecture is a people problem. It just kind of comes back to that for me. Like when you're designing this system, and this isn't something that I've ever heard an architect like sit down and think about. And that's kind of the nuance that I'm trying to highlight. It's like, it took me a long time to catch this point. Like I don't see it. I've talked about it at conferences and other things, but it's real. You need to think about the people that are going to be working in the system you're designing. And you do need to consider their anticipated skill level. And like, what a like subtle nuance that comes out after you just like go at it a lot of times. I think you're not going to pick up on this. I don't think in the first one or two systems that you're the architect for, but in time, I think it hits you in the face if you don't notice it beforehand.

[00:35:54] Dave Adsit: That is a very, very interesting point. I think that I've heard that referred to as the habitability of the system, that the people who live and work there have to be able to live and work there. And I've usually heard that in terms of using esoteric programming languages and extraordinary complex toolkits or patterns, but this applies at all levels of complexity for a system. You know that you're going to be hiring a bunch of developers right out of school, and you're probably only going to keep them for a few years before they have enough skill that they go get a better, higher paying job on more complex problems or more interesting problems or whatever. Then maybe you should be considering the advantages of having a monolithic system with maybe some well-defined layers or some rules in the system that You and I might chafe at with more experience, but they make the system habitable for the people who are there. I recently, speaking of architectural decisions, my youngest recently turned five and I went through my kitchen and aggressively removed all of the child locks from the cabinets after years of frustration looking for that magnet that opens the cupboard that has my Tupperware in it. Why were you locking up your Tupperware? Just every cabinet? Every cabinet had locks on it. Every lower cabinet had locks on it because the thing about two and three-year-olds is that they don't believe in cabinets. If you can't see it, it might not exist. So everything in every lower cabinet must be pulled out onto the floor on a regular basis so that you can do inventory and make sure that you still have the pot that you can bang on and all of the lids are appropriately stored far, far away from the glass dishes that they belong to, et cetera. Ph.D.:

[00:38:03] Allan Stewart: I can confirm this is a true thing.

[00:38:06] Dave Adsit: And at a certain age, those things become significantly less interesting than playing Minecraft or whatever. So you don't need to lock up the plastic dishes.

[00:38:20] Allan Stewart: Going back to what you were saying, Dave, I worked in a Ph.D.: Yeah. Preston Pyshko, Ph.D.: at one point where one of the architects was really trying to templatize a lot of stuff. This is how you do it. You are going to work in this framework. And for me at the time, I really chafed at that. I was like, this is stupid. We should be building it up from first principles every time. And now I'm starting to realize, oh, given the context of that particular company, that actually might've been a really smart decision to push a lot of This particular framework is just the way that we're going to do it because it's not the best, but it's consistent.

[00:38:59] Matt Baker: Yeah. It's almost like, look, is it delivering? Then stop the conversation kind of thing. That's almost how blunt it has to become. It's not that we couldn't change it in fantastic ways. It's just that we don't need to. We're going to templatize this thing because, and I wasn't there, so I'm out on a limb, but we're going to templatize this thing because we don't care about it. We're not looking to innovate on it. It's just a necessary evil that we have to deal with. In order to get to our ultimate goal.

[00:39:24] Dave Adsit: Yep. So Matt, you've often told me that sometimes the answer to a question in software is go develop software for 20 years and then come back to me. I'm really excited to hear the answers to the software questions in a year or two. The jig is up.

[00:39:52] Matt Baker: The real. Yeah. It'll probably be some answer about like the more you do something, the less, the more you learn, the more you learn about something, the less, the more you realize, the less, you know, or what, what there's a saying there that I'm just can't get.

[00:40:07] Dave Adsit: As long as you don't come up and say 42, because that won't help me at all.

[00:40:15] Matt Baker: What if it's the binary representation of 42?

[00:40:19] Dave Adsit: Oh, it'd take me a while to figure that out. And then I will send you a link. I'll send you a Rick roll or I'll feel as though you have sent me a Rick roll. Okay. All right. So what's the next architectural experience that we have had?

[00:40:33] Matt Baker: Yeah, I've got one on deck here. I'm calling this one, the smart ish pipe architecture, the reference being to smart pipes, done pipes. I'm not going to go into that pattern, but there's lots of arguments about it on the internet and this architecture, I feel like cut up the middle in an interesting way. Okay. Okay. So let's start with the smart ish pipe, the TLDR on the architecture, a big monolith broken out into a distributed system. Surprise, surprise. Everyone's heard this, right? The interesting part though, is the integration between those subsystems that were broken out. In this particular case, they chose WCF. It's a Microsoft technology that allows you to write. I'm going to misrepresent WCF. And I'm sure someone who knows it better is going to say, you don't know what you're talking about. But my, what? My experience was is that you create interfaces that are very easily exposed over TCP and you can invoke them. Sometimes you can invoke them in a, in a way that seems like you're running a, like you're, you're invoking on your local machine, but you're not, it's, you know, it's a remote procedure call. And other times it's obvious that you're calling across the wire, but they handle things. Like, like I said, they'll expose the operations for you in a nice way and type languages like C-sharp. They also would handle GU. That. Okay. Okay. Okay. about like network retries. They just got it out of the box three three attempts over a TCP connection, and then it would fail and that was configurable and then just stuff like that. When I got there, this system was already mature when I arrived. And I spent some time with the architect over the I think I was there like a year or two in my time there. And one of the things one of the conversations I remember having with him was about his choice of WCF. And I was harping on it, I told him you should have just paid HTTP is the lowest common denominator and what allowed for like some some variants when people wanted to integrate subsystems. But in this case, we all have to use WCF, ergo we ought to use.net. He so I pissed him off when I said it, I didn't mean to but it you know how it goes. So he he just kind of in a quick manner, like a bursty fire manner said like, Well, here's all the reasons but but but but but and listed off all these reasons to back up his rationale. And then he was like, Do you have answers to those? And it's like, Oh, no, I don't. And then we he kind of walked off. But I appreciate what he did, though, because he he highlighted for me, like, as I thought about it that I didn't even like lead with, why did you make these choices, I just immediately condemned and went, you know, whatever, you chose the wrong thing HTTP, because dumb pipes, right. So I thought that it almost like a from a purist perspective, when you connect subsystems in a larger system, you do it with pipes that don't know anything about anything. Other than they are responsible for carrying data across the wire. That's what they know, they know where they receive the data, they know where they drop it off. And that's it. And so the smart ish pipes is a little bit up from that, right? Like it, the pipes became a little bit more interesting, they had some fault tolerance, and then they understood what a dotnet interface was. And they knew how to take calls to that interface from one remote computer and patch it over to another and, and it worked. And he was right. And they never had any problems. They're still a very successful company growing quickly, people that have stock, they're very happy with it. And they'll probably continue to be that way. And there's no reason for them to dumb down their pipes in this particular case. So I would call it a win, I call it an effective architecture choice. And like I was saying earlier, who cares what you think it works, and they're rich. And it's the smartest pipes architecture.

[00:44:16] Dave Adsit: I think one of the things that you mentioned there kind of in passing that I think is worth mentioning, is that you mentioned the idea of the ! The the the the the incidentally to another decision. And sometimes a decision is made that reinforces an existing decision. And it sounded like at the time you probably were chafing at the fact that you had to use a .NET language in order to build components for that system. That's right. And I suspect that that was intentional, at least on some level. I think you're right. Reinforcing the requirement that components of this system are written in .NET. And that goes back to the socio-technical system, right? That sometimes we want to put a constraint on because that actually allows us to facilitate other types of operations, like moving people between teams or between projects, because it's all .NET. I have personally worked in a system where we explicitly did not have a constraint around which programming language or toolkit was used to develop applications. And it created a lot of problems around .NET. And flexibility in the system. If we only have one team that knows how to write F-sharp, it's hard to hire for that team. We can't let anybody on that team go work on any other part of the system because we have to have the F-sharp developers work on the F-sharp component. If you have a system that's written in .NET, but that team is mostly working on Node.js, this may or may not have really happened. And that team consistently hires Node.js programmers and gets to the point where all the .NET programmers, they're кот кот кот кот кот кот кот кот кот кот кот кот кот кот кот кот кот кот кот кот кот кот кот кот кот кот

[00:46:49] Matt Baker: You said it reified, is that what you said?

[00:46:52] Dave Adsit: Yeah, reify is basically just an excessively fancy word for give a name to a thing that you are doing. So we do that with patterns a lot. You observe a pattern for a while and then you reify it by giving it a name.

[00:47:07] Matt Baker: I like that. Yeah, and it makes total sense. And I think it was like a smart move on their part, you know, to say, and it's interesting because thinking about it now, they never said we are a .NET shop. It was just baked in.

[00:47:23] Dave Adsit: You couldn't not do .NET because you couldn't communicate with the rest of the system if you weren't doing WCF.

[00:47:30] Matt Baker: Yeah, and even if you came in and said like, well, you know, let's say you became a bit of a smart ass and thought, well, I'll just write a .NET edge and then I'll write an integration to that edge that you won't see and I'll write whatever language I want. And like, I feel like as an architect, you could just say something simple like, well, okay, like, do you think that's, that's the wise way to spend the company's money? Can you justify, you know, why you want to do that? I like that no matter which way, like you spin it, you just baked in a requirement that you don't have to fight about anymore. You just can't work in the system without it.

[00:48:01] Dave Adsit: And the thing is, is that sometimes we make decisions that do that accidentally with good or bad consequences. And sometimes we do that intentionally with good or bad consequences. One of the hardest parts about being an architect is knowing the consequences of the decisions that you are making. that you're making for and with the teams.

[00:48:21] Allan Stewart: Another example of trade-offs in a system that all three of us worked together on is a microservices system. And my team was involved with authentication into the system. There weren't a lot of the things that now exist for like API gateways, managing authentication tokens for you and whatnot. And so we created one ourselves. We use the JSON web token, or JOT. And on the whole, I feel like it was pretty successful, but the requirements that we had changed over time. Early on, a big requirement for us was that it needed to be easy to use. And we wanted to make sure that all these different parts of our distributed system could independently verify the JOT, even if the authentication microservice was offline. So we decided to use a symmetrical signing key, and we would share it with the client. And we would share the signing key across multiple teams. We also provided another way because it was early on and we didn't know what made sense or what was going to be most effective for teams. So we were also publishing messages that said, hey, this token, if you see it, it's good until this time. And so teams were able to choose how they wanted to deal with authentication. And it was largely successful, as I said, but we ran into some problems because of those decisions. One problem that we ran into is that sometimes teams would want to abuse the shared secret. And I went back, I was like, you know, really, really wish that I had made this a public private key kind of situation and that we would have the private key and everybody else would have the public key. But at the time, the public keys were much harder to use with the JOT and to verify, especially across multiple technology stacks. Elliptic curve signing hadn't really caught on yet. And so even just sharing the secret was more difficult because of the size of the secret and how many bits long it had to be. But I think the real thing that changed on us that impacted how I thought about the authentication was when we started having more and more enterprise customers who wanted features like single logout. We weren't really maintaining sessions. We were issuing tokens and saying, this token is good until X time. Until that time, you were logged in. And if someone hijacks that token, then they're logged in, as you know, as a token. And they're going to look at your newsletters and they're going to look at newsletters and they're going to look at newsletters and they're going to look at newsletters and they're going to look at newsletters and they're going to look at newsletters and they're going to look at newsletters and they're going to look at newsletters and they're going to look at newsletters

[00:51:09] Dave Adsit: and they're going to look at newsletters and they're going to look at newsletters and they're going to look at newsletters I had given very explicit instruction that we use events for communication between different services in the system. And you took that to heart and you were following that rule. And at one point you said, come look at this flow and help me understand what I'm missing here. The whiteboard diagram that you had drawn up included messages flying back and forth. You basically replicated the TCP handoff or the TCP SYNAC cycle with messaging, one-way messaging. And I realized in that moment that my instruction had been far too rigid. And I changed it in the day. On the day I said, obviously you should be using an API here. And so that day I updated all of my architecture slides to say, favor message. You can use API based communication, use API based messaging or synchronous communication where appropriate. And it simplified the work that your team had to do by probably a week or a month, like maybe a year. I don't know.

[00:52:27] Allan Stewart: Yeah, for sure. We were reinventing some kind of saga and it was very complex, but it didn't need to be that complex. I remember you came over to the whiteboard and you're like, here one line. I was like, I'm going to make this one line. And all this other complexity goes away.

[00:52:46] Dave Adsit: Make an API call. And that was my fault. I had intentionally put a constraint on you that without considering the consequences. And when I saw the consequences, I realized that I was wrong.

[00:52:59] Matt Baker: Allan, what was one of the indicators that told you the JOT strategy you had devised was no longer the needed strategy?

[00:53:08] Allan Stewart: At some point I realized that we had. Done. A lot of extra work to make sure that there was very little centralization of anything. Then when these other requirements came in more than anything, it was security requirements that I became aware of that drove that realization for me. So, you know, when we did a pen test and we got some bugs around logging out and then reusing, like manually putting their job back into place and the cookie, and then you were magically logged in again. Some of the single logout requirements. Or even just abuse of the shared key. There was a team or two who really wanted to manage things in a different way and say, well, let us just sign some JOTs for a particular purpose. And that was breaking our security model. Those are the things that really came back to me and challenged my assumptions about why we had made the decisions that we had made.

[00:54:05] Matt Baker: It's interesting with those changing landscapes. It's often surprising where the pressure. Comes from that shows you your design is no longer going to work. I know in that case, there was an element of the customer there. A big customer had said, we will cancel if you don't allow us to log out. And you know, the, the job model wasn't allowing that right now. And it's just interesting because I, I suspect when you were designing the job at that whiteboard, you weren't thinking about this big customer that could come in and throw their weight around and give you a job. And you know, the, the job model wasn't allowing that right now. And I don't know what the, I don't know what the lesson is there other than the, I guess the things you take for static are, are no, I don't know what the lesson is.

[00:54:54] Allan Stewart: Well, if I could propose one, it's that your architectures aren't static. Things are going to change. The context of the business is going to change. And so that's one of the things that makes architecture important and difficult. There are some things that we have learned as an industry. That generally work pretty well. Low coupling and high cohesion. As an example, that tends to allow you to organize your code in ways where you can make changes. Some of like the solid principles, open, closed and single responsibility, dependency injection. These are things that we've generally learned. And then we find that as an industry, like you get a new job and, and it hasn't been as learned as well as you had hoped. Generally, we know that there are some things like that. That most of the time. Work, but a lot of it is guesswork. A lot of it is I understand what our business is like now, and I have no idea that we're going to make an acquisition later. And I have no idea that, you know, we, we were planning to grow. And so we were designing a system for growth and now we can't grow because the financials of the company won't allow it. And so it changes our decisions. It changes the context of, of where we were working. The lesson there is that you just have to watch out for those changes. And be willing to go back and revisit your assumptions and say, this worked fine. Right? So like that job, it really worked well for a long time in that system. And it allowed us to learn a lot and it allowed us to break up our monolithic system and have teams that were able to work independently and much more rapidly releasing their products. So I think it was okay. I don't think it was a bad architecture. We just always have to be watching out. The future is going to change on us. The company's going to change or the world is going to change around us and that will impact our architectures.

[00:56:49] Matt Baker: Yeah. That's one of my bigger takeaways from our discussion today. And it's something, it's one of those things that, you know, but you, you know, you relearn all the time. And I guess this one for me is a reminder to not so quickly dismiss new and interesting information. You know, I, that's super general, but what I mean is I know when I design a system and I see it work. For a year or two, it's hard to convince me that it's no longer working. Like I'll see the anomalies initially and I'll chalk them up to be just that anomalies. And I'm resistant to the pattern that's emerging sometimes when that occurs to me. And it takes me a little while like, ah, crap. Okay. This isn't working anymore. And like the bigger the investment in sometimes the bigger the investment out on those for me, you know? And so if I'm really, if I really put a lot of work and, and really, really put a lot of work into it, I'm going to be able to do it. And if I really became confident that it was what was needed, it's going to take me a minute to let go of it. That's the lesson I'm taking away is just remember this. Sometimes I can get a little fixated.

[00:57:53] Allan Stewart: Yeah. It's hard not to.

[00:57:54] Matt Baker: Yeah.

[00:57:55] Allan Stewart: Well, I hope in this episode, we were able to provide some examples that shed a little bit more light on the context around some of the decisions that were made. There really aren't good or bad architectures. They're either fit for purpose or not. And like we were just saying, they can change over time. They can look different. They look different. They look different. They look different. They look different. They look different. They look different. They look different. They look different. They look different. They look different. They look different.