Crafting Code Podcast

[00:00:16] Allan Stewart: Welcome to the Crafting Code Podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately I've been thinking about the many and varied trade-offs that you can consider when

[00:00:31] Dave Adsit: deciding how to work in your startup. I'm Dave Adsit, a VP of engineering, and recently I've been thinking a lot about deadlines, estimates, and commitments, whether they are intentional

[00:00:43] Allan Stewart: or accidental. Our topic for this episode is event-based strategies. So in our prior episode, we talked about event-based systems and how events can be used to help you decouple a system and some reasons why you might want to use events. This time, we're going to talk about some specific strategies that have been successful for Dave and I. Right. And one of the first things that

[00:01:05] Dave Adsit: comes up when we start talking about events is naming. Naming is critical in almost all things software development. In fact, we joke that there are two hard things in software development, cache and validation and naming, and off by one errors. And when it comes to events, it's critical. In my experience, names for events should always be past tense. An event is something that has happened already. It may have just happened. We immediately publish event as soon as this thing occurs, but the event itself is something that has happened. It's in the past. And therefore, given the laws of the universe that we currently understand, it cannot be changed.

[00:01:52] Allan Stewart: Yeah. I think once you start naming the event, it also becomes a lot easier to understand what it is that you're talking about. What your like events can be kind of this more nebulous concept within the system. And you start talking with your developers and you're trying to figure out what does this mean? Once you've given it a name, you say, oh, this is the event. You know, so like in JavaScript land, you might do something like on click, like, oh, okay, now I know. It's like, it's when this is clicked, you know, on drag on mouse over, right? Things like that. Now those ones are more of like a present tense concept because there's a little bit different situation going on when you're using events like in, in the browser and the dom than it is if you are publishing events in a system. But as soon as you've named it, it helps. And like, honestly, I could kind of get behind if they, changed other than the migration of everybody's code that they've written for the last 20 years of JavaScript. If it had been mouse clicked with a D at the end, you know, the ED telling you that this happened, the mouse was clicked. What are you going to do? Like it happened, right? Like the, the on click happens once they click it. It's not, you know, it's, it's then and, and, or it just happened rather than in the future.

[00:03:25] Dave Adsit: Ambiguously. Right. And I think that that is one of the critical things is that when we start talking about events, it is, it's easy to think about the purpose for which we are using the event. And then the concept of the event gets diluted by its use. For example, we might publish an event to kick off a long running process or to defer a processing or some kind of a workload to out of band for the current. HTTP request. Or the current pro whatever process we're currently doing. We want to kick something off and have it processed somewhere else. If we're thinking about that use case, we might start thinking about things in, in terms of how we want to consume it. But we, I have found that we get much better understanding of the events and the event system. If we think of them as things that have occurred. Um, so much so that I have made this a rule in the systems that I've worked in events are named in the past tense. And when we do that, I mean, you do, you do run into some weird edges when you get into your, your framework. So some of the frameworks I've used to have had things like mouse down on mouse down. You, you know, you create a handler for that and then on mouse up. Well, a mouse click is really a mouse down and mouse up. In sequence. Uh, but if you want to do something, when the mouse, when the person clicks the mouse down and then they do stuff and you want to make sure this happens before they let it go, then you can. You could hook some, some of those events in the, in the UI, but in general, when we're, when we're talking about events and an event driven distributed system or an event enabled distributed system, you are talking about things like. Purchase completed card, charged email, sent, uh, address updated these. And these are things that have been happened. They have happened. And now other parts of the system. Are being notified about the thing that has already occurred. Yeah. So that they can then respond to it.

[00:05:33] Allan Stewart: Yeah. And I have liked the patterns that some of the patterns that we've used. So first of all, giving it that name in the past tense definitely helps. And as just as soon as you've named it, like it starts giving you a handle to start thinking about it. So if I, if I tell you payment failed, then immediately you start to get, now I know I got to go.

[00:05:52] Dave Adsit: Yeah.

[00:05:53] Allan Stewart: You get a vibe for, oh, okay. Well. What should we do? Um, yeah. And that we need to go and do something about it. Um, one of the strategies that we have also used in a distributed system is building into the name information about, uh, where this message came from. Um, because you start getting into a bigger system and, um, there might be similar kinds of things happening in different parts of the system where people start using the same name. And so it's. It's kind of like, um, name spacing for your events that you can give them, give them a name that when you see that the full name of the event, usually only part of the name is really like the thing that happened, but it tells you from where it came as well.

[00:06:42] Dave Adsit: Yeah. Definitely found that valuable to say, you know, the, the email sending service dot email sent dot V 12. Right. There are things in there. There, there are three pieces of information that make working in an event based system very, a lot easier. One is the source of the event. Of course, you need to know what the event is, what event occurred. And if you are not a perfect designer who gets everything right on the very first pass, and you actually have to iterate on your systems over time, you might also need to have a version number on your event so that people know. Hey, I am consuming version nine of email sent and the system is now publishing version 12. I wonder what's in there. I wonder if I need to move forward. I wonder if they're going to keep publishing version nine indefinitely. Uh, and honestly, in most of the systems I've worked in, we've had some pretty strong rules around that as well. We say, Hey, you have to, you have to keep publishing V previous. Even if you've moved on, you can have V next while you're developing. Yeah. It. So at most you've got V previous V current and V next. And so you need to go and tell everybody who's consuming V previous, Hey, a new one's coming. So you better move on to our current one before we drop the old one. You know, depending on how big and complex of a system you have and how many rules you need around the, you know, the, the architectural middle space, right? All that space in between all the services that you have.

[00:08:21] Allan Stewart: And some of that metadata could go in the. The body of the message, but I kind of like making it as part of the name of, of a thing like a, like a topic or a queue name or, uh, the, those high level things that you can actually see when you start looking at whatever your messaging system is. So that, you know, yeah. Um, you have an idea of what's buried in there without having to like grab a message in flight to inspect it, to see what it was and where it was going. And, uh, and, and the same thing with, uh, with queues. Uh, in the past, we've. Used sometimes, um, like fan out topics. So you can publish a single message and a copy of that message gets sent to all the subscribers and the names of those, uh, subscribing queues would tell you who to go talk to. So if, if a queue was starting to fill up with bad messages or there's a dead letter queue that is getting bad, you could look at the name of it and know which team to go and talk to.

[00:09:20] Dave Adsit: Or if you are responsible for the email sending service. Yeah. Yeah. That somebody is on a very old version of your email sent message. They're still subscribed to V9 and you are trying to publish V13. Is a good opportunity. If you have, if you have exposed that data in your messaging system, it's a good opportunity for you to go track down the team that you need to coordinate with on that upgrade. So if an event is not named in past tense, what is that? I mean, besides being incorrect, according to our architectural. Decision records that we would give to teams. What is that? Is that like a command, a passive aggressive command? Um, if you were to public, if you, if you were to say, Hey, email sending service, send this message at some point in the future. Is that, that's not really an event, right? That's more of like a, it's more of a command broadcast over the queuing system. Right.

[00:10:17] Allan Stewart: Uh, yeah, I think about these as passive aggressive commands because you didn't actually. Contact right. So like, because it's, it's messaging, there's, there's kind of a built in, um, latency concept, right? Like I'm going to publish this message. It may not get consumed when I published it. And so instead of, instead of calling up, you know, the, uh, the email sending service and saying, Hey, I need you to send this email via like an API, right? It's like a phone call to them, so to speak. Instead, we're just going to leave a note on their desk. Okay. Be like, by the way, I needed you to, I usually email these TPS reports.

[00:10:58] Dave Adsit: Uh, the, the email requested V2 message or, you know, event, right? Like, okay. Email requested. That's a very passive aggressive form of communication between services. Right. Yeah.

[00:11:14] Allan Stewart: And I, I generally tend to avoid those wherever possible. And sometimes it's, sometimes it's difficult to figure out. So. It will depend on how your system set up, right? So if your email service is smart enough that it knows how to concoct various different kinds of emails and structure them based on data that it's collecting, and there's a team that really owns that like outbound messaging, then maybe they don't want to, uh, you know, please send this email, send requested message. But instead what they really want is. An event like, you know, payment failed, like we talked about before, or payment succeeded. And then they will know like, oh, okay, great. It's got this payment ID. It's got this order ID. We're going to go send them the email that says your stuff is, has been paid for and shipping will be happening shortly or whatever it is. But in some other cases, the email sending service might just be like a facade over like whatever email provider that you've got, in which case it needs. Yeah. To know more, right? Like me being a part of a different part of the system, I need to craft the message and tell the sending service what to do with it. And in that case, it might not be a good fit for events. Um, unless you have some other, some other mechanism, because those passive aggressive events do require you to get into more of a saga style pattern in order to make sure that it happened. So you can say, Hey, I put this. Passive aggressive command there. And now I have to wait for some other response event to make sure that they did the work that I told them to do. And it's personally, I, I like to avoid that wherever possible.

[00:13:04] Dave Adsit: Well, and I think part of the reason that we fall into those, uh, those patterns or anti-patterns, I don't know, they're, they are patterns, right? Part of the reason that we'll fall into that. Passive aggressive command over the queue is because. We are looking at the queue as a single purpose tool, right? We say, Oh, events are published over queues so that they can be picked up asynchronously by the consumer when they're ready. Well, I want to put email at the end of a queue because I don't know, like maybe sometimes we have to slow down because our reputation is low or the email sending service that we're using externally is down or whatever. And we need to let stuff queue up. So instead of email. Hitting an API and hitting and executing a request, a synchronous request against an API, because I want this to queue, I'll just put it in the message broker that I already have. And I think that that is, that's a mistake. I think that if you have a messaging system for event handling, you should use it for the types of events we're discussing that are past tense. There are events that have occurred. You know, the, the payment has failed right now. Yeah. So if I want to put all the rules for email sending into that email sending service so that it decides when to send a message, then it would listen for that event. The payment failed and then send an appropriate email at that time. And if I want to flip that around, make my email sending service a little bit dumber, a little bit easier to interact with. I'm just going to expose an HTTP API. And one of the messages on there is send payment failed email. And it takes specificity. It takes specific parameters. And then behind the scenes, it could potentially drop that into its own internal queue. That's part of that service, but is not part of the overall event broker service that we're using for event communication. Right. Queues can be used in a lot of different places. Just because we put one up in one place doesn't mean that's the only place we should have it or use it.

[00:15:11] Allan Stewart: I think that gets into this idea of orchestration versus choreography. We talked a little bit about that in the prior episode. But. In the context of strategy, I think that orchestration is more of that command and control style where or even the passive aggressive commands where you're saying, I'm trying to orchestrate. If you're using your events as your orchestration method, then yes, you can do that. And there are patterns like the saga pattern that you can use to work that all out. But I kind of prefer the events as choreography. When you're looking at a bigger system. And we want to do, you know, two, three, four different things that are mostly unrelated. And we want to decouple them so that they're not all together. And so having that past tense event and you say, hey, here's the choreography. When this happens, everybody do your thing instead of trying to. To put it in, you know, the send a command event and then there will be a response event. And then you listen for the email has been sent event. And and you're trying to keep track of it back and forth. It can be done. It can be done. There are patterns for that. But I like what you're saying that if you break up those concepts and say, we're going to use our. Our system level events for choreography, then you can use you can still use cues for other reasons. Just don't use the same cue or maybe even don't even use the same cueing technology just so you don't accidentally start sending messages as passive aggressive commands that somebody else starts listening to.

[00:17:09] Dave Adsit: Well, and that's that's exactly right. If you're looking at a system that is choreographed, you're looking at having basically the central brain with a bunch of. Yeah. Subordinate components around the edge. Right. So basically it all has to live in that central brain, which is your orchestrator. Right. It's the conductor of your orchestra. And so if you have one event that occurs and it comes into the conductor and the conductor says, well, now, based on this, I have all the rules for these five things have to happen. Now this central brain becomes complex because it has to. Track. Each. Of those five things. And ensure that they all happened. And then there is confirmation of each of them, et cetera, et cetera. And retries occur and think all of that. And if we work in a choreographed style, then each system is responsible for its own. Rules. We can take the complex business rules and break them down and spread them across different teams of smart, capable, capable developers and say, okay, you guys are responsible for all of the customer communication. You know, all the rules. You know that customers can opt out of. This to class of email, but not this class of email. You know that some customers want every single email as soon as it's generated. And some customers want a daily or weekly digest. You know that, you know, all of these rules, you know, that we have a business rule that we don't want to send a customer. More than one email per hour, unless it's a certain class of transactional email. And if I have all those rules and I've written them down in notion and I tell everybody, okay, whenever you do anything with email, you have to go read. And familiarize yourself with all of the email rules. Make sure that you don't introduce anything that's going to cause problems. That is potentially a harder system to build or understand than one in which we say, Hey, here's all the email rules. They're in the email code base. They are understood by the email developers. And everybody else just publishes events when things happen and the email people listen to those events and they know when to send mess or when to send off emails to customers.

[00:19:19] Allan Stewart: And this gets us into the whole concept of Conway's law and the socio-technical system, because how you're going to build the system is going to depend on the people as much as the technology. Right? So actually in this case, I think it's more about the people than the technology because you can, there is technology that you can let you do it either way. But if you, if you are trying to set it up in a choreography style. But then all of your team members. Are not broken up that way. You don't have an email team. You just have everybody sometimes works on the email service. Well, you're still kind of back to a lot of the same problem of, okay, what were the rules for this again? So figuring out the right way to break up your domain. And so you're sending events to different parts of the domain that need it. I think is an important thing to consider as you're designing those system. Level events, as opposed to things that you put on a queue, right? If you, if you're putting a message in a queue, potentially you could think about that as an event, even if it's just internal to one part of the system. But I like to have that distinction. Like we were talking about that. There's the system wide events. That happens at a different level than, oh, I just needed to do background processing on this thing. That's fully within my part of the system. And I understand it. And I own it.

[00:20:49] Dave Adsit: Yeah. We definitely use cues for multiple purposes in software development. Sometimes the things on a queue are tasks that need to be handled in order. Sometimes they're tasks that need to be handled at some point. Right. And sometimes they are actually events that are communicating something about the state of the system to other parts of the system. So as we talk about these different ways of breaking down a system and where you would put the logic or the rules. For how the different parts interact. I would say in general, it is a harder concept to wrap your head around to use a choreographed system. And I think that's because you're giving up some amount of control. Right. It's easy to say I'm putting all the control for this whole system in this one place. And that place is going to be the conductor of my orchestra. And so there's a temptation to do that. There's a lot of power that comes both in the organizational structure, but also in the we've talked about it before. The emergent characteristics of your system. If you go towards more of a choreography. I think that one of the other things that comes up quite a bit when we talk about the event based system. Is the source of truth for data. You know, if we've got data flying all over the place. What is that data's home? Where does it live? How do we define it? The question that comes up often is what is a customer? The customer might be a lot of things, right? The customer includes demographics and their address and or like their shipping address. And their contact info, including a phone number and a name and an email address and their payment info. And their order. Order history, et cetera, et cetera, et cetera. And so what is a customer and how is a customer defined and where is a customer defined? So when it comes to the source of truth, we have to think about source of truth in a distributed system differently than we would in a monolithic system.

[00:22:48] Allan Stewart: Yeah, I really like looking at the concept of bounded context from domain driven design. When I think about this problem. Because in the past I have lived the enterprise data modeling world. Where we're just going to try to figure out like the one true perfect representation of customer or user or student or whatever it is. And those just have so many problems. Don't worry, you guys. This time we figured out what an address is. And inevitably we haven't figured it out. And inevitably the system gets so complicated to use. Because it's like, oh, well, why do we have this field? Oh, because this part of the system cares about it. Oh, but it sounds kind of like the same concept that this other team uses. And they just stuff some data in there sometimes too. And before you know it, it's really difficult to understand the model or understand the data that's inside the model.

[00:23:52] Dave Adsit: I would suggest that if you think that you know the canonical definition of a customer or the canonical definition of an address. You go find yourself a copy of the EDI standards. Which we have been working on as an industry since at least the 80s. Probably longer. Probably since the first computer we've been working on the electronic document interchange standards. And go look up what is in a customer in that format. That we, you know, many smart brains over many decades have been trying to refine what is a customer. And you will find that it is quite bloated. In fact, I would go so far as to suggest that any time you are doing an enterprise data modeling exercise, you are falling for one of the classic blunders. There's a quote from an author that I really like, H.L. Mencken. He says, there is always a well-known solution to every human problem. Neat, plausible, and wrong. And if you know what a customer is, if you think you know what a customer is. I suggest that you are wrong. And if you think that you can solve this with enterprise data modeling. I would also suggest that you are wrong. So, Allan, if that's not the solution, what is the solution?

[00:25:12] Allan Stewart: Right. Well, at least the solution that I prefer is this bounded context concept. In which we say that within each part of the system, right? Like there's a boundary that you draw. It's kind of an imaginary. Imaginary line. Much like the lines between states or countries. Right? They exist. It's a real thing. But it's not painted on the ground. And similarly, oftentimes in our systems, there's not really distinct ways to break those up. But you need to make those more distinct. You draw this boundary and you say, okay. Within this context, a customer means this. It has a name and context. And over here in a different part of the system that deals with like shipping fulfillment, it might have more information about like postal codes and shipping address and something else. And another part of the system might not care about either of those things. And they're just like, I just want to keep track of all the things that they have rated. And all, you know, we're going to calculate the suggestions that we're going to give to the customer based on their preferences. And so they keep each one of those keep separate pieces of data. But ultimately, someone has to define who the customers are in the system and provide some kind of a customer ID. And whoever owns that is the source of truth for the system. And I recommend that you always find a single source of truth for every unique entity. Like anything that is going to pass between multiple places in the system. You need to know that it's like, okay, well, over here in the, you know, customers bounded context, that's where the source of truth is going to be. Or maybe it's not even that. Maybe it's in payments because nobody's a customer until they've paid us money. And so the payments team, they get to be the ones to define who the customers are. Now, the other teams might still do stuff with that customer ID. And they'll probably want to denormalize the data. They don't want to have to look in your table that you keep messing around with. And you don't want them looking in your table because they keep coupling to the data. And then you want to make a change and you can't because you have to orchestrate that change across seven different teams. Data coordination meeting.

[00:27:36] Dave Adsit: Change management board has to be convened. Exactly. Update the definition of customer.

[00:27:40] Allan Stewart: And so you denormalize that data. Which, again, it makes the system more complex. At a systemic level, it's more complex. But when you're working on each one of these pieces, those pieces get simpler. Right. Because they know, oh, well, I got the customer ID and I also asked them for the name. So now I know the customer's ID and their name because I care about that. But now I'm going to hang all this other data that I care about in my part of the system to that ID. And if they tell me later that they've changed their names because they got married, now they've got a new email address, whatever it is, okay. I'll copy that data as need be. And you do start getting into questions about caching and everything. Sure. Sure. Sure. Sure. Sure. Sure. Sure. Sure. Sure. Sure. Sure. Sure. Sure. Sure. Sure. Sure. Sure.

[00:28:45] Dave Adsit: Sure. Sure. Sure. Sure. Sure. Sure. Sure. Sure. Sure. created event is that payment system. And the payment system creates the ID and they own certain aspects or certain fields on that customer. But they probably don't own all of the customer. Like the email sending service we've talked about might be the source of truth for the customer email address. And so the customer would change their email address along with their other email preferences in that email sending service. And maybe they don't even publish that out because maybe it's not relevant to any other part of the system. Maybe it's the authentication system that owns both the customer ID and the email address definition because we sign in with email addresses and then we replicate that data across the rest of the system. Right. Right. And so what would happen then is. Whatever that source of truth is, would be the place that publishes events about creating, updating and deleting the entity as a whole. But then other parts of the system may or may not publish events about things that happen in those parts of the system. It might be that the customer demographic service that is used for tracking marketing data is where we keep track of the customer's address. And they may publish the address, address. And then the customer's shipping address changed, the customer's home address changed. And now if I'm over in the warehouse management system where order fulfillment happens, I might say, yeah, that's good to know I'm going to track that data going forward for this customer. But for orders that have already been shipped, I'm not going to update the address because that is something that has already happened. Those shipments were already sent to those addresses. And if we were to build a customer, you know, the, Yep. if we were to build a customer table that was consistent across the whole system, and we were to represent it in a relational database as a single fee, a single table and the address gets updated, well, we might now lose data, right? So that's one of the things to think about is like, how am I using this data? Maybe for the sake of the demographic system, I don't ever care about the history of the customer's home address. I only care about where they are now, but if I'm the warehouse system, I probably care about the history. If I am the financial system that is calculating the collection and payment of sales taxes to all the various jurisdictions where we are shipping products, I probably care about history. But if I'm marketing to a customer based on where they are now, based on their current location, you know, you moved from one state to another. If you moved from Alaska to Hawaii, I'm probably going to stop trying to sell you parkas and I'm going to start trying to sell you sunscreen.

[00:31:54] Allan Stewart: I've noticed that this kind of problem happens a lot in systems that I've worked in where I come in and I see how things have been used in the past. And oftentimes they're used in a very naive way, especially when there's a lot of normalization. It's like, Oh, well, we're going to put this data over here. And, and just like you say, we're going to update the address. And now all of a sudden the address that was associated with a bunch of previous orders is different now. And, and, and then there are bizarre bugs and you can't tell what was the state of a thing in the past when it happened. And so I think about that and, and, and think, yes, what we're describing here is a more complicated system, but oftentimes we need a more complicated system than the naive solution that gets generated in a lot of, And I see it repeated over and over. And so I, I believe that this is an anti-pattern that exists quite a lot where it's just easier to just use the data, which is everybody bind to the same source of truth. And so effectively you have created that enterprise data modeling. It's just less intentionally with less purpose because everybody's using the same chunks of data. And even if you're in a monolith and you don't need separate, separate events with like an event bus or a pub sub system, it can still be very valuable to break these things up and say, no, this was the address at the time that we shipped. This was the price and the name of the item at the time that it was invoiced. This was the, you know, whatever at the, you know, the email address of the customer at the time that we sent the email, all those kinds of things can be very valuable to, to separate out. And, and, and the more and more I find myself wanting to avoid joining tables in a database. I want to break it up based on entities. And when I start wanting to join a lot of information that tells me something, it's like there, there's some kind of a reporting concept most likely that wants to get out of here. It's it's reporting or it's to, you know, paint a list of data on the screen so that people can quickly find something that they need. And that's a different use case. Right? And it can probably benefit from having some denormalized data and you'll have to have a strategy and it might go along with your eventing system to keep that those things up to date. But breaking those up makes all parts of the system easier to, to use because you've lowered the coupling, the data coupling drastically.

[00:34:37] Dave Adsit: Well, and I'd like to point out that a lot of the things that we did for normalization were strategies invented under a specific context. We, we've talked about this before. I know we have my very first hard drive costs several hundred dollars and was 80 megabytes. I recently installed a two terabyte M two SSD into my brother-in-law's PS five. And it cost $85. So the data constraints that we once have for spinning rust, it was hundreds of dollars for 80 megabytes. And now for extreme, extremely fast SSDs, you're spending less than a hundred dollars per terabyte. It's we're not operating with some of the same constraints that we once had.

[00:35:25] Allan Stewart: Yeah. Right.

[00:35:26] Dave Adsit: Data storage is substantial, substantially cheaper, and we are often paying for more for compute. And so if we can do things to minimize compute and to, you know, even at the cost of duplication, that could be a beneficial thing for us. Yeah. In fact, you know, we're creating caches of data all across the system so that we have easy, fast local access to them versus making a network request to get them from some other system that has the, that is the actual source of truth for that data is, is tricky. You know, it's, it's a thing that we can do for optimizing performance for optimizing control. I will cash your data so that I can control the shape of it because you broadcast it in a certain shape and I needed, I need to read it in a slightly different way. That's fantastic for me.

[00:36:18] Allan Stewart: Yeah. Or I need different indexes on it or I need to join it with some other piece of data before I save it into my, my, my table or whatever. Yeah. I think that that's really important. And I think what I've seen in a lot of systems is that staleness is a lot more, it's much more tolerated than we seem to think in general. Yeah. You know, there's kind of this idea that's like, Oh, well, you know, they updated it. It has to be immediately. That update has to immediately propagate to the entire system. It's like, well, no, probably not. Most of the systems that I work in, they can propagate within a couple of seconds. So probably by the time that the user meanders over to the next page, it's already up to date. But in so many cases, I found that data that's a little bit stale is, is good enough.

[00:37:15] Dave Adsit: It's good enough.

[00:37:15] Allan Stewart: And there's only, there's only a few times where like, you know, they missed their payment and now we're kicking them out of the system. There's only a few things like that that really matter as far as here it is. This is the cutoff. And even then there's oftentimes like grace periods or things that, that can be done in a system without much trouble.

[00:37:38] Dave Adsit: Right. So I want to get into anti-fragility. And an event based system. But before we jump into that topic, I want to talk about a couple of things, characteristics of messages that we didn't really get into previously. One of them is we talked about a little bit about message brokers and message cues and how messages are propagated around a system. There are characteristics to those that are important to understand. Some of those message brokers or message cues can be ephemeral. The message is put on, it's stored in volatile memory. And sometimes we say, no, this message is important enough. I need it to be written to a durable store until you have confirmed delivery. And the reasons you would do that are because you are making a trade-off between reliability and speed. Right. In memory messages are going to be delivered, are going to be processed faster, whatever that means in your system, because you do not have to pay the extra cost of writing it to the disc. However, you do not pay the cost of writing it to the disc and something happens. You've got a multi-node message broker and the message is only held on one node if it's ephemeral. And that node goes down. Well, the whole broker might continue to operate, but you may have lost a message that was important to you. I would say, think about how important reliability is for that message and then use that as one of the determining factors for whether or not you make it a volatile or a durable message. Right. If you're doing things like tracing data or, you know, analytics data inside of your web application, it may not matter if you keep every single one. In fact, you may choose a sampling strategy regardless, even if you did have the ability to get every message reliably. Right. Just because of the volume of data. But if you are doing things like payment processed, that might be really critical or even more important, payment failed. You know, an order was created and the payment failed. Right. It might be critical that that gets processed reliably so that we can contact the user and give them the opportunity to correct that problem before we kick them out of the system, as you mentioned. Yeah.

[00:39:56] Allan Stewart: And some of these systems, if you use something like Kafka, they take that durability to the extreme. It's not just we're going to save it until the, you know, until it's confirmed, until it's confirmed, delivered or confirmed, processed. But we're going to also keep it forever. Yeah. Or until some later compaction event or much, much longer terms that you can use. And I think there are different characteristics like this that you want to examine because there's a lot of systems out there. And especially now there's a lot of hosted services. You can go to your favorite cloud provider and ask them for a queuing system. You can ask them for a pub sub something. And they have different flavors and different guarantees about them. You know, another one was the delivery guarantees. We talked a little bit in our prior episode, you know, can have at most once or at least once delivery. And another one is order of delivery. Does it need to be first in first out or does that not matter to you? And so these are the kinds of things that you need to look at as you're trying to figure out what kind of delivery you're looking for. What kind of broker or system you want to use to transport your events?

[00:41:15] Dave Adsit: Well, yeah, that, and those definitely affect the overall reliability and performance of your system. If you say, Hey, you know what, these have to be done first in first out. Well, now all of the nodes of my broker have to coordinate around receiving a message, which means that publishing messages gets slower. And if I say, Hey, these have to be just, I want, I want delivery guarantees. I want to, I want you to ensure that all of these are delivered at least once. That's probably the easiest kind of guarantee for the broker to comply with. Because now you can give me the message, one node can receive it, and now I can say, got it. And now we can move on. I've got it. The broker has it. It's written to disk. It's durable, whatever. And then now I'm going to deliver it to the consumer. And if anything goes wrong, I'll just deliver it to this consumer again. The hardest guarantee, and in fact, I think it may be mathematically impossible, but the hardest guarantee for a system to give you is in order exactly once delivery. Because now I have to have coordination between all parts of the system that it is, you know, now I'm falling into some of the challenges around CAP, right? The consistency, availability, and partition tolerance. I may not be partition tolerance if I have to have that level of consistency and availability. Right. So,

[00:42:37] Allan Stewart: reliable delivery then is one of those anti-fragility requirements that you got to consider, right? Like if you want your system to work reliably, well, you're going to need to know that we do deliver messages. Because if some messages just don't get delivered and they disappear, that could be a problem, depending on the kind of message, like you were saying before. If it's some kind of like tracing analytics thing, maybe it doesn't matter. Maybe you were going to sample it out anyway. But it, but the, the payment messages, those ones do need to go through. Another one is item potency. And item potency is all around, the term comes out of like mathematical functions where you say, hey, if, if you run this function with this input, you always get the same output every time. And for an event-driven system, item potency is kind of that similar idea that it doesn't matter if you got the same event once. If you run this twice, three times, you're going to end up with the same result. And so sometimes that, you know, like our example with the email sending service, that can be pretty important that you need to, it's about how you handle your message. Because if you accidentally get two of the same message, because the, the queuing system that you used sometimes sends you a second copy, or because the team that you're working with, they, they had a problem on there. And now they're republishing some messages that had happened before. And if you just firing off an email blindly every time, well, that's not going to be so good for your email reputation.

[00:44:18] Dave Adsit: Well, and if you look at it from going back to those passive aggressive commands, like payment requested, payment requested. If I get that message twice and process it twice, the same way, that could be very detrimental to my business. My customers may, might not like it very much, especially if I get a message caught in a loop. And for some reason, I got 50 copies of it. That one customer is probably going to cancel. Rather than pay the thousands of dollars that. Right.

[00:44:45] Allan Stewart: But for the two items that I ordered.

[00:44:48] Dave Adsit: So, so we do have to have item, we item potent message processing, right? So if I say, Hey, an order was created, the payment was processed and I'm the fulfillment center. I'm like, okay, so it was full. Now I can fulfill it. And then I get payment was payment was processed again. And payment was processed again. I'm not going to fulfill it multiple, multiple times. Right. I'm going to say, Hey, I've got this order. Payment has been processed. Fantastic. I can release it to shipping or to packaging. And I'm only going to do that one time. I'm not going to keep releasing it to packaging and shipping, even if I get a payment process message more than once. So that takes us to our next concept, which is, you know, data caching. We've talked a little bit about data caching, right? I might have copies of data that I don't own. Yeah. If I am the fulfillment service, the warehousing service, I might not own the order. The order might be owned by the order management system over there on the other side of the world. And that system says, Hey, I own the definition of this, this order. The order was created by me. I'm the only one who can add items, line items to it, remove line items to it, whatever. I, and I'm the one who, when I make changes to orders, I send off, I publish an, order updated. And now the payment or the, the payment processing system says, okay, the order has been updated. I either need to charge the customer or refund the customer or whatever. And then when that event gets published, that it's been charged, the warehousing system says, okay, I'm going to ship it. And you're like, well, what am I going to ship? I know that payment was collected for order one, two, three. What is the definition of order one, two, three? Well, luckily I was already listening when the order updated was published. And I, I have created a local cache of that data. And so that's your ideal scenario, right? All of those things happen in order and I have local caches so that I can look up the data that I need to do the right thing at the right time. And if something goes wrong, I have to have a fallback mechanism. So if I get, if I receive a message that order one, two, three was paid for, and I don't know what order one, two, three is. Now I have a dilemma. I need to have a way of healing my data so that I can do my job, which is fulfill this order. So one of the strategies that we've used effectively in the past is to expose an API to get a fresh copy of that data, right? So if, if I can then say, okay, well, typically I don't have to do this, but in this case, I'm going to go ask the order system. Hey, what is, what are the details of order one, two, three? I see payment. I mean, I don't even need to tell you that I know the payment was made. I just need to ask you, what are the details? And you tell me the current details. And now I can write that record, write the payment process data, and then I can go fulfill this order.

[00:47:52] Allan Stewart: We talked about in the last episode, the differences in definition that Martin Fowler gave between event notification and event carried state transfer. And I think this is an example of to lean towards event notification, because there are those times where I don't have the data that I need and I need to go and get it. And so I'm going to call an API and get the data. And if I'm going to have that built into my system anyway, and I have to understand your API and how to use it, well, then I don't want to also have to understand how you put all the data on your message if you're doing event carried state transfer. Because then now I'm going to have to So now there's two different, potentially two different ways that I'm getting data from your system. And I just like to have one because fewer is simpler and easier. And so I've kind of come to really appreciate event notification as that mechanism. It also makes item potency a little bit easier in handling some of those entity updated style messages, because I don't have to worry about what order they got published in, or if you published it twice or three times, or 17 times. Each time, I'm just going to ask your API, oh, so I heard you updated that. What's the latest data? Okay, great. Got it. And then if I've got stale data for whatever reason, or I'm not sure, or I had to clear my cache because there was a data corruption problem or something else, it's okay. Because when some other message, even though I didn't get customer updated, but I got payment processed, well, I can go and get the customer data that I'm missing in a unified kind of way. But again, it depends on the type of system that you're building and the type of messaging that you're using. Because if you're going to use something else, like if you're going to use Kafka, well, there's no point in that because Kafka will have literally all the data. And what you need to do is go and replay the stream. And that's a different strategy. It's very viable, but you just have to understand what the strategy is that you're going for

[00:50:01] Dave Adsit: and use that. Well, and one of the things I like is that if I'm asking your API for the data, one of the pieces of information that you can give me is a cache duration. Maybe you know that customer data is not very volatile, or customer data is volatile and order data is not, or something, right? And you're like, hey, this could change in the next five minutes. So if you need it again in five minutes, you should mark down that this is expired after some timestamp. That doesn't necessarily mean that you need to drop the data from your table. You can still keep it in case maybe you can't. Can't get an updated data. We talked about how stale is often better than null. Right. I would say that I have been leaning towards the event notification strategy more often than not as well. I feel like the event carried state transfer is a cache optimization strategy. And if we go back to our fundamental principles and guidelines, we should not do premature optimization. Because they create complexity without value. So when I have a problem where I need, I don't know, like we can't scale your API to meet the needs because everybody's hammering on it all the time. Maybe now I can start, I choose to start publishing some data in those events, in those messages, so that you can say, hey, this data is current. You don't need, as of the publishing of this event, this data is current. And then you don't need to, you don't need to call my API to get the current state of the data. Because I need to take, I need to reduce network traffic for some reason. I'm trying to work on a cache optimization strategy in my distributed system, which if that doesn't sound challenging to you, you haven't built a cache and validation strategy for a distributed system yet. But that is, again, that's an optimization that I may not need to do. And when I do get around to doing that, kind of an optimization, I need to become very familiar with things like vector clocks, because time, timestamps, those can be problematic on a day in any kind of distributed system.

[00:52:20] Allan Stewart: And I think it's also important to go back to the context of what you're building, right? So we're talking about the systems that we've worked on. It has tended to be that that event notification works really well. And that event, you know, I think, is a premature optimization. Yeah. And so, I mean, I think between the two of us, we've seen enough systems that we can make a little bit of generality and say, for many web-based SaaS products, this holds true. But that's not to say that it's always true. And you might have some other kind of constraint. You might really want to keep all the events that ever happened in the system, because you're going to do something else with it. Like you are doing some kind of, you know, you're doing some kind of guarantee around, I don't know, transaction processing or maybe stock market something. And you need to be able to actually go back and repeat all the data at a later point, right? Like maybe all these events happened, but we want to explore alternate scenarios, or we don't actually clear funds until we've run it across three different variations that agree. Or it depends on what you're doing, because context is still king.

[00:53:40] Dave Adsit: Context is still king. Well, I will say we've been talking a lot about some fairly complicated things you need to consider in your event system. But that's not to say that events need to be complicated. Having an event-driven system does not need to be phenomenally challenging. If you put in place a few basic rules, if you say, hey, every event is past tense, every event is next. Every event is named in a way that you can tell where it was published from. Every event is versioned. Every event uses a common format for encoding. All the messages are JSON, you know, plain text, UTF-8 JSON. And if we say all events are going to be PubSub or, you know, publish, subscribe, they're multi, they're fan out, they're multi, multi-consumer, or they're capable of being consumed by any number of consumers. We're going to use a common message broker. For the whole system. If you do all of these things, you can make it simple to use events for communication in your system. And it can unlock a lot of power and enable your dev teams to work independently in a way that they otherwise would not be able to do. And again, I say teams because I think of this as a strategy for a growing and scaling organization. I've used it on a, you know, we started introducing events on one team. When we had about 12 developers. And at the peak, we had 35 dev teams with over 150 developers building independent parts of the system. And we were still using the same messaging strategy all the way to that scale. And if we were 3000 developers, we probably would have needed a different strategy again. Right? Right. So where, where are you and what is your, what is your system look like? What is your socio-technical system look like? And that can help guide you on some of the things that you're doing. Some of the decisions you need to make in order to use events, but it doesn't have to be hard.

[00:55:42] Allan Stewart: I remember a blog post I read a long time ago from Etsy. They were talking about how they were instrumenting their system and making it so that they could understand how often do things happen and basically improving the observability of their system. And this was, this was before a lot of SaaS products and tools were on the market to, to, to, to implement. So yeah. So yeah. Maybe we'll want to know this. And so they started gathering data just in case. And I think events can be the same way. It's going to take a little bit of effort to get going and to come up with those rules. But once you've codified those rules, and oftentimes literally, you've codified them into like a shared library or something, then it can become extremely easy to use messages to the point where you can say, hey, yeah, it's not a problem. This event happened and you want to know about it? Yeah, I'll add in an event. We will publish an event whenever this happens because it's easier to do that than it is to invent some other way.

[00:57:14] Dave Adsit: Yeah, we need to make it easy to fall into the pit of success.