Crafting Code Podcast

[00:00:16] Allan Stewart: Welcome to the Crafting Code Podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately I've been thinking about the differences between sci-fi and fantasy.

[00:00:30] Dave Adsit: I'm Dave Adsit, a VP of engineering, and recently I've been thinking a lot about the challenges of translating straightforward level progression systems from games into the real world of career ladders and personal development.

[00:00:45] Allan Stewart: Our topic for this episode is event-based systems. There are many ways to build a software system, but when and why would you want to use events as part of that?

[00:00:57] Dave Adsit: That's a very good question, and I think that there are a number of reasons why you would want to use events. So events are an alternative to request response APIs. They can be used in addition to request response APIs. You can use events to move long-running processes out of band, and you can definitely use them to decouple processes from a single trigger. So if we go back to those and talk to them in order, I mean, we use HTTP. We use TCP all the time, right? And deep under the hood, there are actually a whole bunch of one-way messages that occur. And you could think of that as, you know, one-way events, right? We're propagating a message one direction, and then a response comes back, which is why sometimes when you say something to a programmer, they'll say, ACK, capital A, capital C, capital K, which means acknowledge versus I don't like what you just said, I'm responding negatively.

[00:02:00] Allan Stewart: Yeah, I think that there's been a long bout, I don't know, maybe 20 plus years where request response was a primary way of building applications, certainly web applications, but there were also a lot of interactions with client side and mobile apps where building out that kind of response over HTTP is kind of a... Yeah, I think that's a natural thing to do, right? It seems like this is the way that things are. If you go back far enough to some of the desktop applications and other kinds of mobile apps, there's certainly a lot of areas where event-based architectures have been in use, right? It's certainly not a new concept, but I feel like in the last 10-ish years, it's caught on more and more as people are working in software as a service companies. They've got... And they're thinking about events and thinking about how can they decouple their system. And so, yeah, you can use them as a way to communicate beyond just that HTTP layer. And I like that one. And we say, you could just use it instead of, or you could use it in addition to APIs that you already have. But there are some interesting differences between them.

[00:03:28] Dave Adsit: Well, and the two that come to mind for me most often is, you're right, when it comes to desktop applications, the desktop user experience toolkits that I've used are almost always event-driven. You know, it's like when a user clicks a button, do something. When a user enters text into a field, when they exit the field, you know, when the user does things, we respond to those things. And that's the type of user... UI or UX that we expect on desktop apps. And more and more commonly on web apps as well, right? So if we don't know when the user is going to click the button, we just have to wait for that event to occur and then respond to it. And then we may choose to do a whole bunch of things based on that event being raised in the system. In fact, I would say that was one of the primary drivers of early Windows. Based toolkits like Visual Basic was to create a user interface tool that made it easy to build what had been very complex UIs before that. You know, it was like hooking those events and looking, monitoring system interrupts and things like that was very challenging before the advent of tools like Visual Basic.

[00:04:52] Allan Stewart: Yeah, I definitely remember a lot of PubSub. We had a lot of style systems coming online a while back as a way to create some of these bigger systems, especially as things were starting to move into the cloud or getting away from dedicated data centers. And I think that that initial push, you know, it's like, oh, well, we've been doing this one way. And now we're going to move to this, you know, PubSub event broker kind of model. And it's easy to just get sucked into that and say, oh, well, we're going to do this instead. And so then there's a lot of learning about like, how do you handle these handle things? Because anything that you can do as a request response, you could do as an event or vice versa, right? Like there's you're just kind of turning it on its head or turning on its side, I guess. It's just a different perspective of how you're looking at it. And so I think it's important to be thinking about are you using it instead of or in addition to a regular way of communication? But there are other ways to use it. One of the things that I like to do with events is have it as a mechanism for moving long running processes out of band or decoupling multiple processes from a single trigger or inciting event. Right? So in the system that I work in right now, there is a event that can occur or something that happens when a user clicks and does something that then wants to update projects. And thenededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededed us some interesting abilities to retry, to not have to have it happen all at once. We don't have to wait for a third party service to complete a request before we can respond. And sometimes it's just nice from the code to decouple all those things that were happening, right? Like the user clicked this one button, but there are two or three distinct things that have to happen that really aren't related to each other at all. But except for this was the thing, this was the event that triggered it. Yeah.

[00:07:21] Dave Adsit: Well, and I think that one of the things that's really interesting is that we're talking about events from vastly different perspectives, right? Events can be a thing that happens inside of a single process in a single application on a desktop or events can be a tool that we leverage at scale across many machines in the cloud, building vast distributed systems. Yeah. Yeah. Yeah. Yeah. And we're talking about using them in a distributed system. We have a concept of like, how are we coordinating the event and the response to the event? And what that can come down to is a typical concept, a concept that comes up a lot is, are we using an orchestration style or a choreography style? You know, are we, do we have a central conductor that is directing the traffic as you will in an, like an orchestra with a conductor that's telling the oboes to get louder and the drums to get quieter, whatever, I don't know. Or are we doing more of a choreography where every individual dancer has their part to play, but they are not being centrally controlled, right? And so those are things that come into play as we start building bigger systems with events as part of them, right? Where does the control? Where does the control come from? Where does the locus of control lie for this system? Is it with a central hub or is it with each individual actor in the system? And the way that that decision will have a huge impact on both your distributed system and your socio-technical system that supports the distributed system.

[00:09:05] Allan Stewart: Yeah, absolutely. Right. We're looking at one of those Conway's law. Problems, challenges. Yeah. And ramifications because how, how those things work together is going to also mirror how you talk to each other, right? So if, if I have more of a orchestration set up and I am going to publish this event, that means, okay, now other things have to happen in other parts of the system. Well, then I might have to do a lot more coordination work to talk with the other team and say, okay, this is. This is what the event or events are going to be. And this is how we expect you to respond. And this is what, you know, what guarantees we're going to have between each other. But if it's a more of a choreography, then that might not matter. It might be that one part of the system, like the part of the system that I'm working on. I just say, Hey, sometimes this thing happens and other people want to know. So I'm going to let you know when it happens. And what do you do about it? I don't know. I don't know. I don't care. Maybe. So Dave's part of the system is going to react to the, the fact that an event occurred, but it may not be something that I have to know about at all. Or I might have like some cursory knowledge about it, but it's going to be handled right. Just like the dancers on stage go, well, they know what they're supposed to do when this happens, go and do it. I trust that it will happen.

[00:10:40] Dave Adsit: Well, and to use your earlier example from your. System where someone pushes the button and then several things happen in an orchestration type of a system. I might, as, as the event handler for the button press, I might create a new task to update the project and a new task to send the payment request and a new task to create the PDF and then await all. And now all of the problems of error handling and retry and timeouts. Et cetera, are my responsibility as the button press handler. And in more of a choreographed system, when the button is pressed, I might shout into the ether. Hey, this button was pressed. And then a listener somewhere will say, oh, well that when that button is pressed, I know that I need to update a project and another listener somewhere else that may or may not know about the, the first one says, oh, and I need to request, I need to request payment. The button was pressed. I. I need to request payment now. And the third listener might say, oh, when the button is pressed, I create the PDF. And so all of those things could happen completely independently and they are responsible for their own air handling retry logic, et cetera. Like, you know, you've received the event. Now you make sure that you handle the event as it was intended in your space. And so there are trade-offs between orchestration and choreography in orchestration. You can. Create some kinds of guarantees around things, right? You also have higher complexity at the place where you are doing the orchestration from in choreography. Each individual thing may be simpler, but the system itself may be overall be more complicated because now I may have three ways I handle retry logic. I may have three ways I handle, you know, long running processes or timeouts or whatever. And then there's the further. Problem of in orchestration. I know when everybody's done because I've been waiting for them all to get done. And in the choreograph system, I may have to listen myself for a collection of events. To decide if everything's been done so that I can update the status or maybe not. Maybe I just have a checklist on my screen. That's like, as each thing happens, I turn it green. And some of the things may not ever turn green because those things didn't didn't finish. It can get very complicated. And, and I have. לק לק

[00:13:27] Allan Stewart: לק לק לק לק לק לק לק of the tasks to be done and you're controlling it all, you can give some guarantees about, okay, I know when this stuff has been done and whether it's going to get done. But in the choreography world, you might not know. And sometimes data propagates across the system in some interesting and sometimes unexpected ways. But I think it's important for us to consider that that's not necessarily a bad thing. There are many, many cases where having data eventually consistent is fine, right? Like if in using the application, that inconsistency of data is a problem for the user because they are on a couple of different screens that tend to get used together. And one of the screens says this and the other screen says that, and they just don't line up. Well, that could be pretty bad. And then they won't want to. So you have to continue using your product. But there are plenty of other situations where like exact numbers, exact counts don't exactly matter, right? Like I think about banks and your bank statement. And oftentimes on an online bank, it'll say something at the top. It's like, oh, well, here are pending transactions and we're keeping track of it. But what's your balance? Well, it's probably about this much, but we're not going to say for sure until we've closed out the day. Because a daily roll up of, you know, what, how much money do you have? How much money was in your account at the end of the day? Well, it was good enough.

[00:15:20] Dave Adsit: One of the things I was thinking about is that we as software engineers use terms in a different way, possibly a more precise way than people in other parts of the business, right? So if I say to someone, or if I hear from someone, this has to be calculated in real, real time. Well, now all of a sudden I'm building a very complex system in my head. But if I ask the follow-up question of what does real time mean to you? I might hear something that sounds silly to an engineer like, well, within 30 minutes or definitely by the end of the day. Yep. Okay. Well, now all of a sudden I'm building a much simpler system. If I have to do the calculation by the end of the day, I'm not nearly so worried about the constraints of hard. Okay. Time and system errors when things don't happen instantaneously. Right. One of the things that we've talked about extensively in the past is distributed systems turning into distributed monoliths. And there are times when that feels like the inevitable consequence of using kind of an orchestrated style, right? If I've got one central controller that is talking to every other part of the system and controlling which parts of it interact. With which other parts of it, et cetera, that can be come a very coupled mess very quickly. And in a choreographed style, there is a possibility that not only do I not have a central place to go to find out what the logic of the system is, the system may start to have some emergent properties that no one ever predicted and can't replicate or diagnose or trace. And so, you know, there's, those are some things that become very challenging when your system is, And I would say probably most of the distributed systems I've worked on have actually ended up being distributed monoliths of one type or another. There's very few times when I've worked in a system where you really can deploy some of the component parts without deploying the whole. Yeah. You have to be very careful. But I've also mostly worked in distributed systems that rely on APIs and events can help substantially decouple some of the components.

[00:17:38] Allan Stewart: And speaking of coupling, events bring us to a interesting type of coupling that we don't always think about. And usually we're thinking about like the coupling between bits of code within the system, right? Like how many things does module A know about outside of itself? Or does this class, how many other classes does this class know about or depend upon? But another type of coupling that is pretty important into a system is temporal coupling. And events give us a way to decouple things that otherwise would have been together in time that don't always necessarily need to be right. So, so there are a couple of different ways to think about temporal, temporal coupling. One of them being just that some things are together in code because they just happen to be together, right? Like the example I had before where this particular button that they push that approves the thing. Well, that's going to, they just approved a payment. They've just approved a PDF. They've just approved a project. All those things have to get updated, but they may not, they may not be very related as far as what needs to happen. And so you can have a high degree of that code coupling if you're doing it in an orchestrated way. But if you're. If you move to a choreography and put some events into place, well, that might break that up. And now when those things happen might be now decoupled from when it from the event that triggered it. And this can be especially nice if you are working in systems that have a lot of APIs, right? Like when API is the only language that you have to communicate with another part of a system. Well. Then you have to do all these, you know, request responses and how do you handle errors and all of that. And it has to all happen at the same time. And, and you are dependent on that other API being available when you need it. And if it's down for whatever reason, then you have another set of problems that you have to figure out how to deal with, which is just different from when you decouple those temporally, that, that it doesn't matter when, when that event happened, only that it did happen. And now we're going to react to it.

[00:20:09] Dave Adsit: Yeah. And I will say, um, in my career, I learned the definition, various definitions for temporal coupling in an uncommon order. I think you mentioned that the more common or traditional definition of temporal coupling is that you put the code together because the things happen together in time. And that's unnecessary. Uh, the definition that I learned first is the. Probably the second definition. Or maybe a definition. Most people don't even ever think about is that the things are happen in a linear order because we write code linearly. Right. And, and we've talked about given multi-process or multi-threaded type of systems, stuff doesn't have to happen in the same order every time. And it certainly doesn't have to happen in a prescribed order. If I throw out an event, the things that react to that event could happen in all kinds of orders. Yeah. And that is one of the forms of temporal decoupling is like, oh, well don't rely on, I mean, I, I honestly find this happens in test libraries, test suites quite often. Don't rely on things happening in the same order every time. In fact, one of the failure modes that we've talked about in unit testing, or I guess more integration testing for a long time is assuming that your test will run in the same order every time. So if you have test one, set up the database. For the following tests, test two, manipulate the database in a certain way and do assertions test three, manipulate it in a different way and do other assertions. If those run in a different order, all those tests will fail. And that's not good. Um, but we need to think in terms of an event and system about what things need to be done in certain orders and what things kind of can be done in whatever order it might be that in this original discussion, this original simple example. When the button is pressed in the approve, the proposal is approved. Maybe we have to update the project before we kick out the event so that create payment and create PDF can happen in whatever order. But if the project hasn't been saved, we're in a bad state, bad state, because we're going to create a PDF on, on a stale data. We're going to request a payment on incorrect stale data. We could be in all kinds of trouble. If there actually is a dependency on the order, if there is a need to do things in a certain order, well, you gotta be mindful of that in events. You can't rely on events. You can't rely on the responses to events occurring in a specific order. If you need them to happen in a specific order, you may need more events. You may need a, Hey, somebody pressed the button event. And then you may need a, Hey, the, the, uh, project has been updated event. And now I can start working on. The. Request a payment event and the cur and create a PDF event.

[00:23:10] Allan Stewart: Right.

[00:23:10] Dave Adsit: They can respond instead of re instead of responding to the button press, they could respond potentially to the project has been updated.

[00:23:18] Allan Stewart: Yeah. I like what you're saying there, especially around how programmers over years have been trained to think in a particular linear way. Like this needs to be done before that this database transaction has to clear before we can do other thing. Yeah. Um, and so I think we lean into that pretty heavily as, as programmers sometimes, but if, but when we take a step back, we realize that things don't necessarily always work that way in real life. Um, there's, there's a paper that, uh, I guess an article that Gregor hope wrote one time about, uh, your coffee shop doesn't use two phase commit, because a lot of times we think that these things are very important. It's like, oh, well, you know, I'm not going to use it. But the, the database has to have complete data integrity of all these things. We can't possibly do things out of order, but in real life, there are lots of scenarios, right? Like the coffee shop where things might happen in different order or, or a fast food place. Right. We, we tend to think about it in terms of, oh, well, we have to have payment. And if we get paid, then you can put the fries into the hopper and, and cook them. But that's not how a lot of fast food places work. They know that the lunch. Rush. Is coming. And so they dump in one or two baskets of fries and they get them going because they, they know that usually they, there are enough customers that they're going to use up all those fries plus some more. And so instead of waiting for the payment to happen first, the fries happen first. So that then when the payment happens, you get your fries really fast and everybody's happy. But if, cause if they waited and they're like, oh, well, I'm going to put in enough for one thing. A fry. You know, one order, two, two large fries worth of fries in the basket. And then the next person orders another large fry. Well, you know, well, crap. Now we've got to get, get all of these running in a batch. We need more fryers. Because, because we're thinking about it in this very strict linear way. But in actuality, there's a lot of situations where in the real world, that's not how things work. It's not how they're modeled. And in software, oftentimes we're trying to model. Real world processes or real world or concepts that we've translated into it from a real world into a digital world. And when we stop and, and think about it, it turns out, no, actually these things don't have to happen in the order that we're used to thinking about it. Just because that's how we were trained to write code.

[00:25:57] Dave Adsit: Well, and I'm imagining your coffee shop doesn't use two phase commitment. If it did, like I get out my, I place an order for coffee. And then I get out my. And I hold it out and the barista takes hold of the cash so that we're both holding it. Neither one of us is willing to let go until we, until the other person brings the coffee over. And now we're both holding onto the coffee and we count one, two, three, and I let go of the cash and they let go of the coffee at the same time. The world can't operate that way. Instead, we, we do things with compensating transactions. I order coffee 10 minutes later. I haven't seen my coffee. I go, Hey, where's my coffee? And they're like, Oh, sorry, we forgot. Or some random person took yours when they were taking theirs. Uh, cause you didn't hear us call your name or whatever. We'll make it again. Here's your coffee. Right. So we'll do compensating, but your $10 bill back or whatever. Right. And so we do compensating transactions to account for things not going well. I think, um, in digital ledgers, we try to make sure they're perfect all the time. And in. Physical ledgers, they're written in pen because you can't edit them. This is one of the big differences between a database and a ledger is that in a database, we just update stuff all the time, destroying all history. But in a real, in a ledger, it's like, Oh, well I wrote that wrong. Now I have to write a compensating transaction and explain why I'm changing the, you know, and someone can recreate the whole history of the events that occurred by reading through the ledger. Uh, and so those compensating transactions allow us to get to a point where we have the truth and it's working, but without, you know, holding the world hostage until every single thing is lined up perfectly.

[00:27:50] Allan Stewart: Yeah. Yeah. It, it does sound your version of the, uh, two phase commit coffee shop does sound like a hostage negotiation, right. Or, or some kind of like a, you know, standoff in, in the old west with two or three groups, all pointing guns at each other. Waiting for whatever it is that it's going to decide whether this is going to end peacefully or if we're all going to be dead. Right.

[00:28:14] Dave Adsit: Right. So events, they're a powerful tool for us to have in our toolbox as developers. Um, and the main thing, the main thing that I think about is, you know, they allow us to separate the trigger from the work that is triggered by that thing. Right. That's the core concept, right? The simplest way of describing the event. Is a trigger occurs, something happens. And then work happens after that. And so it can take all kinds of all kinds of, uh, forms. Um, you know, when we talk about having multiple cores, multiple machines, distributed systems, where we can parallelize a bunch of work that would have been done serially on a single core or a single by a single person. Um, I remember. This is going to date me again, years and years ago, one of the games that came out was the Conan MMORPG. And the biggest technical advance on this game, which I believe was a flop. If I remember, if I remember correctly, is that it could actually use two cores on a multi-core machine. And I think at that point, most of the computers that you would buy off the shelf, specifically gaming computers were like 16 or 32 cores. Right. They were substantially more powerful. Most of the games that you play. And I think this is still true. Most of the games that you played at that point, all of the games that you played were single threaded. And so you bought this super beefy computer because you wanted it to have a super high clock speed. And it came with all of this parallel processing, processing capability, and none of the games could use it. So most of your computer was basically sitting idle most of the time while you were playing a game. And we don't want to do that. Right. We want to take advantage of the physical resources that present themselves. And so we want to do things that allow us to distribute that load across multiple independent workers. And events are a powerful way of doing that. Martin Fowler wrote an article on the blicky a few years ago. I think the title is just event driven. And he talks about different types of events. And how. How they can be used where he has seen them used. So the first one is event notification. And that is very similar to what we've been talking about. Just basically an event occurs because something happened in the system. Either a user interaction or a computer process or process finished or something. And we publish an event notification that can be responded to somehow.

[00:30:59] Allan Stewart: I think one of the keys with that one event notification is it's. Really just about saying this happened. And what distinguishes it from the next one event carried state transfer is the state. Right. When when you record an event. What do you record. Do you record just that it happened or if you're publishing an event like this happened. Right. And so like if you if you take it too far it becomes useless. Right. So it's like oh the button has been pushed. Or even just a button has been pushed. Right. That's not. Something happened. The event. Oh well what was the thing that happened. I don't know. Something happened. OK. Well that's completely useless to me. Which which button got pressed by which user on which project on which page. Relevant to which IDs. Yeah. And so and so you have to have something. But but the big difference there is that event notification. Yes. Is mostly about. Identifiers. Yeah. These identifiers are related to this event that happened. Whereas state the event carried state transfer says oh by the way our event is also going to tell you what is the new state of the thing or what is the transform or some other information about what happened. Not so not just that they clicked pay now but they clicked pay now and the amount is.

[00:33:02] Dave Adsit: ededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededed with the ID of the project and nothing else. And an event carried state transfer, it would come with a serialization of the entire project. Project updated, here is the new state of the project. And that comes with benefits and challenges. There are some fantastic potential... If I just give you the ID, now you have to go look it up. If I give you the event with the entire project, great. Now you don't have to look it up. You can assume that you have it. Unless, of course, there's a possibility of multiple delivery or out of order delivery, in which case you have to worry about things like item potency and the order of the events. If something goes wrong on the network and you receive the project updated event five times and you kick off five requests for payment, that might be a challenge with your customers. Even if it was... Really only triggered by a single event. And if you have multiple ways to update a project and you receive project updated at timestamp two and then project updated at timestamp five and then a second copy of project updated at timestamp two and you assume every time that the current state carried in that event is correct, well, now you might have a bad time. Because now... Now you may have overwritten later events with earlier events.

[00:34:38] Allan Stewart: When we first started working together, there was a big push for us to start using more events in the system that we were working on. And I remember learning about it. And it was really powerful, right? Going back to what you were saying about, can we execute things in a more parallel way? That can be really powerful. And I think that that's important. But it's also important for us to realize, well, how do you reconcile those? I know we've shared a couple stories in the past, but one of them, my team was working really hard to honor this new event style of working. And so we were sending messages back and forth using RabbitMQ. And we ran into a particular kind of problem. We're like, oh, well, this is getting really complicated. And it turns out that for the specific thing we were trying to do, it made a lot ededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededededed stuff. So one team would publish an event with all the data and they were expecting to receive events with data like that. And there were questions around, well, how do we be item potent? Right. Item potency is all about like, it comes from like the functional background of if you run the function with this same data twice, do you get the same result? And so things like side effects, right? Requesting a payment or sending an email might make you not item potent anymore. And so they were worrying about that and like the order of events. Meanwhile, there were other teams that were saying, why are you going to all this trouble? Like this is good and this is ridiculous. What we really need is for you to expose an API so that when you publish your message, I'm just going to take the ID from that message and then ask you, going to call your API when I'm ready, when I'm good and ready, I'm going to call your API and ask you about that piece of data. And if your API is not available, well, we'll, we'll kick it onto a retry queue and then come back to it later. And, and then we'll ask you. And, but then they weren't worried as much about like order of events or what it, whether some data was stale or if the messages came out of order, because every single time they were going to just call up the API and this data. And so the difference between those two caused a lot of conflicts. And especially early on, we didn't understand why it's like, well, it doesn't make sense why we're having these problems because aren't we all just using events? Well, no, we, we were, but we were using them differently.

[00:37:51] Dave Adsit: That's definitely the case. Well, and I I've thought a lot about that and those just types of distributed systems since then. And it, there are a lot of advantages to building different types of things and, and there's just no quick wins when it comes to building a distributed system. There are some very hairy problems that have to be overcome. There's there's a concept in the distributed system world, or more specifically in the distributed queue world of at least guaranteed at least once event delivery guaranteed exactly once is I think mathematically impossible. I think that there are proofs. That prove that you can't have guaranteed exactly once delivery, but you can have guaranteed at least once delivery. And you might notice immediately that there's a problem there because if you receive a message once and you respond to it, that's different than receiving the message twice and responding to it twice. And yet it is generally Preferred over. Maybe once delivery. Maybe once delivery is definitely worse unless it doesn't matter that much. Right. Right. Like UDP. Okay. Okay.

[00:39:02] Allan Stewart: And honestly, UDP is basically that whole concept.

[00:39:05] Dave Adsit: UDP is that concept. I was going to say that the whole thing there is that when you are using these concepts at scale, you have to be thinking about building anti-fragile systems that can self-heal when things get out of sync. And there's a lot of characteristics to that. Like caching becomes very critical. If I've transferred state with an event and you take that state and write it to your local store as a data cache, how long are you going to keep it before you ask for a refresh? Maybe you consider it to be valid forever, but maybe it's not. I don't know. And now you have another problem. Not only are you working in a distributed system, but now you're thinking about caching on a regular basis. And that is one of the biggest challenges in software development. Caching, naming, and off-by-one errors. Right. And how to name your cache. And naming of the cached items. But those aren't the only types of events, right? When you start talking about events, things come up like event sourcing. What is event sourcing? Is that a separate thing? Is that going and finding events? Well, no. This actually more closely relates to the ledger system we were talking about earlier. Event sourcing. Event sourcing is basically recreating the physical ledger in the digital world, where instead of overriding the state in your database all the time and losing your history, instead you turn every update of an object into an event. And so now you have a replayable stream of changes that have happened to this project, this order, this payment. And so you can then take that full history of all the changes, and you can roll it up into a current state, or you can figure out what happened with the object and how it got into a specific state. You can do a lot of interesting things with event sourcing. And it's sort of related to the concept of events that we've been talking about so far, but it's not necessarily a distributed systems tool. It's more of a way of representing data on disk or in your system to preserve the history of that. Yeah.

[00:41:25] Allan Stewart: There are some interesting side effects to using event sourcing too. I've found that there are certain kinds of questions become easier to answer if you have event sourcing, and not just the history of a thing, but where was something at a certain point in time, which is related to the history, but other kinds of questions like statistical analysis becomes different. And an example that I like to use is that you're going to create, a score keeping app for, I don't know your, your favorite sports ball game. So let's say, let's say you're going to create the basketball, the best basketball point scoring app that's out there. And if you do it in the kind of traditional way without event sourcing, it's easy to start thinking about how this application begins. Right? So it's like, okay, well we've got these two teams and each team, they've got an integer, which represents their score, like how many points they have scored. And we're going to update that integer every time that a score happens. And then if something, you know, so you start getting into some interesting scenarios where you say, okay, well, what's, what is the score at the end? And all you can really know is right now. And so then if there's a foul and it's like, oh, well we counted the score because the ball went into the hoop, but then the ref said, no, that doesn't count. And so now we have to have like this way of subtracting, subtracting off. And so now our system gets a little bit more complicated or we start adding in like these transactional concepts. It's like, oh, well we'll allow this if that, and well, can it go below zero? And like you start getting into some interesting, interesting scenarios, but you don't necessarily have the ability to answer certain questions. Somebody wants to ask you, well, well, which player scored the most points? And you're like, oh, well, crap. I didn't, I didn't, I didn't count that. And so then you start adding in, it's like, okay, here's all the players on the team. And, and every time that they score, I'm going to add points to them. And then when they're fouled, then we have to take off points, but also from the specific player. And like, it starts getting more and more complicated. And then somebody says, oh, well, who had the best hot streak during the game? And you're like, oh crap. I don't, I can't even answer that. But if you instead use an event sourcing model, every time that something happens, you just record, the relevant data and you say, okay, well, player a scored this many points at this time. And then you start adding in some of the other things like, oh, well they were, they were fouled. Okay. Well then there's a compensating transaction that subtracts the right number of points. And it keeps track of the player who committed the foul and like all, all of these things that are just facts. And you just put all the facts into play and you have this big list of facts, but then suddenly you, you can start answering those other questions. It doesn't, it's, it actually becomes much, much easier from a logic and data processing mentality to be able to go through and say, oh, well, yes, I can tell you what the score was at halftime because I can just run through all the events and stop at halftime and say, yep, here, here's the answer. And instead of having the, the one system, uh, like you were saying before, like if you had a database and you just keep overwriting, the data, well, now it's gone. I don't know. Maybe we can pull a backup. Yeah.

[00:44:58] Dave Adsit: We could tell you how we can, how often do we put, how often do we send a backup offsite? You look at that data, right? Uh, there's another concept that comes up a lot in when people start talking about event sourcing or event driven systems. And it's somewhat related, but also different. It just, they just tend to come up together. And that is, uh, CQRS or command query responsibility segregation. Um, and basically the idea there is that you have a data model for commands that update data and a data model for queries that return data. And they are tuned specifically to those use cases. So in the case of point scored as a command, you might want to know who did it, how many points and what time so that you can then add it to your event source ledger. Um, but when you query it, you might only want to know current score for both teams. Um, but then you might have a secondary query for total points scored by player, right? So instead of, it basically is just a way to decouple the read and the right data based on the specific things that you're doing with the data when you're writing it. And when you're reading it, I only have this data available when I'm writing it, but that's enough. And when I do the read, I want the data to be, um, aggregated in a certain way, join these tables or roll this up or whatever. And so those are related concepts that come up a lot in when we start talking about event systems. Um, but they're not specifically event driven, right? The event notification event carried state transfer. Those are components of your, those are like core concepts that you need to understand to build an event ed system and event sourcing and CQRS. Are related there. You might say that they are coupled in some way, but they're unintentionally. So they just, they, they tend to come up in the same conversations a lot.

[00:47:02] Allan Stewart: Yeah. And because we have such an acronym soup in the industry, I think it is also important to point out that CQRS is different than CQS, even though they use almost all the same words, right? So CQS is command query separation, which is, which is, it's a related concept, but it's kind of different, right? Uh, CQS is about a command is different from a query and queries are immutable, right? They're pure functions that don't have side effects. Um, or I guess they don't necessarily have to be pure. Like you could read out of a database and have a query, right? But you don't, you don't create the side effect of I read from the database. And so then I also sent an email. No, I just read the data and every time that I read, I should get the same, the same result until the underlying data changes. But a command can mutate something, mutate something. Yeah, exactly. But then the command query responsibility segregation of CQRS is, yeah, it's all about those different data structures to say, Hey, just because I wrote the data in one way, it doesn't mean that I'm going to read it back that way. And that can be convenient, right? There's a good reason that we've been doing that for so long in the industry, is that I want to say, Hey, I've got this data, you know, these fields, a, B and C, and I just want to stick them in the database. And then when I ask for the data back, then it comes back exactly how I had saved it. That can be very, very useful. But if you're doing event sourcing, it's probably unlikely to be what you need. You probably want a different shape of data that you save as part of a command, right? Writing the data versus, when I read it back, what, what do I want to know from that event stream could be very different.

[00:48:56] Dave Adsit: And there's one type of event that we haven't really talked about yet, which is the interrupt or the system interrupt. And frankly, I haven't thought that much about these in a very long time since I was plugging devices into comports and trying to interact with hardware, right? These are very low level types of things that occur usually at the hardware level or the driver level. And, and they are of a type with the other events that we've talked about, but, um, certainly a lower level. Yeah. So honestly, I'm happy to leave those to other developers to be concerned with. Anyway, with that, I guess we can wrap up our discussion of events. And just as a reminder, events are a very powerful tool to have in your toolbox, and they can help us all build more dynamic systems, including, but not limited to distributed systems. And I will say no matter what you hear, they are no magic bullet that solves all of your distributed system problems. Building distributed systems is going to be hard no matter what. So, uh, at a certain point, we will very likely revisit the concept of events and talk more about some of the strategies that you can use to make your event based system successful.