Crafting Code Podcast

[00:00:00] Allan Stewart: Welcome to the Crafting Code Podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately I've been considering the utilization of space in businesses and public spaces.

[00:00:31] Dave Adsit: I'm Dave Adsit, an engineering leader, and recently I've been thinking about the benefits and potential risks of experimenting with local food options while traveling.

[00:00:41] Allan Stewart: Our topic for this episode is, "is your refrigerator running?" Right. Which is a tongue-in-cheek title, of course. We want to talk about observability and why it's so important to understand how our systems are running, whether they are robotic refrigerators or web apps, mobile apps, et cetera. How do we know if they're running, if they're working correctly, if things are in a problem state or if they're in the correct steady state that we want to see them in, and basically get that visibility into the things that we're creating with code?

[00:01:20] Dave Adsit: Yeah. Yeah. Like you say, we need to know what's happening inside of our systems in order to run them effectively and optimally. And if you think about modern machines, they are full of sensors that are continuously capturing all kinds of data about what's happening. If you look at your car, it's going to have an OBD2 port that is going to be constantly streaming data from all of the sensors everywhere in the car and the engine and everything. And that can be used to diagnose a faulty O2 sensor, punctured vacuum line, or a failing engine cylinder, as well as many other things about your vehicle as you start scaling up. And our software systems are the same. There's so much we can know about them. And it is critical for us to know the right things in order to make the right decisions.

[00:02:12] Allan Stewart: So when I think about this, it makes me consider that what we want to achieve is something that goes beyond just mere logging. The term that I've been using most recently is observability, which I like because it helps us understand, like, "hey, this is, like, more of a measure of how well we can look at or watch our system," right? Like, it's the ability to observe, right? Or if, you know, we don't want to just see things happening, right? I am reminded of that BBC Sherlock series. He's telling people, you know, "you see, but you don't observe." And that really is us a lot of the time. You know, we're not paying attention to the right things. And so once we recognize that this isn't just about logging, like, logging is important. It is one piece of instrumentation, right? The instrumentation is where we put in things that give us data, right? Sensors, logging, probes, whatever it is, your server metrics, app metrics. But that by itself isn't enough. You actually have to monitor, right? So monitoring and alerting is another term that I've used in the past. And I like it, but I like observability as kind of the umbrella for all of this stuff. Monitoring is, "are you watching?" Alerting is, you know, add some automation in so that you can sit up and notice at the right time when something's going wrong, so that you don't have to be watching all the time. Something can be watching on your behalf.

[00:03:52] Dave Adsit: Yeah, that's right. When we are talking about these systems and observability, one of the things that we really want to do is we want to start catching a baseline of how our system behaves, so that we can then start detecting anomalies. Or bottlenecks, or something that's happening that needs some kind of human touch, or human or even a machine to go in and do something about it, right? So when we first start, we're going to start with some some basic logging, uh, and monitoring package or whatever, and we're going to start capturing data, and we're going to use that data to set a baseline and understand where are our alert thresholds. Like, what is the system doing now? You know, we have maybe set a goal that we want every web request to get a response within 200 milliseconds. Okay. Well, what is the system doing now? Oh, most of our web requests actually take three to four seconds to come back. So we've got some work to do, right? So we can use this to start defining what needs to be done to the system, but we can also just use it to understand how the system actually operates right now. So usually the first thing we start with is instrumentation, whether we are doing it manually, which you and I have both done on many projects, where you just go in somewhere in the web stack, you hook an entry point and you say, "log that a request has started and log that a request has ended and log how long that went," right? You can do that manually. You can grab many production or many products that will actually monitor your system for you and just capture all of that basic common data that everybody wants to know and put it into a nice graph or chart or whatever, so you can see it. You and I have built that manually using Influx and Grafana. Not an uncommon path to start if you want to, especially if you want to minimize your costs, but you can use one of the professional products that can get spendy, but can give you a huge amount of data about everything that's happening in your stack.

[00:06:06] Allan Stewart: Yeah. Starting off with those common things, right? You're setting up your log stack, you're setting some server metrics, or you install an agent or something like that. I love those for setting that baseline, because then you can actually see, there have been many places that I've worked where, when I started the job, there was very little observability. And frankly, nobody knew what the baseline was. And so all you would get are just some kind of vagaries. It's like, "well, the system is slow," or, "you know, we perceive that there is a problem because our customers are complaining, but we don't know where, we don't know what," you know, When is something anomalous? You can't really tell if you don't know what the baseline is. You don't know where to set your alert thresholds. You don't know where to make your improvements, or tell if you made an improvement without having that. And so laying down that initial layer is really helpful, because there are a lot of common things that everybody does. Your server metrics are probably given to you by your cloud provider these days. There are standard logging mechanisms, because you always need them. And so that's the first layer that I like to place down. But then on top of that, I really want to get some custom data. What are the critical parts of your application?

[00:07:37] Dave Adsit: You can grab an application performance monitoring package and throw it into your stack and just run it. But that is exactly what it is. It's generic data about the system. It's not specific to your use case. And that's where it starts to become a problem, or rather, it's a start point and it's not the end point. Yeah. Throwing, throwing that on does not solve your problem. It enables you to solve your own problem. So the very first thing, the base, the basis of all of this, the basis of basically all of it is logging. And when it comes to logging, I have developed a pet peeve over the years of people not knowing the base of foundation of logging, which is that there are log levels, and they mean something, and you should use them properly, learn them and use them, right? There's fatal, error, warning, info, debug, trace. There's potentially a few others, but you should at base be using those five. And there's very important reasons why, right? We want to know about every fatal exception. We want to know about most of the errors, or errors in aggregate. We want to know about warnings, so we can prevent errors. We may want to know info and debug when we are doing things at a lower level in the stack, right? So not everything is an exception. Not everything is an error all the time. And one of the reasons this is critical is because it leads to too much noise in your observability platform, too much noise in your logging. And if we have too much noise, we stop paying attention. And we're gonna hammer on that a lot as we talk about this topic. It gives you additional benefits. For example, as we talked about, the productized professional application performance monitoring packages can get expensive. And if we are ingesting too much data or indexing too much data or whatever, that's a problem. So maybe I'm only going to ingest warning, error, fatal in production, unless I have a specific need for info and debug at this time. But maybe I want to ingest all of that in staging, so that I can actually debug a problem that I'm seeing that I need eyes on it as I'm operating it. The problem doesn't happen in my dev environment, but it does in staging and production, right? So you can use these as a lever to control costs. And that can be very important as you start to scale your system.

[00:10:19] Allan Stewart: And like you said, it helps you find the signal in all of the noise. That's right. If you have everything turned on for debug all the time, well, that's going to be probably too much. That's so noisy. And you don't, like, you have to wade through all of this stuff to find anything that's important. But even if you've just got it set down to, like, your errors and warnings, categorizing those correctly is important. Maybe warnings are things that, "hey, after a week or two, I'm going to go and look at the patterns of all these warnings." And so it was really important that I captured them, but I didn't need to act on them right away. Whereas an error should be something that I act on. But if you start putting informational data in, or worse, debug data in, and it's showing up as an error, those are not actionable. And now you're starting to filter through to try to distinguish, what do I actually care about? If I go and look in my tool and it says there have been 600 errors, how am I going to find the ones that actually mattered?

[00:11:25] Dave Adsit: Yeah. So one of the things I think about when it comes to logging is that logging is part of that total productive maintenance of the system. We are going to have to continuously update and refine our logs and what we're logging. And the reason why is because we should have a goal of being able to diagnose and understand customer problems from our logs. You know, we can't reproduce the exact scenario that occurred for the customer, but we should hope to have enough info in the logs that we can figure out what happened, so that we can prevent it from happening to the next customer. Yeah. But that means that we are going to be constantly moving things up and down the scale of coarse grain to fine grain. Our logs are often, when we look at the logs, we're often going to say, "this isn't enough data," or, "wow, this is way too much data." And so we're going to be moving things around continuously based on our current needs. And one of the insidious things that I've seen in many systems is that there are things that at one point are a strong signal that is useful and valuable, that at a later point become noise. And so we need to be cleaning those up and removing them from our logging, so that we can increase that signal to noise ratio. We want to minimize the noise and increase the signal at all times. That's the point of the observability system, is so that we can observe what is happening in the system, not so that we can flood ourselves with a wall of text.

[00:13:00] Allan Stewart: Right. And I will point out, when we're talking about this observability, when we're talking about instrumenting, things like logging is kind of like that base, most basic level of capturing data that we're familiar with, or we tend to be familiar with in these kinds of systems. It might also be time stream data, right? Or time series data that you're capturing. It might be that you've hooked up an agent or something else. Like, any of these things can fall into these same buckets. And sometimes what you use matters for different purposes, right? So there's some data that just fits better in a time series database and gives you an indication of how often a thing is happening. If you throw that into your log files, well, now you have to go and do a bunch of aggregation and figure out, okay, how many times did it happen? That's already done for you with other kinds of systems. And so I think making it easy to add instrumentation, easy to measure what you need to measure, is important. But then you've got to refine it. You've got to keep an eye on it, because otherwise you're going to start capturing that data forever, which leads us to the question of, like, retention and sampling. What happens when the data is just too much data?

[00:14:27] Dave Adsit: I mean, my first advice is, do not be a digital hoarder. I have worked on systems where people have kept all of the web and database transaction logs in S3 buckets for a decade. You don't need it. You're not going to use it. No one will ever use it. And you're just paying for 40, 50 terabytes of storage in S3 for no good reason. It's a waste of money. And worst of all, it gives you, like, a false sense of confidence that you could go back and diagnose something that happened before, when you very well likely can't. So don't be a digital hoarder. Don't try to keep all of the telemetry of the system forever, just in case. Keep what you need for as long as you need. And so when that comes, when it comes down to it, like, you might need all of the debug info or the debug and info logs for everything that happened in the last hour or the last day, but you certainly don't need them for a month. You may not even, you may have realized that in your system, you're unlikely to go correct errors that occurred a month ago. And you have been, watching the patterns, and you know what to do anyway. So those are also not creating value. So, you know, when we look at some systems, like a time series database will give you the ability to automatically compact data and roll it up using certain algorithms or characteristics as it becomes older. So for the last week, we're looking at data on a second by second basis. And then for the month before that, it's five minute windows. And before that, it rolls up to weeks or days or weeks. And you can keep a representation of how the data looked that goes on for a long time, but is very compact. It doesn't waste space, either on disk or cognitively. And so those are some things that you need to be thinking about as you think about how much data is in my observability system.

[00:16:26] Allan Stewart: I feel like it's always easier to generate more than you can reasonably utilize. Yes. We can just keep adding more and more and more, but you really can't keep it all. So, like what you were saying, there's going to be different times. Like, maybe, maybe I need debug level info sometimes. And so we have to figure out, what's the right way to get that? Do you, do you just have it for a certain amount of time in staging? Do you only record it for your new features, and you're disciplined about going back through your old features and cleaning those out? It's like, "hey, yeah, no, this works, and we're satisfied, and we're not going to capture that anymore." Are you going to use some kind of a sampling thing, especially for things that happen a lot in your system? You probably don't need all the detail of every single time that the authentication check came back. Okay. Yep. They're still logged in on every single web request or whatever. That just becomes noise so quickly. But you might want to know that it's happening, right? So some sampling of that data. It's very frequent, but we don't need all of it. We're just going to take some pieces of it, and then we can use that to look for or catch patterns. Another thing that I have been thinking about, I have not done this myself, but I think it could be really interesting to create some kind of dynamic logging situations, where we have the capability to log detailed information. But we only, we only use it if there is a problem, right? So if we detect that we're in, like, some kind of a warning or error state, now let's log out additional information. Let's, let's capture that. And some tools will help you do this a little bit. Like, you can add, like, breadcrumbs to, um, to a logging entry, and then it only gets kept in certain conditions or things. But, um, It's an area that really makes me interested about, like, "hey, could we turn down the noise by only caring about the details when those details matter?"

[00:18:42] Dave Adsit: I think we actually have used something similar to that with one of the software packages that you and I have used in the past. There was an option to open a web page, and you could see a trace of every request that was made into the system. And basically the way that system worked is requests were continuously coming in, but that debug level information was being thrown into the bit bucket, unless someone happened to have a web browser open to that page, subscribed to that stream of data. Like a live trace. In which case it was streamed. Yeah, that live data was streamed to that window. Otherwise it wasn't captured at all. And so I know we've used a couple of systems that have done that in the past. Yeah. So using those, or also setting retention policies for data.

[00:19:33] Allan Stewart: There have been many times that I've gone in and seen that there's just a lot of data for something. It's not always logging data, but often it is. It's just piled up, and, like, well, maybe after some time, just throw it away.

[00:19:48] Dave Adsit: Yeah. Sometimes it's just negligence. Like, "somebody else will come and clean this up eventually," right? Well, that somebody might be you. So just go clean it up.

[00:19:57] Allan Stewart: So that brings us from the idea of capturing the data, right? We've instrumented, but then the next thing that we need to do is have some kind of visibility, because all that instrumentation is great. But if the data is, it's going essentially nowhere, or it's too difficult to see, or, like, it's in our third party data package, but we haven't actually set up the dashboards that we need, then it's really not doing you any good. So set up dashboards, graphs, whatever it is that you can actually look at, see what's going on, and organize it. Right. There might be a lot of information there. What are the things that you actually care about, and, and set that up? So for, and this, this becomes really important in my, in my mind, because, like, a lot of the time cloud providers, they give you a bunch of tools, and there are graphs. There are things that you can see. But is it the right thing that you're seeing, right? So for example, if you're using a cloud provider and, like, a hosted database, they might give you a bunch of tools. You might be able to see a CPU and such things, but can you see the queries? Sometimes they'll show you things like, "oh yeah, these are slow queries." And, like, they've learned over time that we should record slow queries. But what about that query that is only sometimes slow? Like, there's only certain conditions where it's really slow, like, figuring out how to get down into the visibility of the piece that you need is important. Otherwise, it just stays a black box, even if you've got the data.

[00:21:42] Dave Adsit: Yeah. And to me, this goes back to that old saying, "a picture is worth a thousand words." If you can show a graph of things happening in the system over time, that is going to speak volumes in comparison to just saying, "hey, did you know that the P 95 on this end point is three seconds?" You know, you're, most of the non-engineers in the group are just going to glaze over when you start talking about things like that. And if you say, "hey, let me show you a chart. Here's all of the end points in the system. And then here's this one." And, or, "here's all of the end points in the system over time. And then we made this change, and then notice how all of them are now different. We need to go do some more work on that change, because it made some stuff worse." And so they're a very valuable tool for ourselves, but they're also a valuable tool for communicating with people outside of the engineering space.

[00:22:36] Allan Stewart: Yeah. And just making things visible becomes just so important, because otherwise you don't know. An example that we discussed prior to recording was your car's gas tank. If you didn't have something on the dashboard that told you how full it is, how would you know when you need gas? Like, over time, you might figure out some things, right? Like, you could go drive around and find out, like, you know, how much gas mileage am I getting? And I know the tank is this big. And so I can make some estimates about what's going on, but you're going to be surprised if there's a leak. If you, If you just suddenly run out of gas one day, it's like, "I filled it up yesterday. What happened?" You got to have this visibility. Otherwise, it's just, like, it's a black box.

[00:23:27] Dave Adsit: That brings a couple of things to mind. One of them is, if you took your gas tank size and the EPA estimated fuel economy for your car and multiplied them and said, "I can drive this far before I need to fill up." You will be very surprised to find yourself stranded on the side of the road several times before you adjust that estimate of fuel economy. Also, I think I was thinking about, you know, I had a car in high school that wasn't very great, and I had to check the oil regularly. And that's inconvenient, because you have to pop the hood and pull the thing, the dipstick, and do the stuff. And it would have been great if I'd had an oil, uh, oil monitor on the dash, but also that's a pretty good sign that your car is broken. So that would be considered noise in most scenarios, right? So it's easy to monitor the things that are important. You should make it easy to monitor the things that are important and continuously changing, but maybe don't necessarily make it easy to monitor things that don't matter. Yeah. Don't, don't introduce noise into your system because you could monitor how much oil is in the engine. It's just not the right thing to check constantly.

[00:24:36] Allan Stewart: Right. Yeah. So you've got to collect the data. You got to make that data visible somehow, right? There has to be a way to get at that data. And then you actually have to look at it. You have to do the observing part of observability. This is where you figure out how your system functions. What is that baseline that we were discussing before? If you make some changes, can you see the effects? "I was trying to make this endpoint faster. Did it actually get faster?" "I was trying to add a feature. Did that feature do the thing that we wanted it to do?" We made a change to the code and we weren't expecting it to change anything, but all of our API latency went up by 30%. We might want to go in and look at that. It's like, "what is going on here?" A good example that I really like, I heard in a conference one time, and they were talking about observability, and they were talking about how, going back to cars, if you drive a car for a while, you start to get signal. You start to understand that car, and you learn what feels right. Even things that aren't shown on the dashboard, you can start learning. And so if there's something that just sounds off. It sounds a little bit wrong. It's like, "oh, the, I feel like that fan didn't used to sound like that." It could, it could tell you something that, you know, you might be able to detect a problem before it becomes serious, right? Like, the car feels sluggish. Okay. Well, what's wrong is, you know, do I need to, you know, get some more power steering fluid? Are my, is the air pressure in my tires low? Like, you, there are physical feedback mechanisms that you can receive as the driver of a vehicle that tell you something about that vehicle. With software, we have to make sure that we instill these things into the system, so that it's possible to, to recognize them. But then we also have to be comfortable enough about looking at this data, observing what is normal, so that when something is not normal, we actually notice it.

[00:26:57] Dave Adsit: Well, when you say observe, one of the things that comes to mind for me is actually looking at. And so instead of just having dashboards that you can access through the web, and they're not really, no one's actually watching them unless they pull up that webpage. One of the things that I like to do is set up monitors in the office or in your home office, where it's constantly displaying a dashboard, and it's observable to everyone. And then we used to call that, like, the, you know, NOC, or the network operation center, whatever. But a lot of companies have put these up in areas where everyone can see, here's all the purchases and where they're occurring globally, or here's the stream of people consuming our product, to, like, people starting a new, you know, consuming the, the, for the first time, versus people who are repeatedly coming back, whatever your product happens to be. We've got some friends who used to work at a SaaS company that one of the executives had a monitor in his office that was constantly streaming the checkouts or the new purchases of their SaaS subscriptions. And over time, he got a feel for what are the typical patterns for time of day and day to day, day of week. And he would sometimes observe an anomaly in the data. And our brains are really good at pattern recognition. So he may not have even known why it was a problem, but he could have seen a problem. And he often, according to the stories I've been told, he would call down to the software team in charge of e-commerce and say, "hey, I'm seeing something weird in the data, what's going on?" And so he would be the early detection system for finding and resolving problems before even customers noticed them. And if I understand that story correctly, it's not like he had, like, a complicated graph either. Wasn't it just, like, a very simple spark line? Yeah, it's just a very simple spark line of, like, the number, how many purchases happened in the last five minutes. And as that streamed across, he just knew the pattern over the course of the day. And so by, he would just see it over and over and over out of the corner of his eye, constantly observing it and, and detecting anomalies, and, you know, correcting the problems, or rather calling the engineers who then corrected the problem, if one existed. Right. And hopefully they had deeper levels, more granular data that they could use to say, "yes, indeed, you're correct." Or, "no, you're seeing something that's, you're, you've detected a false positive. There's not actually a problem. It's, it just, it's just the way the data is today."

[00:29:38] Allan Stewart: So the next thing that I think, once we've started getting these concepts down, we can move from a kind of passive reactive state into a more active state, where we're saying, "hey, we don't want to just see what has been happening, but we want to proactively poke at the system." And one way that that is often done is through synthetics, right? Some kind of, basically it's a test against your system. I am going to create traffic that looks like regular traffic, but we know that it's artificial in some way, right? Like, maybe it's part of a test account, or it has a special header, or whatever it is that lets us know. It's like, "oh yes, this is our ability to test the system." It's not a real customer, but we want it to be as lifelike as possible, because if it has a problem, it can help us understand if customers would experience those same kinds of problems.

[00:30:41] Dave Adsit: Right. One of the companies that I worked at, we did regular synthetic tests of our entire fulfillment system. And so what you would do is you would just go into the system and you would make a purchase. And if you used a certain keyword in your first and last name, that transaction would flow through every part of our system, just like any normal transaction would. And then when it got to the fulfillment center, the pickers who were packing boxes and shipping them would see the keyword that basically indicated, "don't fill this." And they would set that order aside and then move on to the next one. And so it, it went through every part of our system, just like every normal transaction, without any special code pathways or anything. Um, and then it got to the end, and it just wasn't picked and fulfilled. And so, and then at the end of the day, one of the managers in the fulfillment center would take all those up and then cancel all those orders and refund all those cards, and everything would go back to as though it had never happened. Uh, though we actually also got to test the refund and reversal process, because if, if If you used your personal card to make one of these synthetic purchases and then didn't get your money back, you would definitely notice and tell someone in the office to fix it.

[00:32:00] Allan Stewart: What I really like about this is that it gives you these opportunities to force certain conditions to happen in your system. Especially if there are error cases that you want to make sure that you're handling correctly, just make them happen. There's been plenty of times in my career where I've written code and I said, "okay, I think this should work. I'm pretty confident that it will work, but I won't know until it happens." So make it happen. Make it happen in a safe. You've got this safety net that this wasn't a real customer. It was me. And so I can detect it, make sure that I am handling it correctly. And you just skip the part where you throw the red alert out into the world.

[00:32:45] Dave Adsit: Well, and sometimes we haven't even skipped it. You and I have put an expected exception in point, undocumented, into some APIs. And then you call it a few times and see if you get the alert. Yeah. Because all it does is throw a certain exception, and then you can exercise that exception pathway. One of the things that I like about synthetic tests is that you can, using a lot of toolkits that are available now, you could just set them up and schedule them. You say, "I want to make this type of request from this geography against our website. And five minutes later, I want to make a similar request from another geography." And so you're checking, is my site accessible? And these are kind of like unit tests, in that they may be a little bit brittle. And sorry, did I say unit tests? I mean, these are kind of like UI tests, in that they may be a little bit brittle, and they shouldn't be run excessively, because they can become expensive. You're either paying for someone else to run them for you, or you're setting up a system specifically to run them. And they also put load on your production system. So you don't want to overwhelm it. But they let you verify. And as an engineering team, they create an early detection system for any type of issue before your customers notice, before there's enough signal to the customer success team that they call engineering and say, "I think something's happening. And this is what it looks like," right? You've already detected it and hopefully fixed it before you even get the call from client success.

[00:34:19] Allan Stewart: Synthetic tests also give you that ability to test things out in production, which, if we're not careful, sometimes we just have the flippant reaction of test testing production, because I didn't do any other testing or thinking about it before. It's like, no, that's, that is bad. That's not what I'm talking about. I'm talking about the kind of testing production where we say, "hey, this is the actual system," because your other environments, your dev environment, your staging environment, they're, they're not like production. They never are. Right. There, there was a great talk I heard once where the staging environment was compared to a rundown hotel. It's like, it's got all the rooms, like, the kitchens there. It exists, but it's not the same. Yeah, that's definitely the case.

[00:35:05] Dave Adsit: I've never worked at a company where the staging environment was anywhere close to the production, if only for the fact that it has different data, different data volume, and virtually no load on it. If your production system has 10,000 requests a minute, your staging system has four. That's not the same environment, even if you have a recent copy of the database and all the hardware is scaled the same. So the next thing that I wanted to talk about is taking that concept of testing and production one step further, and kind of refining what we're talking about when we say you need to have your own observability into the system, not just the generic stuff provided by the logging tool or the APM tool or whatever. You need to get specific and real with, what is it about my system that matters, and what do I care about? And so sometimes we can't generate that through a synthetic, or it's unreasonable to generate it through a synthetic, because there may be intermittent issues or intermittent behaviors, or, you know, behaviors that happen at volume that are important. And like I said, when talking about the guy who is looking at the patterns in the sparkline, like, sometimes it's not a single data point that is telling you a story, but it's a collection of them. Sometimes I want to observe behavior, log the behavior, and then I want to assert that certain things are happening the way I expected them to. And I like to call these, this type of test, a data probe, right? And so let's use an example to demonstrate this from a completely different domain. I like to barbecue. And so when I put a pork shoulder on the smoker, I try not to open the lid more often than necessary, right? There's this phrase in barbecue, "if you're looking, you're not cooking," because all the heat and smoke is gone as soon as you open the lid. So I use probes, and I stick the probes in the meat, and then I connect to them with either Bluetooth or wifi or whatever. And I watch the graph of the temperature of the meat rise slowly. I might set an alert. "Hey, come and get me when the meat hits 204 degrees or 208 degrees," or whatever the temperature is for this particular cut that I'm working with. And then I'm going to do something, right? That's a trigger. And I will watch that graph, and it should be monotomically increasing. That number should be going up continuously, at hopefully a steady rate. And if it stops going up, or it starts to go down, now I've got an event that I need to worry about. What happened? So I'm going to go to the smoker, and I'm going to start looking at, did the fire go out? Did the pellets get stuck? Did I run out of pellets? I hope not. I have a monitor on the pellet level as well. Did it start raining or snowing on my smoker, thus affecting the external conditions in a way that I could not have foreseen or controlled? Did someone else come by and open the lid so they could see the meat cook? I don't know. I don't know, but I have probes into my system that tell me about very specific things that I care about in this system. We can do the same in our software systems, right? It applies exactly the same. We can monitor very specific behavior and ensure that certain events are happening. And what we're looking for is uncommon conditions that don't comply with the expected behavior of the system, or we're looking for something that is hard to detect, the absence of normal behavior, right? So maybe I could share, like, an analogy in a specific software system. If we had an e-commerce platform, we would expect, if we had an e-commerce platform, we would expect to see successful transaction against the payment provider at a certain rate. And if we see none, that's a cause for alarm. So I need a probe that tells me when the event occurred, and then they need to create a monitor that checks for the absence of that event in a time window. And that, to me, that's what I mean when I talk about having data probes. They allow us to test for those kind of weird anomalous things that are not commonly observed in other systems. Like, if I am grabbing a generic logging or monitoring tool, it's not going to have a monitor for zero financial transactions in the last five minutes. They don't even know if you're doing financial transactions. And if I'm doing financial transactions, I might need one of those to know that my system is healthy. So basically, but this doesn't even need to go through the logging system, right? I could write it anywhere. I could kick out an event. I could do something else like that. But basically I want, for everything that happens, or all of these things that are critical to my system's health happening, I want to create a data probe where I am emitting the data somehow that I can capture. And then I want to have another system that's observing that data and monitoring for it. I want to create the automated version of that executive who had the TV with the sparkline. I want a computer monitoring for it, and then creating alerts that it sends out to the developer on call. Right. And so there might be some scenarios, some that are, I've seen before, where we're not getting any errors from the payment processor, but for some reason, every transaction we send to it comes back with 200 okay, "transaction declined." You're like, well, I know that not every one of our customers has a bad credit card or has hit their credit limit, right? I had 500 of those 200 okay transaction declines in a row. That's probably something going on, and I need to know about it so I can go look at it. Maybe the, something is going on at the payment processor. Maybe my key expired. Maybe I didn't rotate it properly. I don't know. But, but that is definitely something worth looking into that I'm not going to get through my standard logging and monitoring system. It's not going to tell me that there's a problem when every API request is coming back 200 OK. But I know it's a problem, because every 200 OK transaction declined is, if everything's coming back in that state, that's wrong. But also, I don't want to turn a card decline event into a warning or an error, because it's not. That's typical standard expected behavior for an e-commerce platform. So I need something more. I need this kind of data probe that is posting this or publishing this data in a way I can consume it, and then react to either the presence or absence of specific data conditions.

[00:42:05] Allan Stewart: And the context is really important, right? So another example along those lines, if you have a login form, some people are going to mistype their password. And you don't want to be alerted when that happens once, or even two or three times. Because let's face it, sometimes we just really are ham-fisted in the morning or whatever, and I can't quite type my password right. Or I didn't notice that the caps lock was on. And, like, I very forcefully pushed all those keys in the correct sequence, but it came out wrong because I had the caps lock on. So you need, you need context to go along with this. It's not, it's not just the individual data points, but it's, it's that collection of them. It's like, oh, if there are a handful of spikes in the graph that don't get very large, it's like, "oh, yeah, there's a spike of five times that they type their password in wrong. It's not a big deal." If there's a spike of 5,000 times that they put their password in wrong, all within 30 seconds, well, maybe you've got a bot attack or something like that that's going on. And so we're kind of aggregating this information, or taking multiple pieces of data, whether it's across a time stream, or when these two conditions together indicate something that we need to understand. That's what I like about these data probes, giving you those additional capabilities. And there are some tools that will help you do these things. And you can set the conditions, and it will show you a graph or even trigger an alert. But ultimately, you're going to have to figure out what the right conditions are to set into the system.

[00:43:52] Dave Adsit: Well, and I think that the thing that is critical to consider here is that this is real production user data. This isn't synthetic data. This isn't test data. This isn't whatever. This is observing the actual behavior of the system when real users are using it, but in a way that's non-obtrusive to their use. And so, you know, you might say, "I'm going to throw this into my logging system," but it's not an error. It's not a warning. It's not info, because I want it even when info is turned off. And it's definitely not debug, because I want it when info and debug are turned off. So you might create your own custom logging level that's like Dataprobe, or Dataprobe 1, or, you know, P warning. I don't know, P info, something that you're grabbing the data regardless, because it's a custom level of logging. Or you might say, "this doesn't even belong in that system. I'm just going to publish events to our message bus. And then I'm going to aggregate them in another system that's looking for these anomalous conditions across multiple parts of the system, because I'm aggregating the login page and the checkout form." And if somebody logs in and fails to check out over and over and over and over, maybe that's a condition that I care about. The thing is that the point here is we need to create monitors and alerts in our system that are custom to the domain that we're working in. And they're not going to come for free out of the box with whatever provider you grab.

[00:45:23] Allan Stewart: Yeah. Thinking about that context again, it makes me consider that there are different times where these things matter, right? So going back to your barbecue example, if the temperature is dropping and the burners are turned off, that's a good thing. And I don't want to be alerted right now. It's only when I think I'm cooking that I want to be alerted if the temperature drops. Conversely, if it's meant to be off and it keeps getting hotter and hotter, that could be a really bad sign, a potential disaster in the making. I want to know those things. Those combinations come really important as we get into alerting. Alerting is where we say, "hey, now we're having the automation," right? Like, we can just, we can just have the graphs, right? You can just have the indicator, and I can see, oh, the temperature is going down. And I know, because I turned off the grill, that that's okay. But if I'm not watching that, who's going to let me know? That's where alerting comes in. That's where we say, "hey, we need a machine who won't get bored, who will pay attention and let us know when things are wrong, so that we can actually do something about it." Yeah.

[00:46:40] Dave Adsit: It's that annoying little check engine light on your dash. That's telling you one of your O2 sensors is out, or one of your hoses is blocked. Like, "oh, okay. I'll go fix it." We're not, we're not going to do the Homer Simpson thing and just put black electrical tape on top of that thing, can make the problem go away. Right. Yeah. Yeah.

[00:46:59] Allan Stewart: Because we don't, we need to see when they're happening, which means that we need to, again, filter out noise so that we're just getting signal. Alerts in particular need to be actionable, because if everything is alerting, how, how do you know what to deal with? If there are things that alert and they're not actionable, and you'd be like, "oh, I know that it was just one of those alerts that comes," and there's nothing you can really do about it. Like, you're going to train yourself into desensitization, where then when a real alert comes up that you should take action on, you won't recognize it, because it's easier to just say, "I ignore the alerts because they're usually not anything." Yeah.

[00:47:44] Dave Adsit: That's the alert fatigue, right? When everything is alerting all the time, it might as well not do anything at all.

[00:47:51] Allan Stewart: Yeah. And if you're in a new system, or, like, I've worked on a lot of systems where I've come in and there wasn't observability. And once we add some observability, all of a sudden it's like we went from zero errors today to 10,000 errors a day. And that can be overwhelming. And so sometimes we have to pick our battles and say, "okay, actually, let's set some of these to warnings." Even though some of them probably are legitimately errors, like, we should take care of this. But we can't take care of all of these right now. So we're going to put a pin in it and say, what are the most important things? Let's get those under control. And then we can expand out to some other things.

[00:48:35] Dave Adsit: Yeah, that's definitely the case. You can easily get overwhelmed if you're not being careful. And again, that goes back to signal the noise. I don't want noisy alerts, because I don't want to be desensitized and I don't want to ignore them. I want, and I only want the signal, and especially actionable signal, or, and sometimes the action is just raise the warning flag, tell everybody in the company, company, "something bad is happening. We think it's this, we're investigating." And again, this is an opportunity for us to preempt the customer call of, "hey, I can't do X, Y, and Z," like, "oh, engineering is already working on that. And they will have a solution for you shortly. Oh, look, they just notified us that it's fixed," right? Like, that's our goal. We want to reduce that mean time to recovery when, we want to reduce the mean time to recovery by quickly observing and quickly fixing any issues that happen in our production environment. Sometimes the issues could exist for hours or days before enough customer reports come in that somebody thinks to pass the information on to engineering. Alerts also bring up the concept of being on call,

[00:49:56] Allan Stewart: And I don't think we want to go too deep into that topic today, but I think it's important for us to consider, as professional developers, we want to make sure that our software works, and so we need to be ready to be on call, right? Like, if we're delivering something that's meant to have a high uptime, and we're saying, "yes, we can build that. Yes, we can provide that for you, business." When it fails, we need to be there, ready to help. And that means that we need to understand, when we're on call, like, what kind of tier we're on. Or, you know, is there somebody who's going to field some stuff for you, and you're only getting called when it gets really bad? If you're going to be away, like, how do they escalate? So if you're not available for some reason, that somebody else can look at it too. Those are all questions that are related to your observability, to your alerting in the system, that are important to consider as you're setting this up.

[00:50:59] Dave Adsit: Right. Essentially, there's no point in having a fire alarm if there's no firefighters. Yeah. You have to have somebody to be able to respond to the alerts. So really, all of this is in favor of tightening our feedback loops. We've talked about that a lot of times before. It's very important in software development. You want to tighten those feedback loops so that you can do the right thing at the right time.

[00:51:24] Allan Stewart: And these are bigger scoped feedback loops. I think it's so important for us to be considering these different scopes, because some feedback is very, very fast, very, very tight. Typed the code in incorrectly, and your compiler immediately tells you, or maybe even your IDE immediately tells you, there's a syntax error here. You forgot the semicolon again, or you type the keyword wrong, or whatever it is. You find that out immediately. There's other things that you find out when you run your tests, when your CICD pipeline runs. But what we're talking about here with the observability of the system is much broader, right? You're at a further out level, because it has to be out in production, and you're monitoring and observing things. Sometimes there's things that you've developed years ago, but you need to know it, right? Like the login page that we talked about before, you probably did that very early on in the creation of the system. And you're not thinking about that anymore. You're not thinking about password credential management kind of stuff. But if something goes wrong there, you need to know. And so it's giving you that feedback and allowing you to tighten those feedback loops, because you know what's going wrong, when it's going wrong, so that you can improve that system.

[00:52:49] Dave Adsit: Right. And at the end of the day, it allows you to create those improvements and ratchet them down, so that you don't have regressions later. If we create a series of alerts around long page loads, and we release an update, and suddenly all the pages are taking twice as long to load, we'll know right away we need to roll this version back and figure out why that happened, right? So you can really create the type of world-class software system that we would all be proud to work on by leaning into these concepts around observability, of creating the right level of logging, the right amount of signal, and reducing the noise, and getting the right level of production testing through synthetics and probes and whatever, that allows you to know the system is doing what it's supposed to do. I mean, if somebody says, "what keeps you up at night?" And you say, "nothing, because my monitor or my alerts will wake me up if anything needs me," that would be a phenomenal place to be for most of us.