Crafting Code Podcast

[00:00:16] Allan Stewart: Welcome to the Crafting Code Podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately I've been thinking about the value of tried and true desktop software versus a software as a service product.

[00:00:33] Dave Adsit: And I'm David Adsit, and I am a VP of engineering, and I've been thinking a lot about 80s video games. Some of them were so hard, they were bordered on impossible. And part of me thinks that this is to compensate for how little content there was. And it has me thinking if there's anything that we do now that's kind of silly, but is a compensation for something that we're not doing well.

[00:00:57] Allan Stewart: This episode is about AI. And when we say AI, and right now, at least as of the recording time, when people in the industry on blogs and news articles talk about AI, what do they mean when they say AI? Because I don't think it's a general artificial intelligence. Almost always they're talking about large language models.

[00:01:25] Dave Adsit: Yeah, that is definitely true. That is definitely one of the things that as a kid growing up in the 70s and 80s with science fiction, I expected AI or AGI to be a lot more impressive than what we get with our LLMs. Yeah.

[00:01:42] Allan Stewart: And I'm not an expert in large language models, but my understanding is that basically it's applied statistics. So it's not artificial general intelligence. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. I remember like 20 years ago, it was neat that you could do that, but it didn't work. Right. Like even the early versions of like Google assistant or Siri, like we're still leaps and bounds better than what you could do 20 years ago. And then now LLMs are doing really kind of surprising things now that I'm not used to, or I haven't been used to a computer being able to do. Right. But I think the thing that really gets me is that the more I interact with these LLMs, they're not making decisions. They're extremely reactive to their input. And sometimes they're excessively confident or overconfident, but they don't, they don't decide on a course of action. They wait until you prompt them over and over until you can kind of bully them into doing what you want.

[00:03:20] Dave Adsit: Yeah. One of the, one of the content creators I watch online talks about that a lot. He shows examples of how he asks one of the AI products to solve challenging math or statistics problems for him. And it will confidently give him the wrong answer and then try to convince him that the wrong answer is correct. And then when he points out specifically where it gets problems wrong, it will, you know, then change its mind entirely as though it never gave him the wrong answer to begin with. And that, that to me, that kind of AI hallucination is one of the challenges that we have. And one of the things that we need to talk about when, when it comes to AI, the, it doesn't give you a confidence interval of whether or not it actually understands or knows the thing like you would get with a person. If you ask a person a question and they give you an answer, you can typically read from their body language, whether or not they think that's a good answer. AI is always a hundred percent certain, or at least the LLMs are a hundred percent certain. I assume that once we get to an AGI, it will have human foibles like insecurity.

[00:04:37] Allan Stewart: Yeah. It's kind of a, the LLMs are kind of a Dunning-Kruger writ large, right? They, they're convinced that they know. And, and we, and I think part of the problem is we, we anthropomorphize them quite heavily, right? We, we say things like even just now, I think I said, and I heard you say that it knows, right? Yes. But it doesn't know that's, that's the thing. Like it doesn't have a comprehension or an understanding of the problem or the topic that you're talking about, because under the hood, what's really going on is some magic in a vector database. Right. I watched a really interesting video a while back. I should have probably rewatched it to freshen up my memory before we had this discussion, but it basically said that these LLMs work in three general phases. First, take all of the variations of like all the different ways that somebody could give you input and narrow that down to basically some kind of like an intent. Like what are the keywords? Like what is the, what is it that is desired? Right. And then then you can take a lot of different ways you can express yourself and ask for the same thing. And then it consults the vector database to see what words are really close to like statistically close to the topic that we've narrowed down to, or in the case of like AI art, which what pixels are statistically close to what you want, what you're desiring. And then, so there's like kind of like this funnel that goes into the vector. Database. And then we use the deck vector database. And then there's like our inverse funnel that extrapolates a result and broadens out and fills in the details. And so it doesn't, there's not an understanding built in there. It's really, it's really just figuring out from these vectors. What, what are the statistic? What's the statistical likelihood that this is what the person is asking for or, or what um, what responses are statistically. Significant for this kind of input.

[00:06:52] Dave Adsit: Right. And I, one of the things I think about is, is the necessity of training the models with good training data. And I just have this thought that what if you trained your AI model strictly on sci-fi fantasy and alternative history books, it would confidently give you answers that are completely disconnected from the real world or any events that have occurred in it. Right. So. You might have, you might have one that's trained on world war two alternative histories, and it would know a lot of the terms and words about world war two. And you'd be like, tell me about this battle and tell me who won this until, and it would do that. It would, it would look up in its database and it would say, oh, these are the things that I see. And okay, I'm going to regurgitate this answer. I'm going to extrapolate a result based on this database of things that I have. And it would be completely disconnected from reality.

[00:07:49] Allan Stewart: Yeah. Yeah, absolutely. I think it could give us some really interesting answers about which, uh, which star Trek character or maybe Brandon Sanderson character is most likely to go back in time and kill Hitler, but it, but it's completely fanciful. Right. And it doesn't know the difference between that and, and reality because it's because the reality is just what we pour into it. Yes. That's all.

[00:08:19] Dave Adsit: Because it doesn't actually know anything. Yeah. Right. It's just doing statistical inference based on training data.

[00:08:28] Allan Stewart: But we can do a lot with that. Right. And I think that's one of the things that's why the hype cycle is so huge on AI right now, because it is very surprising. It is useful. It is leaps and bounds above what we could do in a similar vein before. So what can we do with what we have? Right. So. Um, I've been using GitHub copilot for the last, uh, six months or so, and it's pretty okay in my opinion. Um, there are some times where I am astonished. I will, uh, like I, I start writing up a data structure and then I start describing what I want a test to be, and it will suggest a really good unit test that then I can use. To test drive my code. But then I have to keep like, in order to really do a really good job of TDD, I have to keep prompting it with, okay, well, what about this? And now this, and now this, but still it's surprising and useful at times. But then it also for the last like two or three months has been stubbornly insisting that there should be a enumeration in my code called pricing type dot fixed price. Because we have in the software I'm working on, we have this concept of pricing types. Fixed price is not one of the ones that we use, nor will it ever be. But it really wants it to be. And so I'm constantly having to tell it, no, no, still no. It's still not fixed price because it doesn't know. It's just somewhere in its vector database. It thinks that fixed price is a really good name for a pricing type.

[00:10:20] Dave Adsit: Well, and it might be for many, many other software systems that are not yours. Exactly. Perhaps most of the ones it was trained on. Exactly.

[00:10:30] Allan Stewart: And so it gets back to your question, right? Have we trained our AI on good code?

[00:10:34] Dave Adsit: I would suspect that we have not because I, in my experience, have found very limited amounts of good code. The signal to noise ratio. Is way off.

[00:10:47] Allan Stewart: Agreed.

[00:10:47] Dave Adsit: Code that I consider to be really good code exists in very limited quantities, especially compared to all of the code available in the world, or even all of the code available on GitHub.

[00:10:59] Allan Stewart: Right. I'd agree with that. And to further complicate things, we can't all agree on what good code is.

[00:11:07] Dave Adsit: Well, and that's the other thing, right? Is that what is good depends on your constraints. Good code for embedding. Embedded might look very different from good code for a web application. You may have very tight memory or CPU constraints. You may have to perform at a very high level for a first-person shooter, for example. And so you're going to do things in the code that would be awful in other contexts. My understanding is that in the early spacecraft that we sent up, they unwound all of the loops. Because they didn't want to have any chance of running out of memory while launching a rocket or a space shuttle. And they had a very tight constraint around some of that stuff. So they just unwound the loops. And that's awful. Like, you wouldn't want them to have to maintain that code. Also, you don't want an off-by-one issue when you are traveling 10,000 miles an hour, 10,000 feet off the ground.

[00:12:12] Allan Stewart: Or an infinite loop.

[00:12:14] Dave Adsit: Or hit an infinite loop. And it just keeps accelerating forever until it runs out of fuel. You're like, well, I guess we missed the moon. So, yeah. So what is good code is also context-dependent. And it turns out that each context is different. Most of the pricing models I've used have had a fixed price option. So I understand why the software is trying, or why the AI, the LLM, is trying to push you to implement that. But also, it's not applicable to your business. Or your business software. Right.

[00:12:48] Allan Stewart: And then there's also different styles. And so, like, we can't always agree on what style of coding. And some of that is, like, formatting style. But then there's other styles, too. Like, I prefer to work behind an anti-corruption layer. So that I can have a big chunk of code that's kind of in the middle of my business logic and domain. That doesn't have to put guards everywhere. That I'm not constantly checking. Is this thing null? Is this thing undefined? Or nil? Or whatever. Sure. Because you already checked before that value came in. Exactly. Exactly. But the AI doesn't know that. And so, the autocomplete continually tries to add those things that belong to a different style of coding in. Even though that's not the right thing to necessarily do. And so, I've been thinking a lot about this. I wonder if AI is going to be an interesting... Like, level set in the industry. Because now, you know, people are using it. They've got a lot of autocomplete and whatnot. And so, it may raise all boats with the tide. Assuming that there are more junior developers who are still learning and who are... That are joining the industry. And so, they're going to get the most help. And it's going to level up their skill set. But I'm not sure. It might lower the tide in some ways as well. Because it's sort of this common denominator that everybody's moving towards. Hopefully not the lowest common denominator. I think you talked about regression to the mean. Yeah. That's kind of how I think about it.

[00:14:34] Dave Adsit: The LLMs are generating kind of the average output of the input models. Right? It's not exactly an average statistically. But it's bringing you to the middle. So, if you are below the mean, it looks amazing. But if you're above the mean or specifically operating in a way out of the center of the bullseye, then it's going to keep trying to drag you back. And if you don't know any better, you will get pulled back. And then you will lose some of the benefits of the specific things you were doing. If we had implemented LLMs on code bases 10 years ago, it would not help you write tests. Because the average... The average code base didn't have unit tests 10 years ago. Probably only the 99 percentile had tests. And so, if you are test infected, if you want to write test-first development, if you want to write unit tests for your code base, if your goal is good unit tests, right? To code with guts, you may have found the AI working against you more often than not.

[00:15:40] Allan Stewart: And it just makes me think about when we don't know any better, We're too busy or distracted that we don't notice that things have gone wrong. And there are some times when it's obvious that the AI or the LLM has given you some code that is bad because it doesn't even compile. I can't tell you how many times it forgets closing parens and braces for me in the code that I write. Like all the time. And I'm constantly chasing that down. But worse... The thing that I worry about is when it gave you reasonable looking code that is suboptimal in some way, or wrong in some way that is non-obvious, non-intuitive. And then what do you do? If you don't know or don't notice, now all of a sudden you're stuck with it.

[00:16:34] Dave Adsit: Or if you're in a hurry. And you just accept that code. So, you know, one of the things I thought about a lot is... Kind of rehashing the regression to the mean. I'm not good at writing marketing copy. I know that most of the LLMs have been trained on most of the marketing copy in the whole world. Because it's out there. It's intentionally, purposefully out there to be consumed. And so the AI models consume it. And now the LLM is good at writing mediocre junior executive marketing copy. Right? And that's better than the... Marketing copy I write. So for me, great. Please write me a sales pitch for this thing. Please write me an elevator pitch for whatever I'm working on. Make it shorter. Make it more energetic. Make it more enthusiastic. It can do all those things. And now I've got something that is better than what I would come up with on my own. When it comes to code, I don't think that average code is good enough for the systems I'm working in.

[00:17:41] Allan Stewart: Hmm.

[00:17:42] Dave Adsit: I want... I specifically train my engineers to be craftsmen. You know, I want them to treat their craft with care and create and deliver software that is free from defects, well-tested, easily readable, easily maintainable, etc. Like, these are some of the constraints that I care about a lot in the code and the systems that I work on that are not generally the case. In fact, even if you were to go to my own GitHub... Yeah. ...account and look at it, there's maybe 30 repos out there. Probably 27 of them are coding katas I've done. And when I write a coding kata, I am trying to learn a specific thing versus trying to build good production software. And so even if I trained it on my own code, I would not like the code that was output by the LLM.

[00:18:36] Allan Stewart: Yeah. So branching off from that, what do we think... Yeah. We're kind of entering the speculative, predictive world. Is AI really going to change how we code? Because there's a lot of talk about it. A couple weeks ago, I was sent an article. Apparently, some executive within AWS was saying that if you fast-forward 24 months from now or some amount of time, they wouldn't specify exactly how long. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. as I've been using GitHub Copilot, because it is better than I expected, but it's not that great still. It's got a lot of room to grow still. Like even now, it's kind of frustrating. In some ways, it's a regression because my IDE used to be really good at giving me code completion based on what was actually in the code defined by my types in TypeScript or C Sharp. And now it invents things that aren't really there. So I just, I can't see it. I don't see this as a thing. What do you think?

[00:20:15] Dave Adsit: Well, I think that's wishful thinking on the part of an exec who wants to cut his, wants to spend his budget on something besides these very expensive opinionated developers. Hmm. And I don't think it's even possible that most of the AWS developers will stop writing code within 24 months. I'm reminded of all of the other times we've heard similar predictions. Yes. Rapid application development tools, the rad tools that were going to be so great for the industry. We were going to produce so much code so fast and they were literally only optimizing for the very first part of the project, right? You can say, oh, I'm going to walk through this wizard and I'm going to put in a couple of types and it's generate an application for me. And it sure did. And that was some of the ugliest, least maintainable code that you've ever committed to source safe. So you would often have to read, you would have any time the constraints or the problem statement or the specs changed, you would have to go back and regenerate the whole application again, because you couldn't maintain any of it. I'm thinking some of the code there specifically was the UI code. Behind the WYSIWYG editors, you drag and drop your UI components around and now you have a working Windows UI, but good luck with maintaining any of it. Good luck. I mean, you'd basically start over anytime you had to, because even though we know that the creation phase is short compared to the overall length of time that the system exists, we built tools that optimize for that space. We've also built, like the case tools. They're like, Hey, just define a UML schema in this specific tool. And it will generate output code in whatever language you want and database models and all of the things will just be perfect. And it's magical. And you basically just shifted where you were writing your actual code, right? There's the code is where you write this, the ultimate specification. And so if you then transform that, then that's, that's something else. If we were to look at C sharp, C plus plus Java, whatever, one of these statically typed compiled languages, the code is written in a human interpretable language, right? And then we compile it and we generate byte code or whatever, or straight up machine code in the case of C plus plus. And that's not really human readable anymore. It's not really usable anymore. Yeah. So the case tools were kind of like that. They're like, what if you took your good expressive Turing complete language and then put a worse crappy or harder to use language in front of it? And that actually became your code layer. And so UML became the language that you were expressing your software in. And then it had to go through multiple compilation steps, one step to compile it to C plus plus, and a second step to compile it to machine code. And if you ever, ever edited the C plus plus that was output, well, now your case tool is going to be disconnected and not going to work for you very well anymore.

[00:23:36] Allan Stewart: Yeah. Yeah. I, I agree. I think it, it smells of that same kind of wishful thinking. The same kind of wishful thinking that we've seen in more recent years with things like low code tools. Yeah. Yeah. You don't need a software developer. You can just have your marketing department do these things with the, with these tools. after about six or eight months, they, they end up at the door of your engineering leader asking for help because they can't get it to integrate with Salesforce the way they want it to or whatever, because they they've gone outside the bounds of what that tool was able to really do by itself. And I think LLMs are kind of in that same vein. It looks really promising. Yes. And I can understand that. Right. I am not a very good artist. And so there's part of me who looks at some of these AI generated art things and think, Oh, that's pretty cool. Because I like to do things like play role games. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. pay five bucks and have access to one of these AI tools. And so I get that because I'm not very good at art or marketing copy, then these AIs can be helpful for me. And so all these people who are not writing code, it's easy for them to see a future where this replaces those pesky developers. And yet I also know enough about art to be able to say, hey, you know what? I can appreciate why you might want a real artist. I had a friend who was doing a D&D campaign and one of the characters is like an anthropomorphized deer or elk or something. And he was trying to get the AI to generate a picture of this character, but it just kept doing the horns wrong. And no matter what he typed, and it just got worse and worse, making less good horns. And no matter how many times he told it, it's like, no, it's more like this animal, not like that animal. It couldn't do it. And so I think you're going to run into similar situations in code where no matter how hard you just try to describe to the LLM what you want, it may not be able to actually accomplish that.

[00:26:22] Dave Adsit: Well, and I'm just reminded of, of the saying that you've heard me say a thousand times, and I will say 10,000 more, whatever your tool does for you, it does to you. So if your tool generates a whole bunch of code that looks pretty much like it's going to work, that's what it's doing for you. And when you reach the edges of its capacity, now it's doing that to you, right? So I'm thinking about in the low-code tooling space that we were discussing, you've got your marketing team generating a solution using their low-code toolkit. And then when they reach the boundary, now it's game over. Now you have to bring in a full trained engineer and maybe start all the way over. I don't know how many of those projects I rescued from Microsoft Access that as soon as they started having 10 people use it simultaneously, oh, suddenly all the data is getting corrupted. And it's not working right. I'm like, right, you were at the limit. Three was the limit and you were well past the limit and now it's broken, right? And I think the same thing is happening with some of the LLM generated code. I mean, you've seen it, you've experienced it, right? It's trained on a bunch of code, probably trained on a bunch of software systems that are written in languages that don't have braces. And so it knows to add the opening brace, but you don't get the closing brace. And now your code doesn't even compile. Or worse, it compiles and does the wrong thing. And I think that to me as an engineering leader, one of my worries is in an effort to go faster, my developers will start taking shortcuts and accepting the code without thoroughly testing it or thoroughly understanding it. And now we've got a part of our system that has a bus factor of zero, right? If we talk about the bus factor being the number of people in your department who have to be hit by a bus on the way to work, to not understand the code. Well, if it was written by an LLM, the LLM has no memory of that. And now none of the developers know it. And now the bus factor on that code is zero. And that is not a healthy place to be. Some people like to call it the lottery factor, but I know that a lot more people get hit by buses than win the lottery. So I go the other way.

[00:28:43] Allan Stewart: That's also on brand for you.

[00:28:45] Dave Adsit: It is on brand for me.

[00:28:48] Allan Stewart: Yeah, no, I don't want to come off too negative because there is value in AI, in value in LLMs, just like there is value in low code tools. There are some times where that's the right solution, even for developers, right? Like there are things that I've done in the past where I'm like, yeah, I could probably throw together a quick like HTML page or write a little app for this, or I can just load a spreadsheet and spreadsheet isn't the right solution. But it's a good enough solution. And so I'm not trying to say that the LLM doesn't have value, but we just have to be careful. To your point, can we go faster? Maybe. Is faster better?

[00:29:36] Dave Adsit: That's definitely the question.

[00:29:38] Allan Stewart: Especially when we're talking about like typing is not the bottleneck. It kind of never has been the real bottleneck in software development. And so getting more characters on the screen faster, I'm not sure that that's going to be any better. So the question is, can it help us with the thinking part of software development? That is the hard thing. And so, you know, beware the day, right? Beware the Ides of March when you stop thinking and just accept the suggestions or heaven help us the day when the AI writes low level machine code and skips the part where we can look at it and see if it's actually what we wanted.

[00:30:19] Dave Adsit: Right? Yeah. And then when it hallucinates,

[00:30:22] Allan Stewart: you're, you're doomed because you don't even know that it's hallucinating anymore.

[00:30:25] Dave Adsit: Right? Well, and that's the thing is like, I, I think that it's, I think that you're wise in saying, Hey, let's not be overly pessimistic because there are some cool things that can happen here. You know, one of my developers tells me all the time that he loves that he can use the AI that we've got, which I think is the JetBrains AI. And he loves that he can use the AI integrated into his IDE to write his PR descriptions. And then we both laugh because the only reason he's writing his PR descriptions is because somebody else on the team insists that he do it and doesn't insist that they be useful. He's like, they do. It does a pretty good job describing what the change is. Also, you could see what the change is if you looked at the code because we commit in small batches and we deploy in small batches and we really only do PRs because some of the people on the team are very uncomfortable with committing directly to the main branch from your local machine, even though that is, by its very nature, a separate branch, right? If we're using a distributed version control system, which almost everybody is these days, my local branch is different than the remote branch, even if they are both named main. And my commit message and the code itself that changed is really enough to tell me what happened and what's different. And so anyway, when it eventually arrives on main, we have a history of what a computer said the change was in the code. I think actually his favorite part is that he can ask for a PR request or a PR description that has more emojis. And it will in fact, write more emojis into the text. And those are hard to type on most keyboards. I can understand that.

[00:32:08] Allan Stewart: But you know what? It doesn't do very well, at least from my experience, when it comes to writing a PR description or even a commit message, it does a great job at telling you what. And to your point, that's what the, the code already was telling you is what, what changed. What changed?

[00:32:26] Dave Adsit: This code file.

[00:32:27] Allan Stewart: It's, I find it is not good at describing intent. Why did this change? Why were we doing this? Is this a new feature? Are we fixing a bug? What was going on here? And to me, as a human, at least, looking at commit messages or pull requests, that's almost always what I want. I want to know why. I can see what you did. What were you trying to do?

[00:32:53] Dave Adsit: Yeah. Well, and for me, I rarely even look at those anyway. I want to look at the tests because I'm hoping that the tests tell me why you did what you did. But that's a fair point. And maybe, maybe if the AI could go through and read my commit messages and help me bisect the commit log and find where a change happened that I was, I'm curious about, that would be useful. And honestly, maybe it already can. And I just don't use the tool. I don't use the tools enough to know.

[00:33:22] Allan Stewart: So here's a question for you, Dave. Given that LLMs are better in some ways, right? And that regression to the mean kind of concept, should we use AI to replace all the junior developers on our team who have not yet developed themselves to the mean? And so therefore the AI will be better than them?

[00:33:43] Dave Adsit: No, very, very definitely and adamantly no. Because that is a self-limiting strategy. Right? If we replace our junior developers with LLMs, we will soon not have intermediate developers or senior developers. And then the AI gets to take over the world because it's the only one who knows how to write code anymore. No, but honestly, like writing some of the code that you get a human's feedback on and that you work with somebody more senior, hopefully you pair with somebody more senior or mob with somebody more senior. That is how we train and develop. We develop the good thinking patterns that developers need in order to become intermediate, senior, staff, principal engineers. And so even if we had great LLMs that could write good quality tests and code every time that was on brand for our context, etc. It would be a mistake to replace all those junior developers with LLMs. It really does become a tragedy of the commons in that case. For every individual company, the answer could potentially be yes. But for the commons as a whole, the industry as a whole, it would be an absolute tragedy.

[00:35:06] Allan Stewart: Yeah, I like what you said that the AI takes over the world. And I think it would. But not in the like Terminator kind of Skynet style. Right. I think it's a bit of a lowbrow style of like, oh, no, this invasive, this invasive computer program that has destroyed the world and now enslaved us. And now, Neo, you're just a battery to it. Right. Like, yeah, it's not like that. It's more like it's more like T.S. Elliott. And this is the way the world ends, not with a bang, but a whimper. It's more like that. We're just like drive ourselves into extinction.

[00:35:42] Dave Adsit: Yeah, I went way more lowbrow. I thought idiocracy. Right. It's it takes over the world, but it's idiocracy. Welcome to Walmart. I love you. It's not it's not a happy, healthy future for our species. Right. And of course, that is at the extreme. That's only if nobody learns to code anymore at all. But I mean, even for the people who do learn to go, I mean, we always have trouble with needing more developers and we want more developers. There's always more demand than there is capacity to to deliver. Yeah. And so if we were to take actions that further restricted the capacity, that would not be in the interest of our civilization. Our economies are even our own companies in the medium and long term.

[00:36:31] Allan Stewart: Yeah, I agree with you. And I'm I'm not even sure that we should give junior developers access to LLMs because I worry that they're not going to understand when it's doing it right when it's doing it wrong. And they may not learn the lessons that they're supposed to learn because they just hit tab and move on their way.

[00:36:52] Dave Adsit: Oh, come on. Next, you're going to tell me I'm not supposed to give my seven year old a chainsaw.

[00:36:58] Allan Stewart: He loves chopping wood.

[00:37:01] Dave Adsit: Well, I just strapped it forward so that it wouldn't stop when he doesn't want it to. Right. It's like there are power tools and they are called power tools for a reason. Right. They're not for every use case or every. User. And I think you may be right that a tool like an LLM that could generate a bunch of code quickly is in fact a power tool that should be used only in appropriate contexts. And I, I do think that we are not moving that direction right now as an industry overall. We have a mutual friend who is looking at a startup that's doing something he hasn't seen before in the industry. And so he went to an LLM. That's the LLM that he likes to use. And he threw in a whole bunch of parameters and said, can you write me code to do this? Cause he wanted to sanity check whether it's possible. And he's like, it did produce a whole bunch of code. And in the next step is validating whether that code is rational or compiles or is usable in any way. And he's like, and that's where it fell down because I don't actually know how to do that step. He's like, I I've got a bunch of code. And he's like, I could try to figure out how to use it. I could use that code and then see if it kind of sort of works for the use case I have. But yeah, I mean, it's a, it's a power tool in the hands of someone who doesn't know what they're doing.

[00:38:26] Allan Stewart: Yeah. And, and that just brings me back to this idea of, you know, it is the current version of AI, these LLMs. Is it going to change substantially change how we code? Yeah, maybe it's, it's probably going to make some changes if for no other reason than everybody is super hyped on it. And. Just like they wrote all the blockchain software a few years ago and you know, whatever fad came before that, like there's going to be changes, but it's a good, is it going to replace developers? I just can't see that happening. Because you have to get so precise in what you tell it that it reminds me of a commit strip cartoon that is called a very comprehensive and precise spec. And you can look that up. I won't, I won't ruin the joke by. Explaining it or reading it on a podcast. But, but the, but the, the gist of it is like, you have to be so specific. That's just code. You've right. You've just moved. You've moved the fence posts of where the goalposts of where code, what is code, but ultimately it's still code.

[00:39:33] Dave Adsit: Well, and I think that this is exactly why the current generation of LLMs is like the old rad tools. Yeah. And I think that's not true. I think that's not true. I think that's not true. I think that's not true. I think that's not true. I think that's not true. I think that's not true. I think that's not true. I think that's not true. I think that's not true. I think that's not true. I think that's not true. I think that's not true. I think that's not true. It's just, it's the same skill with a different toolkit. So with the RAD tools and the case tools, I had to get very, very, very precise about what inputs I gave them to get output that was useful. And the output was actually one level separated from being useful even. And I think the same thing is true here. You have to get good enough at what your inputs are that the output that is generated is actually useful code.