Crafting Code Podcast

[00:00:16] Allan Stewart: Welcome to the Crafting Code Podcast, where we discuss the importance of doing the right thing at the right time with the right tools. I'm Allan Stewart, a software architect, and lately I've been wondering whether adopting AI coding assistance is effective generally, or if using them merely constitutes a different style of development and the efficacy is tied to how well the developer fits that style.

[00:00:40] Dave Adsit: I'm Dave Adsit, a VP of engineering, and recently I've been thinking about the value of taking time to do fun things together as a team, whatever fun means for your team.

[00:00:50] Allan Stewart: Our topic for this episode is working on cross-functional teams, part two. As we mentioned at the end of the last episode, we've broken this up into two parts, and now the conclusion. So let's talk about another specialty that we work with on a cross-functional team, and this is going to be some manner of operations person. On some teams, this will be a DevOps engineer. Other teams, it's just this is the operations person that works with your team, like the UX designer. The designer is much more likely that they will be shared across multiple teams. There's some operational function that they are providing, and a specialty there of running the software, keeping machines or virtual machines or Kubernetes clusters or whatever it is, legions of Lambda functions running, and that the system actually stays up, that the DNS is not broken, and that you can actually access your database.

[00:01:53] Dave Adsit: Yeah. I can't help but talk about DevOps without mentioning that if you have DevOps in your title, something is wrong. But that's not the topic for today. The topic for today is about how to do DevOps and how to collaborate across these functions. And I've recently been rereading the DevOps handbook, and one of the concepts in there is the idea that you can have embedded DevOps, embedded operations engineers, or you can have operations engineers liaisons. And these are different ways of engaging with ops. One of them is, to your point earlier, the operations liaison is likely working with multiple teams, or by definition is working with a bunch of teams. And the embedded operations engineer is going to be a member of your team and participate in all the same meetings, and they may spend a certain portion of their time actually writing code with the team because there may not be enough server maintenance to do. Yeah. Yeah. Yeah. I think the key concept there is if you are experiencing an issue and you are trying to debug the issue, that puts you in the software developer camp. And if you're trying to restart the server to get it back to a healthy state, that puts you in the operations side of the camp. And so even if we're doing developers doing ops, the mindset matters. And so does collaboration. Again, this is one of those roles where we need to be collaborative. We need to be collaborating across the two sides. I expect that the operations engineers are sharing operational concerns. Hey, every time you do this in the system, utilization on the CPU spikes to 97%. Like, oh, well, we should minimize the amount of times we do that until we figure out how to make it less impactful.

[00:03:44] Allan Stewart: Or put an index on that database table or change our query. There's lots of things that can make a big difference if you're observing, right? If somebody is keeping track of. Is this working? Is this performant?

[00:03:55] Dave Adsit: Yeah. So that's, that's one of the key things that a developer should expect from an operations specialist is observability help with observability, creating dashboards, capturing logs, capturing like often in many organizations, the engineers, the software developers do not have access to the production systems and nor should they. And instead we've handed that off to operations specialists who do have access. To those systems, but don't have access to write code. There are some reasons why you have to divide it up that way. So as a developer writing software on a team, I expect to be able to see what the system that I'm responsible for is doing in production. Can you get me the logs? Can you get me the metrics? Can you tell me the server utilization, the CPU, the memory, whatever? Can you show me how many pods this is auto scaling to in the Kubernetes cluster? And is that within expectation for the number of requests we're getting? So that level of observability in the system is something that I think that we should expect from our operators. And also I'm going to jump ahead a little bit. They should expect it from us. They should expect us to instrument the code and the system in a way that they can provide that because it's valuable to them as well. And so that is one of the key areas where I think it's essential that we collaborate across the functions and we have expectations of each other for how to, what to do, what to deliver so that we can deliver well.

[00:05:26] Allan Stewart: Yeah. And, and one of my expectations there too, is that as they're helping us observe the system, that they're giving us some monitoring capability in a way that is useful and readable. Right? So oftentimes this is some kind of a graph, right? Give, give me a graph that I can read and understand, because I don't necessarily want to have to go log into AWS or log into GCP and go. And. And dig something up, you know, help us unearth those and make it visible and, and similarly alerting, right? Once we know what's going on, let's put some alerts in place so that if it goes wrong, that we can identify it quickly and resolve the problem and, or even know what the problem is in the first place.

[00:06:16] Dave Adsit: Yeah. One of the other things that I expect, and we've already talked a little bit about standing up servers and maintaining servers. I expect help in automating build and deploy, and that can include, you know, writing the Terraform to stand up the servers or the Kubernetes containers or pods or whatever. And it could also include setting up the build pipelines and ensuring the build pipelines are doing the things that we need them to, and they're optimized so that we can take code from commit to deploy in a rapid period of time. So. We can test. I think that those are things that we should expect from our operations specialist.

[00:06:58] Allan Stewart: Agreed. That, that has become one of the things that becomes, it's like a, one of the first things that I look at. If I am working with a new team or if I've switched companies or, or whatever, there's some, or if I'm helping advise somebody, one of the first things that I start looking at is how does your code get from commit into production? Yeah. Yeah. Yeah. Do you have a staging environment? And the answers to these questions, you know, yes or no, they're not all necessarily, oh, this is good. And this is bad. Like, I don't think that you necessarily have to have really complex staging and UAT and like all of these environments, because that can also get expensive. But I want to know if we're going to make changes, how do we get those changes to production? What does that look like? And how do we automate it so that we can make it? And, and then hang all these other features off of that build pipeline. Yeah.

[00:07:59] Dave Adsit: I love that. The assumption is that it's from a commit to a deploy early in my career. It was from a developer machine. I'm just going to FTP this in, or I'm going to log into that machine and file copy over my changes. And so I'm, I'm really happy that as an industry, we have evolved and progressed enough that we are almost all using source control almost all of the time. One of the things I expect, and this is perhaps a change over many years, decades, even is that ops won't be an impediment to making changes to the system. I expect the ops is a collaborator and making changes to the system and not a roadblock. And frankly, that has not always been the case. There was, you know, for a long time in my career, there was an expectation that ops would try to prevent you from releasing new things. Right. Right. Right. Anytime you made changes to the production system, you destabilize or made it less stable and you may, you could potentially cause problems for them because they're the ones on call. Yep. Right. And so I expect them not to block changes.

[00:09:05] Allan Stewart: And I think one of the things that helps with that is when they, and something that I expect is that they'll help us build self-service tooling, right? Like I don't necessarily want to have production access to things. I don't necessarily want to be able, able to. I don't necessarily want to be able to query the production database, but sometimes I need changes to happen to that database and I need their help in making it so that we've got the tools ideally. So they're push button kinds of things, just like with the deploy that here are things that we can do so that I don't have to have the keys and that you've helped ensure that anything that I could press is safe to do because you have that responsibility. You've got those keys, but you're also not the blocker. If you're in a meeting. And I want to deploy. I don't want to have to wait until you get done with the meeting just so that we can ship something to Proud.

[00:09:58] Dave Adsit: Well, and one, one more thing along that line is I expect ops to work in small batches, just like I expect development and product management and design to work in small batches. I don't want to take, you know, huge amounts of downtime so that we can migrate to a new system. I don't want to, I don't want to spend months and quarters. Trying to do an operational change. I want to be working in batches that are reasonable and easy. I mean, reasonable in both senses, easy to reason about easy to reason about, and also not excessive. Yeah. Those are two of the concepts that I want an operations specialist on a team to be thinking about. And of course there's a lot of expectations that the operator should have for the developers. Yeah.

[00:11:22] Allan Stewart: production. Yeah. And I think that they deserve to expect that we will bake in the quality and the observability hooks, like you were talking about before, so that those things are available, that it can actually function. Because it is very difficult to tack that on later, right? If you write code that you didn't think about the performance of it and you get it out there and now people are using it, it's a lot harder to take it back. And so you need to understand how that's working. And then I think that developers should plan to be on call for the products that they develop, right? I feel like it's a partnership. And in some cases, the operations person might be kind of like the tier one person, depending on how that's structured in the company and how many teams, how many people are available. But if something's not working, I think developers should know about it, be on call to help fix the problem. Because it's not always a problem that can be solved by turning it off and back on again, or by throwing more RAM or memory or compute or whatever it is at the problem.

[00:12:36] Dave Adsit: I definitely agree with that. In fact, most of my teams, I reverse that. I make the software developers tier one and ops is tier two. If you can't solve it on your own, you can get somebody but it leads to a different type of outcome, right? Different type of focus on quality and delivery. And one of the things that I think that operators should expect from developers is an understanding that there are shared resources. I really hate to call people resources, but your operators are shared among many teams. And so are many of the systems that you use. If you use databases or message brokers, there might be, priority issues that have to be resolved for those systems that are going to delay the work that we want done for building, deploying, and running new systems that we're working on.

[00:13:30] Allan Stewart: Absolutely. I feel like this is a parallel to what we hope that our product managers will be respectful of, that there are these technical issues that we need to deal with. We need to work on the technical health or pay down technical debt, depending on how you're thinking about it. Well, the same thing happens for operations. At that operational level, there are priority issues that we need to be able to be respectful of and help them with it and also avoid writing code that causes problems in those areas. And then kind of related to that, make sure that we're writing code that doesn't make the operational spend excessive. If we can spend a little bit of time and provide some savings on the operational costs, they're going to help us understand it, observe and know what it's costing us to run the product. And then we help them out with that. I think that one's fascinating because in most of the

[00:14:30] Dave Adsit: organizations I've worked in, responsibility for the operational spend falls to the heads of the operators. But most of the ability to impact that spend is in the operational spend. How we build and deliver the software. And so that one is one that I think, you know, operators really should expect developers to write sufficiently performant code. Okay. So the next role, various titles for similar roles, QA, SRE, SET. So if it's QA quality assurance, SRE, site reliability engineer, SET, software engineer and test, it's a, you know, Like doing testing after the fact. And the first question that comes to mind for me is, should this role even exist and be separate than software engineer on the team?

[00:15:22] Allan Stewart: Yeah.

[00:15:23] Dave Adsit: And for many teams that I've worked with, the answer is no. Responsibility for quality has to fall firmly on the shoulders of the people who are writing the software. And they will get that quality by doing automated and manual testing for each release that they do. But that's not the case for every organization. And so you will have times when you are, you know, working with a QA engineer or a team of QA engineers that are helping your team validate something before you put it into production. I think one good scenario for that is when you have a dozen development teams who are each highly responsible for delivering quality within their own space. It is common that the interstitials, the, or the system at large is tested by another group because no one is thinking about it at the macro scale. Everyone is, you know, they're not necessarily focused on delivering high quality within their service, but you need somebody who is also, also not instead of, but in addition to looking at the whole system.

[00:16:23] Allan Stewart: Yeah. I really do appreciate when somebody does bring this specialty to the team that is, you know, separate or more extensive on the testing side than the typical developer. I think they can help us find problems in a new way, right. Make or avoid problems. Really? Because they're thinking about it in a different way than the typical developer. And I think that there there's a value there. And so we can debate whether it should be a thing or whether it should be embedded as, as part of just the engineering team. But if you are working with them and they're separate, one of the expectations that I have is that they will help the team see those other perspectives on how their code will be used and more likely misused. I know I've fallen into that trap many times where I was like, oh, well, this is, this is the way that it's supposed to work. And so obviously users are going to do this and they're never going to click on that button before they click on this button, but they do. And somebody with this testing mindset is going to help you to be able to see that and understand why it is that your code is breaking.

[00:17:30] Dave Adsit: You know, I'm sure that most of our listeners have seen the video, but we can link it in the in the show notes of the, the square hole. QA and the square hole. The girl is, has various shapes that each have a shape hole that they fit in a child's toy. But it turns out every one of them fits inside this shape hole for the square. Right. And you just have to turn it to the side or, You know, there's one of the common jokes of a QA engineer walks into a bar and orders a beer, orders a million beers, orders negative five beers, orders a duck. You know, like not always are engineers. Yeah. You know, you're, you're thinking about the system in the way that a QA, somebody who's been tasked with the, you know, your, your value is in that you have found ways to break the system that we did not anticipate. Yeah. Right. That is a, a different mindset and it's a mindset or a hat that we should wear while we are doing software development. And if we have somebody who's doing it the all the whole time, they're going to help us see these different perspectives of like, you know, the users who are not quite either don't understand what they should be doing or are intentionally trying to subvert our systems. I think we should expect that a QA engineers are writing automated tests. They are automating the testing that they're doing so that we can run it repeatedly without having to wait for them to become available as people. There's a lot of value in running tests over and over and over, and hopefully it's a, a fast test suite so that we can get quick feedback.

[00:19:12] Allan Stewart: Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. stuff's going to get missed. Stuff will be broken. They might forget to do one of the checks that was supposed to be done. But if we're automating that stuff so that anything manual that is happening is more exploratory and like, hey, let's try something that we wouldn't normally test for, they can find some new and novel things that are happening in the system by doing that.

[00:20:08] Dave Adsit: Yeah, I think that's especially relevant to those larger scale systems where there's interactions between many different subsystems developed by different teams. It is hard for somebody to reason about all of those things all the time. And so doing that exploratory testing and seeing what are the edges, what are the limits, is super valuable form of feedback to the software engineering teams and the product managers.

[00:20:34] Allan Stewart: So what should developers be expected to do, right? Like our quality assurance or software engineering and tests, they should expect that developers will be responsible for the quality of their code.

[00:20:50] Dave Adsit: Yeah, there's a saying that we like to pull out every time we talk about QA is that QA should find nothing. We should have already tested everything and delivered high enough quality code that when we hand it off to the testers to look at, they should find nothing. That's the goal.

[00:21:07] Allan Stewart: Yeah, absolutely. I have seen, I had experiences where developers would just take their first pass at something. And like, I'm not even sure that they would run it, to be honest. Like, didn't even try it. They just throw stuff over the wall and say, okay, well, if it, this will probably work, but if it doesn't work, then QA will figure it out. And that was a slow, agonizing, painful process.

[00:21:35] Dave Adsit: You and I have worked with developers, developers who committed code and went home for the day when that code did not compile. So not only did they not test it locally, they didn't run the unit tests that existed for the system locally. They didn't even build the application. And this is when we were working in a statically typed language with a, you know, a very robust compiler. And so that for me, that is missing the mark when it comes to validating the quality of the code that you have delivered. Yeah. So one of the other things that I think that QA engineers should expect, and this has been a hotly debated topic on some of the teams that I've worked on, but I think as an industry, we have finally gotten into a reasonable, sane place. QA should expect developers to help them provide the testing hooks necessary. You know, early on it was when we're using things like early, early versions of Selenium, and you had to have, you know, a lot of classes or IDs on elements if you wanted to interact with them through the browser automation. I worked with developers who were like, no, we will not add classes to these elements if they are not relevant to the styling of the elements. And it turns out that that was nonsense. In order to have high quality software, in order to validate that software with automated tests and have a good relationship with QA, it's necessary to sometimes add hooks to things. Maybe add extra APIs that are only available for testing and they're disabled in production. Maybe we need to, like I said, we need to add tags or IDs to elements in the HTML, or maybe we need to include a UI testing library into our application so that it exposes hooks that

[00:23:25] Allan Stewart: they can use for testing. Or provide data, easy ways to create certain data conditions or things

[00:23:32] Dave Adsit: like that. Yeah. I think all of those are things that QA should expect developers to do. And again, this comes back to, we want to have a collaborative relationship, not a combative relationship. We don't want to be gatekeeping the software. We don't want QA to be gatekeeping

[00:23:48] Allan Stewart: delivery and deployment. It just makes me think of QA and operations. We don't want them to act like Monty Python's Black Knight. You shall not pass. This code shall never get to production. But instead we want them to be helping us. It's like, oh no, let us help you pass this hurdle.

[00:24:05] Dave Adsit: Exactly. Again, it goes back to a whole team approach of collaborative deployment. So one of the other common roles that we see interacting with a software development team on a regular basis is the architect. That's actually your role. That's your title, right? And that's the role that you've had in several organizations where we've worked together is the architect or the chief architect on the system. This is a role that is responsible for It's a technical vision for the system. Certain decisions have to be made at a system-wide level, and the architect is the one who has responsibility for making them. And so developers should expect an architect to provide that technical vision. In this system, we are going to be doing microservices that communicate primarily through REST APIs, or we're doing microservices that communicate primarily through PubSub through Kafka, all of the messages will be of this same shape. They're either all going to be protobuf shaped if we're doing API calls, or they're all going to be JSON, or we're going to do versioning of API endpoints internally this way. There's a lot of things like that that we expect the architect to provide to the team so that they can stay in line with the vision,

[00:25:31] Allan Stewart: the overall technical vision. Yeah, I think engineers should also To be available to answer their questions, to help them understand difficult concepts. Architects are often engineers who've been around the block, and they can help you understand a concept, but especially when it becomes a systemic thing. It's something that they don't normally have to interact with as part of their day-to-day working on their code, but now they're going to interact with a different part of the system, or they're going to be interacting in a

[00:26:17] Dave Adsit: Yeah, I think that's a really good point. And I think that's a really good point. uh, system concepts as a whole, um, sitting down and teaching. Sometimes it means leading training. Sometimes it means hands-on coding with a team. Sometimes it means showing the code from other teams and how they're doing certain things. And so some of those system wide concepts there, they are the purview of, and the responsibility of this architect. And they are something that the team should expect the architect to provide to them and help them with.

[00:27:07] Allan Stewart: Several of those things all kind of sum up for me is that we want to expect that the architect will stay out of the ivory tower in as much as the ivory tower is kind of the, you know, classic, in my opinion, anti-pattern of what it means to be a good architect. You know, the ivory tower is where they go off to be siloed away from the team and hand down the edicts of, you know, thou must do this and pay thine developer taxes. But without engaging or understanding what's going on. So yeah, stay out of the tower.

[00:27:41] Dave Adsit: Stay out of the tower. I do encourage my architects sometimes to climb onto their ivory stepstool. You know, the, the architect, as we've been discussing is responsible for making good decisions and come and sharing those decisions. And sometimes that comes down to exercising authority over parts of the system. And I also want them to stay out of the, ivory tower, but I do want architects to make clear decisions and communicate those decisions clearly across teams. And part of that means sometimes being the decision tiebreaker. If there's multiple good opportunity or multiple good versions of implementation, we could do X or Y or Z, and it doesn't really matter which, but we have three people who are each championing one of those, one of those options, then the architect should be comfortable coming in and saying, we're going to do this. And why are we going to do this? Because the decision needs to be made so we can move forward. Sometimes it's as simple as that. Sometimes there are actually legitimate reasons like, well, we're not going to run SQL server because we're not an Microsoft shop. And that would be weird when Postgres or MySQL meet our needs just as well. You know, if we have multiple varieties of the same thing, we're just going to pick one and we're going to all use the same one. We don't need to have multiple relational databases in our system.

[00:29:07] Allan Stewart: Right. And you mentioned before, you know, there's, there are often these interstitial spaces that exist between the teams. And I think that that is a good place for architects to have ownership. And so there are things that they can use that authority to say, Hey, this is outside of your box. You know, this is outside of what your team owns. So you don't get to make the decision. This affects many teams. And so the architecture team is going to own it. And that, but hopefully not over-specifying the details, right. They're clarifying why it needs to happen and, you know, how it needs to be, but, but giving some room to, for the teams to build within their boxes.

[00:29:51] Dave Adsit: Yeah. To get really concrete for a second, which is against the rules of the architect, but we're, we'll allow it. I don't want my architects to be doing enterprise data modeling. First of all, because I've never seen an enterprise data modeling project finish or work, at least not successfully. I've seen multiple of them be abandoned, but I don't want, I don't want the architect to deciding what fields belong on an address and what fields belong on a user, because we're all going to use the same user library. I want them to be working on things like, Hey, in the, in this interstitial space, we're going to be using Jason messages over ref over rabbit MQ. That's how we're going to communicate primarily. And when that doesn't work, when we needed a, synchronous communication, it's going to be HTTP with those same Jason messages. Great. I love this for our architecture. I love this for our system, but that goes back to not over-specifying specifying sufficiently so that we can move forward, but not so much so that we prevent progress from the teams. So what can the architect expect the developer to do? I think the developers should recognize that they work within a system and be good citizens of that system. If the architect, Hey, we're doing JVM development. Let's assume that means Java and less otherwise specified. And let's not go write a huge Scala app that no one else can support after we go find a better job that is Scala specific. Yeah. Right. Let's be good citizens of that architecture of that system that we are inhabiting. And I think the architect should expect that. And the architects should expect that along those lines, the developers are doing the best work they can so that we have high quality across the system.

[00:31:41] Allan Stewart: And developers should not try to keep the architect out. I have worked in various positions where I've either seen or experienced that, that there is some systemic thing that needs to be communicated with a team, but the team doesn't want to hear it, that they make themselves unavailable or scarce and it hurts the company as a whole. And so I think that's a good thing. I think, you know, get to know your architect and make friends with them in as much as they're keeping themselves out of the ivory tower. You can let them in to your team a little bit and

[00:32:18] Dave Adsit: work with them. So those are the more common roles that we see as part of cross-functional collaborative teams. There are other less common roles, depending on your team. You may have product marketing or marketing as part of your team. You may work with, you know, you're doing a lot of releases and they're very customer centric and you may work with somebody who is a product marketing specialist on a regular basis. They're probably shared across multiple teams, but might be someone you see weekly in a meeting where you talk about what's upcoming. You may work with customer support or support engineering often. That happens a lot when you have businesses that either have large customer bases that you choose to provide email or phone or whatever, chat support, or you may be delivering a technical solution that requires support engineering to help implement. Another role that I see quite often, and I think is essential for a lot of modern companies is data engineering, data analytics, data science. That's probably three separate roles depending on your company. And, but each of those people is going to have things that they need from your team and provide to your team. And another common one is security, security engineering, a security specialist, a chief information security officer. That's another role that is less common for every organization, but it's still fairly common across

[00:33:47] Allan Stewart: the industry. So regardless of which one of these that we're working with, I think the common things that we've seen across all the others apply, right? We want to work with them to define the boundaries, right? Like what do they own? What, what specialty are they representing that We want to respect and how are they going to engage with our team, right? So working with them to define what is that working relationship like is very important. And it's something that I expect any professional to come and engage with the team and not just assume that the way that worked, you know, at my old school, we never rode magic school buses, right? Like come to me with the context of this company and let's work.

[00:34:35] Dave Adsit: That's right. And aligned with that, they should expect from us and vice versa, but they should expect from us to provide transparency around deliveries and projects and process progress. You know, all of those specialties need to understand what's going on in engineering in order to do their job well. And we need to provide them with that transparency. That could be as simple as inviting them to your weekly, your standup once a week, as Slack message to all of the stakeholders on the progress of project foo, or it could be as easy as just giving them read-only access to all your digital Kanban boards. So they can see what you're working on and where you are with it. Yeah. Or notifications when deployments happen. Yeah. And related to that is be available for collaboration and questions. Product marketing comes to you and says, Hey, we're releasing this new feature next week. How does it work? Give them the information. They need in order to successfully do their job. Like honestly, looking at each of these roles as they succeed, we succeed as software developers. We're all part of the same organism, the same company. And so all of us need to succeed together to really move the market, move the, move the company forward.

[00:35:54] Allan Stewart: Yeah. I think we always want to make sure that we understand what their role is, what they're trying to accomplish, how that's going to help the company then be available to collaborate. Yeah. what they're working on, what they're working on, what they're working on, what they're working on, what they're working on, what they're working on, what they're working on, what they're working on, what they're working on, what they're working on, what they're working on, what they're working on, that I am not currently employing with whatever my regular engineering work is, has been super valuable. And so to kind of wrap it all up, let's be respectful of that. Let's make sure that we're open and inquisitive with all of these things and figure out ways that we can work together, collaborate together and build a better product. The thing that we are creating with code will be better because we allowed them to lend their specialty in helping us build it.