When working with software engineers, try to create metrics that still work when they are gamed … because they will be gamed
In my consulting work – helping software engineering teams and organizations develop a shipping mentality, and better align to businesses – the business half of the relationship often asks for a number.
Something to measure product and engineering by like they have for sales, marketing or customer success.
In this episode of The Hallway Track, Miles Sterrett – a longtime friend and Director of Engineering at CareerPlug – and I dig into what metrics actually make sense, which to avoid, how AI is beginning to warp many of the traditional metrics.
And how software engineers will try to find a way to hack any metric you put in front of them – so try to pick metrics that work even when they’re gamed.
Oh, and if you’re in an engineering leadership position, avoid looking at metrics at an individual engineer level – they don’t reflect value individually,
Transcript
MILES STERRETT
(00:00):
A CEO came and said something to the effect of, “All of the salespeople have one number. What’s the engineering one number?” And I think most engineers will be like, “We can’t do that.”
CHRIS VANNOY
(00:16):
Welcome to The Hallway Track. I’m Chris Vannoy. This week I’m talking with Miles Sterrett, a longtime friend of mine and Director of Engineering at CareerPlug, an HR platform built for franchise businesses running on a code base nearly old enough to have its driver’s license. We spend a lot of this episode on the one question every engineering leader eventually gets from the business. What’s your number? How do we measure engineering? Miles has actually built an answer to that. We get into the deploys per week, the stability metric called bug pain score, and the single most important rule of measuring engineers, measure the team, never the individual. We also get in the why he’ll happily tell his engineers to game his metrics and how AI has doubled his team’s throughput. If you’ve ever been asked to put a number on engineering and balked or felt that measurement turned into a weapon, this one’s for you.
Howdy, folks. My guest today is Miles Sterrett, current Director of Engineering at CareerPlug. How are you doing today, Miles?
MILES STERRETT
(01:20):
I’m doing pretty great. How are you doing, Chris?
CHRIS VANNOY
(01:22):
Oh, not too shabby. Not too shabby at all. So first off, talk a little bit about CareerPlug, what it is, what it does, all that fun sort of stuff.
MILES STERRETT
(01:31):
Yeah. So officially, CareerPlug is a franchise people operating system.
CHRIS VANNOY
(01:37):
Okay.
MILES STERRETT
(01:38):
What that means in reality is we’re HR software and we’re aimed at franchisors and they’re franchisees. So we’re aimed at, you kind of think of it as like we’re aimed at Gold’s Gym corporate franchisors and then the individual franchisees that own a Gold’s Gym location or multiple Gold’s Gym locations, that sort of thing. We started life as an applicant tracking system, which may be a more familiar term for most people. And then we’ve kind of expanded down the line in HR software from applicant tracking to onboarding and ongoing employee engagement.
CHRIS VANNOY
(02:21):
Okay. That’s a lot of stuff.
MILES STERRETT
(02:25):
Yeah.
CHRIS VANNOY
(02:25):
How long has CareerPlug been around at this point? Do you have any idea?
MILES STERRETT
(02:32):
I think that the proper number is 17 years from conception. The code base itself is at least 13 years old. I think it’s older than that and I’ve just lost track.
CHRIS VANNOY
(02:53):
That’s the definition of well-loved. Yeah.
MILES STERRETT
(02:55):
Yes. Yeah, yeah. Cool. The code base can nearly drive. I think we’re getting pretty close to the code base being able to drive. If
CHRIS VANNOY
(03:04):
Anyone would dare give it a license, right?
MILES STERRETT
(03:05):
Yes.
CHRIS VANNOY
(03:10):
How many engineers are keeping this well-loved code base up and running these days?
MILES STERRETT
(03:14):
We’ve got 18 engineers at the moment, including our most recent hire DevOps engineer, mutual friend of ours. And then of course we’ve got some engineering managers and then product people and designers as well. I mean, rounding out the product organization.
CHRIS VANNOY
(03:30):
Yeah. How is the work sort of arranged? I know one of your bosses at least, Andrew Robinson, if he’s listening, hi, Andrew, has given a lot of though into this sort of stuff. And so I’m curious how y’all are working these days.
MILES STERRETT
(03:45):
Yeah. So if you or anyone is familiar with Marty Cagan and other, we’ll say maybe Marty Cagan disciples or Silicon Valley Product Group disciples, that sort of thing, we are squad based and operate with product trios where we have a product manager and a designer and a tech lead or engineering major, some staff level kind of technical person grouped around a value stream. So for an applicant tracking system, we have a squad that we call Attract, and they’re the top of that funnel trying to attract applicants to jobs. And so they’re one squad, they have their own product trio and their own group of engineers that are just focused on that. And then the next squad in the funnel is called Evaluate. So they are focused on evaluating those candidates and trying to get them either into the hired or rejected state kind of as quickly as possible.
And then on down the line, we’ve got Onboard and Engage. We’ve got Insights, which is a lot of data related type stuff. And then platform team, we have a core systems team. So a lot of that, I realize if I have to say this because I missed a squad. I didn’t say them out loud and I think I said just about every other squad. So I have to say we also have a squad that we call Pit Crew. Pit is a specific acronym that I’ll let stay aside for now.
CHRIS VANNOY
(05:17):
You got to have a couple of inside jokes everyone’s talking about. Yes. The thing that’s interesting is I imagine when it comes to this sort of customer and this sort of analytics tracking, that’s a numbers game in a lot of cases. We’re talking about fairly high turnover sort of environments and customers, which why the product is necessary. But I have to imagine that also puts a lot of emphasis internally on how do we measure our own work and that sort of thing. Have you run into that? I’m specifically asking because when I work with clients lots of times, one of the questions is like, how do we measure effectiveness of our engineering org? And I’m like, well, be very careful, but we’ll get into that in a couple of minutes. So how have you stumbled into that at CareerPlug?
MILES STERRETT
(06:02):
Yeah, be very careful. That’s a good warning, a good description. I don’t know. Years ago, probably every person in our sort of position who finds themselves in a director seat or something like that has earlier in their career the story where a CEO came and said something to the effect of, all of the salespeople have one number. One number is the way or whatever. What’s the engineering one number? And I think most engineering. Engineers, we’ll just say most engineers will be like, we can’t do that.
CHRIS VANNOY
(06:39):
Some other time we should talk about estimating. Is this another sort of thing that businesses always want? When will this be done? And it’s like I.
MILES STERRETT
(06:47):
Yep. Sometime within the next 12 months probably. Yes, maybe. Anyway. Right. Yeah. I heard that a lot and was very hesitant a lot and tried different things. But like a lot of engineering leaders that I’m sure you’ve spoken to, the book Accelerate, I think did wonders for a lot of us.
CHRIS VANNOY
(07:10):
It’s around here somewhere.
MILES STERRETT
(07:12):
Yeah, it’s somewhere.
CHRIS VANNOY
(07:16):
My bookshelves are over that way, but yeah. It’s definitely on there.
MILES STERRETT
(07:20):
I know that there are lots of other frameworks beyond what DORA originally laid out in Accelerate these days and space and mixing in more of DevX, like developer experience and more of that employee engagement type of stuff. But when it gets down to the brass tacks, it’s always nice to go back to those DORA metrics and really rely on those and just build a little bit of abstraction on top of those.
CHRIS VANNOY
(07:50):
Yeah. For folks who aren’t familiar, what are those DORA metrics traditionally?
MILES STERRETT
(07:55):
Yes. Oh, good call. I’m definitely going to miss one, I think. No, it’s
CHRIS VANNOY
(07:59):
Okay.
MILES STERRETT
(08:00):
Throughput is a big one. Lots of different ways to measure that. I’m sure we’ll get into that shortly. Change failure rate. So how often are you deploying code that causes a failure? Meantime to recovery. So when there is an issue, and there are different philosophies as to what rises to what an issue is, but when you do have a problem, how quickly do you recover from that problem on average? What’s the one I’m missing, Chris?
CHRIS VANNOY
(08:28):
Dude, you’re the one who brought up the book, man.
MILES STERRETT
(08:30):
Yeah, I know.
CHRIS VANNOY
(08:32):
Regardless, I think you hit on the important ones.
MILES STERRETT
(08:35):
Right? Yeah, I’m sure I have it here somewhere in front of my face.
CHRIS VANNOY
(08:42):
Those are the important ones I think generally when you’re talking about the work that we do inside of software companies. And oftentimes we’re working in software. In my case, I worked in a lot of marketing technology where if you cause a bug, it’s probably not that big of a deal. Probably.
MILES STERRETT
(08:58):
Yes.
CHRIS VANNOY
(08:59):
Yes. There are certain areas where that’s not true. But that’s sort of like, are we making progress and are we not breaking stuff all the time? Is basically when you boil down to it what those metrics are sort of covering. How does that translate into how you work at CareerPlug?
MILES STERRETT
(09:21):
So we categorize our metrics as much as we can into a general throughput, which I remember now. Lead time, cycle time is the general throughput. General throughput.
CHRIS VANNOY
(09:36):
Sorry.
MILES STERRETT
(09:38):
We have a throughput category.
CHRIS VANNOY
(09:38):
Podcasting is a visual medium. I interrupted you by doing a little salute when he said general throughput. Sorry.
MILES STERRETT
(09:47):
Yeah, I’ll probably do that. We’re talking and can see each other on video, right? But that may not be everyone’s experience here shortly. But yeah, so we bucket things into the throughput category or the stability category. Throughput for us is we go so far as to measure deploys per week per engineer. And essentially what that means is how often are we actually shipping code to production? Usually this is a PR pull request on GitHub getting merged into main and auto-deployed. How often are we doing that? Per week and then per engineer. My preferred end metric is also averaged over the previous four weeks, so it’s not so spiky. Otherwise, I would run around pulling my hair out when four people went on vacation or whatever. So that is why I do per engineer. And I don’t actually look at every engineer’s PRs. I just say how many engineers were active last week?
How many deploys? And do the math. You know what I mean?
CHRIS VANNOY
(10:50):
My next question to just clarify is, because sometimes that’s the other question you’ll get from management is which of my individual engineers is being productive?
MILES STERRETT
(10:58):
And oh..
CHRIS VANNOY
(10:58):
No, I don’t want to do that.
MILES STERRETT
(11:01):
Yes. Yeah.
CHRIS VANNOY
(11:03):
And if you’re not careful those metrics, the per engineer one, you have to be a little bit careful about how you define it and explain it, especially to management because that might be one of the first questions of like, oh wait, per engineer? How’s Joey doing? Something like that.
MILES STERRETT
(11:18):
And it turns out that Joey only had two deploys because he was busy helping Frank, right?
CHRIS VANNOY
(11:25):
Yeah.
MILES STERRETT
(11:25):
The whole week. He got six deploys thanks to that.
MILES STERRETT
(11:27):
Yes.
CHRIS VANNOY
(11:28):
Yeah.
MILES STERRETT
(11:28):
Yeah. Yeah, exactly. Yeah, yeah, yeah. Very dangerous. There are lots of senior engineers who would lose their jobs if they were specifically being measured on deploys. It’s one abstraction above measured on lines of code or commits or whatever. So yes. And that’s the thing you got to convince engineers of as well often. Like, Hey, I’m not interested in measuring specifically Joey, your deploys per week, but I just want to know, is the team generally healthy? If you’re familiar with EOS, the entrepreneurial operating system, they characterize their scorecard as if you were for whatever reason stranded on a desert island, but you just wanted to know if your company was healthy, what are the metrics? What would you have to know? Yes. Yeah, that’s usually what I’m thinking about when I’m stranded on a desert island.
CHRIS VANNOY
(12:24):
Well, it’s like we said at the beginning of this, right? At the end of the day, what we’re trying to figure out is are we making progress and are we not making fires along the way? Right? Yes. So that’s really what we’re talking about here is how do we translate that into a number that then we can go satisfy the business’s need for you have a number that you’re accountable to all the time, right?
MILES STERRETT
(12:44):
Yes.
CHRIS VANNOY
(12:45):
And I’m curious about how you’re measuring the second half of that and not making fires all the time.
MILES STERRETT
(12:51):
Yeah. Our, we’ll say flagship metric there is called bug pain score. And I do want to give a shout out to engineering leadership coach Brian Dayton, who established a concept for this for me at least. But I know him as affectionately as Coach Brian. That’s who he is to me. But really good dude. So bug pain score is like, I think most engineering organizations or product organizations probably have a backlog of bugs and those bugs have a severity level or a priority level, whichever word you prefer to use. We use priority level. So a P1 is like an incident down to a P5 is like we’ll get to it if it happens to be in the area that we’re already working in or whatever.
CHRIS VANNOY
(13:35):
Or if the right customer complains about it.
MILES STERRETT
(13:37):
Yeah. That’s very true. Yes. Yes. We have several of those customers. Yeah. Gold’s gym may or may not be one of them. But yeah, so it’s basically, it’s pretty simple. I’m sure Coach Brian has a more sophisticated version, but straight up multiplier depending on its priority level. So a P1 is 40 points. A P5 is half a point. So then every week we measure every week, we take a measurement every week and put it in a spreadsheet. So any given week, I already know our measurement this week was 12, which is tied for the best it’s ever been.
CHRIS VANNOY
(14:16):
Oh, congratulations.
MILES STERRETT
(14:16):
Thank you. Yeah. I know earlier in this year we were looking at 40. And it wasn’t like there’s a single P1. It was like we got a lot of mostly P4, P5, but we had a lot of backlog. But yeah, we’ve really made some strides here.
CHRIS VANNOY
(14:38):
Yeah. Similar metric I’ve used sometimes as rollbacks. We deploy something and then we have to roll it back or we have to chase it with something. That’s again, that same sort of idea though of we’re deploying on a relatively frequent basis and we’re deploying stuff that’s of a quality that we don’t have to go fix it again later.
MILES STERRETT
(15:00):
Yes.
CHRIS VANNOY
(15:00):
In a lot of cases.
MILES STERRETT
(15:01):
Yeah, exactly.
CHRIS VANNOY
(15:02):
And I can’t remember if I’d read Accelerate. This is something I was working at a startup and stumbled on the very similar metrics to this. I didn’t do the bug score sort of thing. I’m going to have to go look up Coach Brian once we’re done because I did something way dumber, which was just how many? Yeah.
MILES STERRETT
(15:20):
Yeah. It’s got to start somewhere anyway. Yeah, yeah.
CHRIS VANNOY
(15:26):
And I didn’t break it down per engineer like you did. I was just doing over a time period, how many deploys is the trend line looking the right way?That sort of stuff. As long as we’re stable and going up on one end and stabling or going down on the other, and the more those lines diverge, the better off we’re getting. More deploys, less bugs. Cool, that’s where we want to go.
MILES STERRETT
(15:50):
Well, and probably a lot of other engineering teams are experiencing this right now too. So our throughput has gone up significantly since the first of the year. I’m sure everybody is saying these two letters on a regular basis, but listen, AI augmented engineering, there’s plenty of things to quibble about with AI, but AI augmented engineering just gets better and better. And so our throughput has gotten a lot better. You can probably guess from our pain score and what I described earlier that some of that throughput is just absolutely crushing bugs, just stomping on them all over it. There were cockroaches everywhere and they’re all gone. They’re smashed on the ground. The other side of that though is that, and this maybe is in our future, but sometimes you want to look at like, okay, but still how frequently are you introducing bugs? Are you shipping more software, but it’s sloppy?
Are you shipping more software, but it’s remaining at the same high quality? I do know that we are shipping the same number of bugs per deploy, we’ll say, as we were in January and February. Problem is we’ve doubled our throughput. So technically we are shipping more bugs. Raw number, right? Raw bug number, we’re shipping more bugs, but we are solving them like that because circling back around and solving them is so easy.
CHRIS VANNOY
(17:29):
Yeah, yeah. That’s actually interesting that it’s been a working theory underneath a lot of the way I think about this. We’re shipping more, but it’s the same number of bugs per deploy. It’s like there’s this constant sort of background hung base level of bugs I think in any software engineering because there’s a fundamental. There’s a level of bug that is just a lack of understanding of the problem you’re solving where you deploy something and it’s like, it’s working exactly how we wrote it and we intended it to. It’s that whole bug or a feature request. Yeah, yeah. Oh God. So many of those conversations.
But there are those level of bugs that sometimes pop up quite frequently that I think just it’s the nature of change. And so it’s actually a little bit validating to see even as AI assisted throughput goes up, there’s still a base level of bugs. It’s kind of just always there no matter what. This is also a thread from the last podcast episode where we had a theory that AI assisted development essentially makes more, sounds like not quite, but more pop-up fires. It just makes them easier to fix. So they end up being smaller and quicker to turn back around. Does that feel like it’s what’s happening at CareerPlug as well?
MILES STERRETT
(18:48):
Yeah. I don’t know if we’ve had. Well, we have had more because our raw throughput is more. So yeah, yes. So with bug pain score, we track a few other things. So some bugs get solved within the same week. So I can tell you even what bugs were open in a given week and their pain score. I measure every Monday. If I measure on a Monday and it’s 12, I can still look back and see it was 40 midweek last week and we just solved them all. We deployed a couple bugs on Tuesday, solved them Wednesday morning. You know what I mean? And so there’s still pain there, but we’re primarily focused on the current, the Monday morning pain score. Are we maintaining quality week over week as opposed to in any given sampled moment? And then things get a little dicey. But yeah, so we’re just able to recover so quickly, I guess.
Similar to the meantime to recovery. Fixing the bugs is honestly, maybe we’ll get there one day. If we could just have customer success describe the bug straight to cloud code, it might get solved that way sometimes.
CHRIS VANNOY
(19:58):
Well, or there are teams experimenting with hooking it up to their error tracking software and saying just make a PR for this, that sort of thing. That’s also interesting. The other thing I’m curious about that I want to pick at for a minute, you mentioned that you measure on Monday. Does the team know you’re measuring that on Monday? Because you hinted out a pattern there where it’s like you look on Wednesday and there’s like 40 and then somehow by the end of the week it gets back done to 12. So I’m wondering if people know what the measurement is and it’s like the Heisenberg effect, right? When you observe it, it changes the behavior.
MILES STERRETT
(20:31):
That’s a fair question. I’m sure that I haven’t been secretive about it or anything. I don’t think that’s what’s happening, but I’ll have to keep an eye out. I also, I’m relatively certain I get up, and this is just people have different schedules. Pretty sure I work earlier than a lot of folks. So maybe they’re like, “I need to close this out on Fridays so that my brain over the weekend doesn’t have to think about how there’s that bug that I didn’t fix.” But
CHRIS VANNOY
(21:07):
Yeah. Yeah. The other thing it reminds me of is that we mentioned earlier that you have to be a little bit careful about how you measure engineering. And I’m kind of hinting at that with that question because software engineers will game and abuse any metric you put in front of them, which I think you can see in the token maxing thing that’s been going along recently is if you’re going to measure them on tokens, they’re going to spend the living crap out of your tokens. If you’re going to measure them by PRs, they’re going to make a ton of PRs. If you measure them by lines of code, my God, they’re going to write some worded code. That’s the thing I like about these particular metrics you’re talking about is even if they gain them, I think you get only good outcomes, right?
MILES STERRETT
(21:51):
I think so. There’s a lot of controls perhaps to put in there. But I did a video series a few months ago just because I know many of the engineers that I work with are very senior and they’ve probably seen this stuff before and they’re like, “Yeah, yeah, whatever. You do your thing, I’ll do mine type of thing.” But some engineers have maybe never encountered metrics or maybe never worked at an organization that cares about them as much as I do. I really love my metrics.
CHRIS VANNOY
(22:27):
Or they’ve worked at companies where they’ve had those really gamified metrics or just dumb metrics. I’ve only hit a couple of dumb ones. There’s worse. Anyway, yeah.
MILES STERRETT
(22:38):
I did a video series a few months ago and I was like, “Here are the throughput metrics. Let’s talk about those a little bit.” And in particular, let’s talk about deploys per engineer per week and that it is not me looking at Joey’s. It’s me looking at the teams or whatever. And one of the things I said towards the end of that video is like, look, I know we engineers love to game systems like this. Please do it. Write real small PRs and ship a ton of them next week. Well, let’s see what happens. I think it’s going to be pretty great actually. I dare you to do it.
And plus, so we do pull request style, like a lot of companies do, I’m sure. And every pull request has to have at least one reviewer. The social construct I think of CareerPlug engineering and probably most software engineering teams, at least most Ruby-based teams that I’ve been on, you’re not going to get away with shipping a comment as a PR or something to game the system. It’s going to have to be meaningful code. So I’m not too worried about it getting too ridiculous. But I have seen it. I will not mention the particular tool. Chris and I talked about this yesterday, a couple days ago. Our podcast chat..
CHRIS VANNOY
(23:59):
Yeah.
MILES STERRETT
(24:00):
Yeah. Yeah. There is a tool that I’m sure this is still out there and people are using it to measure their teams. And at least at one point it would measure developer days. And my understanding is that meant how many days in a given week did you make a commit? But this particular team, I was talking to an engineer that worked there and he was like, “We make it so that Saturday and Sunday don’t count against you, but they can count for you.” So if you make a commit on a Saturday or a commit on a Sunday, it gets divided by five. The denominator is five. But if you do seven days worth, you’re more than 100% productive. And so I know there are multiple engineers on that team that would sandbag commits for Saturday and Sunday. And they would just get up in the morning, pop up with a laptop, get stash pop or whatever, apply the commit, push it up, shut the laptop, go about their day, do the same on Sunday morning.
And now they’re like, “Oh man, Joey and Frank are so productive.” I’m so glad that you said Joey, by the way, because I don’t know if I would’ve come up with a name.
CHRIS VANNOY
(25:13):
Oh, well, I used to be an English major, so coming up with names sometimes is one of the advantages of it. Referencing no particular Joey for the record. Oh man. But that’s almost like in basketball when you make a shot and get fouled. It doesn’t count as a shot attempt unless you make it.
MILES STERRETT
(25:37):
Yes.
CHRIS VANNOY
(25:39):
So it’s always one of those sort of things. Or in moneyball circles where they started looking at on base percentage instead of batting average, how a walk doesn’t count as an at bat. It’s stuff like that that’s a little. Yeah. Again, it’s just one of those, whatever metrics you put in front of software engineers, they will try and abuse it. And in that case, that’s another bit for don’t measure per individual engineer because we mentioned the ill effects earlier of sometimes the business will go like, “Hey, how’s Joey doing?” Or, “Who’s the worst of this group?” Something like that, which is also, just as an aside, a red flag for company stability, but I digress. Or the other side of this is that you will end up with individual engineers because they know they’re being measured specifically that will game it even more for themselves. Which is why I always fought really hard for like, “No, I’m not even going to try and break this down per engineer, individual one.
I’m not looking. I’m not going to run those numbers. You’re not going to get them from me. They’re all my favorite children.”
MILES STERRETT
(26:43):
It is dangerous. And for me, I feel like it’s even, to your point, I don’t even want to run those numbers because I might like looking at them and I don’t want to. Because if I start looking at them regularly, that individual, the flawed individual numbers will seep into my consciousness and into my decision making. And it’s just not the right way to make decisions about your team, about their team’s health or anything along those lines.
CHRIS VANNOY
(27:15):
Especially because like we pointed out earlier, the pattern generally is you get more senior into software engineering. You’re actually producing less code. Because I would tell folks I’m mentoring all the time, there’s two things. One that’s like becoming more senior in engineering is just you’ve seen and made more bugs.
MILES STERRETT
(27:38):
There’s a lot of truth to that.
CHRIS VANNOY
(27:40):
Yeah I know, right? That’s how I was a coding bootcamp instructor and it made me way better of a software engineer because I had all these student bugs all day every day.
It was like an accelerator for that. But the other thing is, as you progress in your career, your value is less in the code you write and more in the productivity you unlock for the rest of your team, which isn’t measured by a lot of these metrics if you do them per individual. But it can have a very big impact. One engineer might be able to push out six deploys in a week. But you can take that same engineer and they’re only doing two this week, but they made it so that four other engineers on your team did six each. So you end up being this force multiplier as you get more and more senior in your career until eventually you become a manager or a director or something like you and me.
MILES STERRETT
(28:31):
Yeah. And then even though you think you’re helping and unlocking people, at least some subset, “Oh, I have my lights on a timer. That’s fun. Y’all who are listening can’t see this, but all of my lights just turned off and I’m in the dark now. It’s a good time.” But then you’re like, “Yes, I think I’m helping even more. I’m leveraging, I’m making everybody better.” Definitely some people are like, “You are in my way. And that’s fair.” Sometimes I am. And you got to shove me out of the way. It’s fair.
CHRIS VANNOY
(29:08):
Some of my engineering manager coaching portion of my business, oftentimes it’s folks that are in transition for this. Where it’s like they are now a manager for the first time or they’re becoming a director for the first time, that sort of thing. And coaching them through the team ramifications of you doing code and where you have to be a little bit more careful about what code you pick up. Because you can’t take all the interesting stuff because then your team doesn’t have any interesting stuff to work on. And you can’t take all the boring stuff necessarily because it’s boring and you don’t really want to be doing that. So you have to pick and choose your spots a little bit.
MILES STERRETT
(29:50):
And I still code a little bit here and there. I’d like to say it’s to try to keep sharp. It is the. That a little bit. It’s also just because it makes me happy. I need to do it a little bit. It makes me happy. But sometimes I just want to like, oh, I’m going to jump in. Nobody else has time for this. I have a couple hours trap. I’m in a trap. But my mind is already made up and I work on our rails upgrade or I work on some docs or whatnot. Chris, today, this afternoon before the end of the day, I shipped a PR from June 15th. If I were to have a conversation with an engineer and I looked and they had a PR open from June 15th, I’d be like, “What’s going on here? Something’s broken, right?” Unless that’s extremely speculative and then you should close it because probably if it hasn’t shipped yet, it needs thrown away.
CHRIS VANNOY
(30:45):
The way I put it is code is like produce. If it’s sitting on the shelf too long, it’s going to rot.
MILES STERRETT
(30:52):
Yes. 100%. So I try not to be that person, but it definitely happens where I think I’ve got enough time, create a couple PRs, and then two months later people are like, “Hey man, are you going to do some of those, do you think? I’d like to close them.” Yeah, fair.
CHRIS VANNOY
Going back to metrics, that’s another interesting one. I think a lot of that sort of time to code review or code review churn, those sort of metrics that are a little bit further down from this can also work pretty well. Because sometimes, especially I think in this AI accelerated age, the bottlenecks for this work ends up moving. And I think one of the ones that’s popped up a lot are the code review. Last episode with Sam, he had a whole day where he did nothing but review code. It’s like he’d get done with one PR and then another one would show up and then another one would show up and then another one would show up. Smaller team than yours, so he ends up being the bottleneck for all the code essentially. But it’s like those other sorts of metrics end up playing. I’m going to ask one touchy question since you brought up we’re recording priority and stuff like that.
Are you in Jira or are you in something else?
MILES STERRETT
(32:10):
We are in Jira.
CHRIS VANNOY
(32:12):
How are you coping with that? I know you’re not a fan.
MILES STERRETT
(32:13):
Oh, is that widely known? That’s awesome.
CHRIS VANNOY
(32:18):
Well, it’s known to me at least.
MILES STERRETT
(32:26):
There’s the saying about what, having the serenity to accept the things that you cannot change. And that’s where I’m at with JIRA. To be honest, I know way more about writing JQL than I really want to. It’s actually kind of fun. Oh, it bothers me how well I know Jira at this point. But you know what? Jira is almost its own operating system in a way. It is.
CHRIS VANNOY
(33:00):
You and I have talked about this in the past, and I think my stance has always been like Jira is fine. Whatever tool you want to use is fine. It’s ultimately like the tool itself doesn’t actually matter. I’ve told clients this, you could put post-it notes on the wall if that works for you. It’s fine. You can do this in a spreadsheet. You can bring out paper notebooks. It doesn’t matter what tool you’re using to run the process as long as you’ve got the process figured out. And the other thing is, again, JIRA is fine and it’s kind of like the default choice, but you don’t really want to move it. If something’s working and if you’ve got it in GitHub issues or whatever, just stay there. It’s fine. Oftentimes when I work with clients, they’re like, “We’re going to move to new system. We’re going to move to Jira, for instance, or we’re to move from that to, I don’t know, Clubhouse or..
MILES STERRETT
(33:57):
Linear.” Linear or whatever.
CHRIS VANNOY
(34:00):
It’s like that’s not actually your problem.
MILES STERRETT
(34:03):
No.
CHRIS VANNOY
(34:04):
The tool is never actually the problem. And a new tool is not going to fix whatever was broken in the first place. Right?
MILES STERRETT
(34:12):
No. Yeah.
CHRIS VANNOY
(34:12):
Yeah.
MILES STERRETT
(34:13):
And I think to be honest, today I’m even more okay with Jira than I was a year ago because I don’t actually. Well, some of it’s that I don’t look at Jira myself a whole lot, but I think any given engineer probably doesn’t need to either because they have an MCP server. And so you could be like, “Hey, my preferred AI tool, here’s the ticket. When we get this ready for a PR, will you also move that ticket to where I want it to?” You know what I mean? Or like, “Hey, Claude, go find tickets in Jira that match this string. You don’t have to know JQL anymore. You don’t have to mess with Jira’s interface if it bugs you.” And you could probably have a whole series of episodes about that as the new interface for a lot of different tools.
CHRIS VANNOY
(35:09):
Yeah. I think that is one interesting thing is how AI can help fill in the gaps that were always there before. You hinted at one of them, getting engineers to go update tickets in JIRA is 20% of the job at least. Guilty. Yeah. I could imagine. Right? Or getting salespeople to go update Salesforce, right? Or getting customer success people to update Zendesk or whatever. Whatever the core system of record is, getting a human to actually go in there and update it on the regular is a pain. Nobody does it. Everybody’s data hygiene is awful. And if AI can help fill those tools, make that work easy, that is in the realm of what’s interesting about AI tools to me. How can they help us do things that were hard to do before or that we just weren’t doing before or were out of reach before?
Because it was too expensive one way or the other to do. This is interesting that that’s one of those tools too. All
MILES STERRETT
(36:11):
Right. Yeah. I already have built internal tools to streamline things that we are doing already, but we’re very manual or in some cases to do things that I never thought we would be able to do because who’s got the time? You know what I mean?
CHRIS VANNOY
(36:27):
Yeah. You got a good example for that? Something you can share?
MILES STERRETT
(36:31):
So I was saying earlier that we track all of our metrics in a spreadsheet. So it’s like week starting X day, and then all of the metrics for that week. A spreadsheet is a terrible, or I should say columns and rows are a terrible interface for seeing trends and making decisions based on data. But we’ve got, I mean, it is a massive. I can’t. Well, I happen to know some of your recent clients, so it’s probably not the most massive spreadsheet that you’ve ever seen, but it is close. I have also seen some actuarial spreadsheets and it’s not that, but second rung under that maybe. In any case, it’s very big. I can read it, but it’s a lot to go through and you can’t really see trends if you can’t squint at it and get a graph out of it very easily. So we created a quick tool that just effectively uses the spreadsheet as its database and creates graphs and creates the upper limit and lower limit based on past calculations.
And so you can very quickly look at a trend line of the bug pain score and see like, oh, the bug pain score has been out of control effectively for the past three weeks. We got to do something about that. Or a much more positive case for us recently is it’s actually been low out of control and out of control in a positive way. We probably need to recalculate what our upper and lower bounds are.
CHRIS VANNOY
(38:05):
Yeah, what the baseline is just shifted.
MILES STERRETT
(38:07):
That’s great. This is stuff I’ve always wanted, right? But who’s got the time? I don’t even have the time to figure out how to make sure there’s these charts in the spreadsheet somewhere to look at.
CHRIS VANNOY
(38:19):
And some weeks I didn’t even have time to up because I stored the metrics I was talking about. This is pre-AI age, right? In a spreadsheet, right? That’s what you do. And some weeks I would forget and I wouldn’t have time to go in and even update the spreadsheet. So I’m actually, where I thought you were going with that was having a tool that would automatically pull the data out of JIRA and go update the spreadsheet for you.
MILES STERRETT
(38:41):
Oh, I do have that also. Okay. Yeah.
CHRIS VANNOY
(38:45):
I knew it.
MILES STERRETT
(38:48):
Yep. I wake up Monday mornings and it’s all capitals, right to scorecard equals true and then a rake task. Yeah. Good old work. And anything that’s not calculated from the underlying data, just gets all the underlying data gets just shoved in there. Yeah, it is wonderful. There’s an iteration of that. Again, if I take the time to do it and make it even deeper, but yeah, it’s fantastic. Very cool. Now I don’t spend four hours or whatever Monday morning writing my spreadsheet. I spend four minutes.
CHRIS VANNOY
(39:21):
There you go. That’s way better. All right. Thanks for the chat today, Miles. This has been absolutely fantastic. Thanks for the invite. I know you like high fives, so I’m going to give you one through my screen. There you go, man. Where can we find you online and sort of follow what you and the team are doing?
MILES STERRETT
(39:41):
Oh yeah. I am mileszs pretty much everywhere. If I’m being honest, LinkedIn, find Miles Z Sterrett on LinkedIn. I don’t post a ton, but that’s your best bet. If you are local to Indianapolis, find me on the IndieHackers Slack. Shout out to Chris, the president of IndyHackers here. Thank you. Yeah, I’m on Bluesky, MilesZS. That’s the other most likely place I might post.
CHRIS VANNOY
(40:12):
All rght. Awesome. Thanks for taking the time today, man. I really like…
MILES STERRETT
(40:15):
Yeah, man, thanks for having me.
CHRIS VANNOY
(40:16):
A CareerPlug. Say hi to all my friends that also work there.
MILES STERRETT
(40:20):
Will do. Absolutely.
CHRIS VANNOY
(40:22):
All right.
MILES STERRETT
(40:22):
Thanks, Chris.
CHRIS VANNOY
(40:23):
Have a great day. All right, thanks.
That was Miles Sterrett, Director of Engineering at CareerPlug. Three things stuck with me. You can measure engineering, but measure the team, not the person. The moment a metric points at an individual, it gets gamed and it poisons your own judgment about your people. A good metric produces good outcomes even when it’s gained. Miles literally invites his team to game deploys by shipping lots of small PRs because that’s just shipping. And in the AI era, the math flips. His throughput doubled and the raw bug count went up, but recovery got so fast that shipping more and fixing faster beats trying to prevent every bug up front. If this was useful, follow the hallway track wherever you’re listening so the next one shows up. I also write a weekly newsletter called Expedite over at getaxiomatic.com. It’s more of this kind of thing on a regular cadence.
And if you run a franchise and hiring is your headache, go look up CareerPlug. You could find Miles himself on LinkedIn and Bluesky. Thanks, Miles. See you next time.