Matty: 0:00

You know, I've spent a lot of my career recently working in developer relations and a lot of folks in DevRel be like business doesn't understand DevRel, and I've come to the conclusion that it's the opposite problem. And the problem is DevRel doesn't understand business and I'm not, you know, slamming on developer relations. People that's me, devrel are the only place.

William: 0:39

And if you talk to anybody else in any other kind of job and said I don't feel like I should show the business value of what I do, they would say what? All right, folks, strap into your X-Wings and power up those hyper drives because it's time. I'm with me. As my co-host, Yvonne Sharp. She is a cloud DevOps Jedi who's been battling the dark side of downtime and latency across the galaxy for ages. You know, when she's not wielding her lightsaber of automation or force pushing deployments to the cloud, she's keeping the rebellion, or at least the servers running the rebellion, running smoother than a freshly waxed Millennium Falcon. Yvonne, they say the force is strong with you. I know the force is strong with you, but I say it's the puns that keep us skywalking. How's it going today?

Eyvonne: 1:21

Let's just say I'm daily battling with the powers of the dark side. Yeah, trying to balance the pull of the challenges of leadership with the forces of the rebellion out there actually doing the real work, so trying to ride the line.

William: 1:41

Yeah, with us we have a pretty special guest, one that I pray I don't need to introduce, pretty well known in the you know, especially the community circles that I frequent in. But, matty Stratton, welcome to the show. How's it going?

Matty: 1:53

It's good it's good, Glad that it's. You know, nobody ever wants to talk about the weather, but in Chicago, when it gets a little bit warmer, that's all we want to talk about. So it's 50 degrees out, which means we should be running around in shorts and tank tops, I guess.

William: 2:05

Yeah, well, that makes everybody happy. So we're not too far from Chicago. Like what? Four hours and the snow is almost completely melted around my house, which is very exciting. We're just talking about this. I've slipped probably like four or five times this year. Usually I'm extremely careful and I never slip on ice or snow. This has not been my year, but glad to see the snow gone. Um, yeah, so the windy city, do you? Uh, I always ask us. I don't know why, but whenever, whenever I'm in Chicago, I always go to lose. I love that place. Do you? Do you like the Chicago style pizza?

Matty: 2:37

I have strong opinions about this. This is this is actually, you know, one of the things that, uh, you know, people like I don't want to say give me a hard time about, but so I don't mind the deep dish. But the like secret of Chicago is that we only eat deep dish when y'all come to visit. Like real Chicago pizza is a style called tavern style. It's the first time we've heard this on the podcast, I guarantee you said you had Tim Banks on the show. I bet he talked about this because he loves to give me a hard time about this.

Matty: 3:09

But yeah, tavern style is like it's a thin crust and it's a party cut, which means in squares. And the reason it's called tavern style is it used to be so at the bars and you know folks would come off their shift and they'd go to the bar right after. They'd get these little slices of pizza on a napkin. So it's sort of like pizza tapas or whatever, so you could drink more. So that's the tavern style and again, I'll do deep dish, like once a year we actually have a tradition with devops day, chicago, you know, when people come in and the sort of unofficial after after event speaker dinner is we'll go go to lose or you know, giordano's or wherever and and you get that. But also, I mean, you couldn't possibly eat that regularly anyway, you know you would even eating it once.

William: 3:49

Honestly, the last time I had, I think the last time I was in chicago, we did giordano's, yeah, and you, it's so good, and I hadn't eaten all day, so it was like I flew in. It was straight to the office, straight with customers and all this other stuff, and then it was like later that evening I was starving. We go to Giordano's and I overeat, it's so good, you just keep eating and then you almost like need a vacation because you just feel so it's just like you just oh yeah, oh, it's terrible.

William: 4:17

And then the next day you wake up and it's it's equally hard, so Hard, so yeah, so one of the I'm. You've been a huge, just advocate for DevOps for so long. You know in modern, you know modernizing enterprise IT practices. I think you co-authored a book about developing apps in the cloud and you co-host Arrested DevOps, which is an awesome podcast for those listening. If you have not tuned into Arrested DevOps, which is an awesome podcast for those listening, if you have not tuned into Arrested DevOps, it's great. So what else are you working on these days?

Matty: 4:52

Yeah, so I, like I said, I started, started ADO a little over 10 years ago. I always say we're the longest running DevOps podcast still running. I'm using that term a little bit loosely, I'm a little behind in publishing, but I also like to. We're always at the top of all the lists of DevOps podcasts and maybe that's because we start with an A, I don't know, I like to prefer it the other way. I also, you know, I started DevOps Day Chicago over 10 years ago. We actually have our 10th installment of that coming up in a few short weeks and I feel like, with that event, I love it so much but I, you know, I've been trying to step away from it because it's a lot and I feel like you know michael corleone and the godfather, where it's like every time you think you're out, you know so, um, but and heavily involved with dev ops days around the world and um, yeah, right now just sort of trying to figure out what my, my next professional thing is. But i've've been involved in lots of different types of work within that space, a lot of the places I've been.

Matty: 5:48

I always like to say, you know, I spent two decades working in technology operations and now they pay me to talk about it. You know. So a little over 10 years ago is when I moved over from the vendor side after how many years of working in technology operations and lots of insurance and finance, because Chicago and and when I was running technology operations at apartmentscom here in Chicago. It was interesting because I was towards the. That time is when I got involved in the DevOps movement early on, but a lot of the what we were doing was like it was like that it's DevOps but we didn't know what to call it. You know, know, because you know I owned infrastructure but we were so tightly coupled with app dev. But also, you know, I look back at the things that we've learned over the last 10 plus years and you think about how you would do work differently and that's sometimes interesting and them directly. As you know, a solution architect or a customer architect, like when I was at Chef or in developer relations, cause I was like when I was a pager duty.

Matty: 7:02

It was traveling the world and working with all these different companies, visiting and seeing and talking and hearing, and you, you see these stories and you see these things and on one hand you sit there and you're like we've come so far and this is so great, but then you sit and you sit. Sometimes you know it's a cognitive dissonance, right, you know. And then you're like, but we keep having to say the you know there's still lots that that has to be done. And I think we've looked at the evolution of you know kind of the starry-eyed early dreams. And then you know it hits the reality of vendors and marketers and you know, and I think about how my you know strong opinions have changed because of other, of collision with reality. Um, do all, do you all give a great example.

Matty: 7:44

So I I was always been a big proponent of you know, I say DevOps is not a tool, title or team. It's a way of doing work. I still believe that to be true, and so I would be very noisy about like oh, there should be no such title as DevOps engineer. That's not a thing. That's like saying you're an agile engineer. You know that's not a job title, and I've changed my stance on that. I still believe that DevOps is a way of doing work, but I don't publicly make a big deal out of like oh, it's a terrible title.

Matty: 8:11

And there were two things that happened that made me rethink that. So one was Ian Coldwater made this point once and they said you have to remember that the people who have the title DevOps engineer, that's not a self-bestowed title. Someone else gave them that title. So all you're doing is making them feel bad and it's not their choice, which I think is a very empathetic way to think about it. Ago, and he said the the pay compensation difference between cloud engineer or system administrator and devops engineer was about 30 percent increase. So I was like cool, call yourself whatever you want, go get your bag, you know. Call, you know. Call yourself the king of france if it's going to get you more money.

Matty: 8:56

So but I still do think that a lot of this is, while org structures are what they are, ultimately, a lot of this is just thinking about how we do work right, you're not going to solve this with and, and I think, in as a, as a community and as a movement. It feels like early, you know, we over rotated, or it appeared to over rotate. On culture, if we think about, you know, the acronym COLMS, which is kind of the prototype of thinking about DevOps, the culture, automation, lean measurement and sharing which I still think applies, and it seems like we really overemphasized or felt like it was all about the C, and part of the reason to me I always thought about this way is all of them are equally important, but that's the one you have to convince people about. You don't have to, like convince engineers to, to, to use automation tools Like they're going to want to do that, like we want to do. Those things come a little more naturally but, like the culture part is also one of the hardest parts.

Matty: 9:55

Similarly, early, early on in the movement, there was a very brief and thankfully brief movement attempted to have this idea of enterprise, devops, which was columns without the C, and there was a person who I won't name, who was a big believer in this, who used to say culture is for yogurt and I was saying, oh, in the enterprise it's all about getting this work done. And, honestly, who saved us from all of that was Gene Kim and DevOps Enterprise Summit because you had this came up and said this is enterprise and it's DevOps the way we talk about it and it kind of shut all that down. That was a really important time.

Eyvonne: 10:32

Well, and you know, I think one of the challenges and this happens in the trajectory with any ideology, right so you know, I was introduced to the Phoenix Project very early on, like 2013,. Right, the book hadn't been out a year yet. It was a shiny new idea. There was all this excitement around it that, oh, like, there's this new way to think about things. And then there was Nicole Forsgren's book coming out of Google and Accelerate and all of the things that we've learned there, out of Google and Accelerate and all of the things that we've learned there.

Eyvonne: 11:12

I think there's a reality to work, though, when these ideas and principles start really hitting the real world and get mainstreamed, that there's just the weight of work and people and processes that gets attached to it over time. That dilutes the idea, and I think that's part of where we're at in the evolution of the concepts is like it was important enough that it did make some changes and that people did make good faith efforts to implement agile development, devops principles, but that has come against things like corporate compliance, a hundred years of enterprise, you know culture, especially in organizations that were built in a manufacturing world, organizations that were built in a manufacturing world, right, and so there becomes all this weight associated with the concepts that aren't the core principles and and people start looking for okay, so what's the next thing? But because it doesn't seem so pure anymore, but really it's all kind of the same thing we just reinvented and attach new words to it because we're trying to get rid of the stuff yeah, yeah, it's I.

Matty: 12:31

I had an episode of of of ado. It was called. You know, platform is platform engineering, just devops with better marketing, right, you know everything old is new. Again, I think a couple of things that you said are really key. One thing that was a problem, if you will, is that you know there was no such. There was no DevOps manifesto. In fact, devops was very, very, not specific and that was by design, and the problem is people want that and so when there's a vacuum, that gets filled, right, so it gets filled with. Well, then it means this.

Matty: 13:00

And the other thing is, like you said, when you run into organizational things, you know, I don't remember the exact idiom, but it's like you will never convince someone, you'll never sell something to someone or convince them when it's contrary to basically how they make a living. And that's what got us safe right. Scaled Agile framework is how program managers get to still have a job and call it Agile, right, and I think you have to figure out. But the thing is, like you said, about as long as you're still getting to the core improvement, and that's the other thing we have to.

Matty: 13:32

When we look back, we can be very frustrated and say, oh, it's not what it was, but I remember, yeah, I remember what it was like.

Matty: 13:39

I did this work right and this is so much better. And I think there's context that we have to take into account and we have to look at the larger perspective. Two things come to mind when it comes to that. One was I remember on my first day at PagerDuty I was in the new hire onboarding and a lot of the people in my group were, we'll just say, newer in their career you know it was one of their first or second jobs or whatever and they're sort of in the onboarding, they're telling the story of PagerDuty and like what it first came out to be. And I remember all these people you know who were much younger and earlier in their career were like I don't understand how that was a problem to solve, like you mean, like getting the alert to the right person. I was like, oh, let me tell you stories about having to update exchange distribution list once a week to whoever was on call and I'm like it seems like such a simple little thing, but it was so big that's it.

Eyvonne: 14:28

That's a bless your heart moment. Right, you're just a little bit sweet summer child, you know like how fortunate for you that you didn't have to.

Matty: 14:36

That this seems like what's the problem, and the other part when you think about the context, when I was leaving my role at Chef and so I had been a pre-sales person and then I also switched over to on the technical side of success. So I work with these organizations at the beginning of their journey and then once they had started to adopt, and when I was leaving to go to my next role, you know I kind of had, you know, the last meetings I would have and I remember when my last week at Chef I had my quarterly meeting with the Chicago Mercantile Exchange, were one of my favorite customers here and I was with them for a long time. And you know one of the people on the team said you know, maddie, I feel like we've let you down. We have not gotten as much done as we should have in the last couple years. And I said hang on, let's take this back. I said you're in the middle of it and you know what you want. I said I have the luxury of stepping back and I'm like, let me remind you, let's go in the Wayback Machine 18 months ago. Look at how much you've done, and oftentimes we don't see that when we're so close to that, and even as an industry, we get frustrated because when our reach should exceed our grasp, but then that means if we're going to have our reach exceed our grasp, we can't beat ourselves up when our reach exceeds our grasp. Right, that's the trick of it and a lot of it. Like you said, that's the trick of it and a lot of it. Like you said, there's all sorts of reasons, but I still think the net is positive and we still have a bunch more work to do and things continue to evolve. The problems we were trying to solve 15 years ago were different than today's problems.

Matty: 16:09

Cloud changes things in so many ways. You think about Javon's paradox, right, like it's, the easier it is to do something, the more there is of it. And you think about the scale we talk about that you can do with cloud and I'm my old school like rack and stack, you know. I remember. You know if I wanted a server when I was at the bank, I mean that was, that was four months, you know, to get that done. And the idea like, oh, I mean even virtualization, like when we rolled in you know vm, where it was like you know what do you mean? It's just like press this button and now I have a server. You don't get things that fast, you know, and that comes with its other problems. Maybe we need to go back to some of those old back in tech days because maybe a little more thoughtful. I don't know, but you know well and as we look forward.

Eyvonne: 16:55

You know there are going to be new challenges with, you know, ai and inference, and there are new problems associated with that technology and new scale that we're going to have to solve. There's a whole arena of AI, ml, ops. That's virginal right. We've got all of these models all over the place that are doing all the things. We need them all to be orchestrated and working together and have kind of the same training and the same backend and we and but we also need them close to the users. How are we going to operationalize that whole process in in a world where there are new problems? So there are fresh horizons out there that the progress we've made in the last 10 to 15 years is going to provide the platform that we build on in the next decade, and that's a whole new set of operational problems that the technology is still maturing into. That I see coming.

William: 18:00

And speaking of hardware, there's an investment in hardware again to run this stuff locally in the data centers that you know. Some of these companies didn't get rid of all their data centers and put everything in cloud. Thank goodness, because there's a balance here and it turns out running some of this stuff against smaller subsets of data and figuring things out before maybe you put it in the cloud is so much cheaper that it is a no-brainer.

Matty: 18:25

Well, I was going to say, with the value you're saying, as we move to these next things, we build on the shoulders of giants, learn their lessons and the core, while the implementation detail will differ, but those core, I wouldn't even necessarily say first principles, but you know it all comes down to. These are socio-technical systems and what has not changed is the socio part of it. And you know, as I say, computers are easy, people are hard. Right, we will learn the new technology, but there's changes to the socio part of the socio-technical, because the, the tendrils of all the llms, like like we used to. You know, separation was easy before, to a certain point, we could argue about it, we could sit there and say maybe some things were too separated, but it was like this. This data was here for the most part. Yeah, you can distribute it, you can do whatever. But now you're sort of like as your, your llms and everything are kind of reaching into all of these things. It's like how do you have that traceability? How do you have that? And then what?

Matty: 19:26

I think the the important part is and this goes back to something you said early on, yvonne is that like when you sort of hit we can't just keep doing things the same way, because the world can see, our worlds continue to change. So what we need to do is say what can we take from how we've done it? How do we apply it? But you know, we've always done it that way. It doesn't work right. You know what got you there is not. What got you here is not going to get you there and I and it's hard because so many things that we've done, and especially when you talk about bigger organizations and people have things they've built and it's you have to. You know, I was anytime, I was working with anybody in transformation or any kind of a change. One of the first thing I say is say every decision you've made up till right now has been the exact right decision, right. None of this is because you want to. Oh well, we do it this way because that's how we did. It's like fine, cool things are different. Now we, we, we have to, we look in the rear view mirror, but we're going forward. So what does need to change? Because? And and how often do we have things that exist? I mean, this is we could do a whole episode of of war stories about weird processes that exist because of one wacky thing that happened once.

Matty: 20:39

There's a couple that just I always think about. I remember when I started at apartmentscom, so we had, you know, the backend database cluster and all the front end databases that serve the website, and you know we would do releases and distributions and I was like, why does it take an hour for data to replicate from the backend to the front end? And it was because it would hit each front end server with 10 minutes in between, and the reason was because one time someone pushed out a jacked up change and it took down all the front end data. So now, to be safe, we do this. And it was like, well, and again, I didn't know as much of that as I did now. Now I would know a little bit more how to frame it. But you're like, again, this goes back to what do we build in the system to keep that from happening? Because what we've done is we made every single release take way longer than it has to.

Matty: 21:25

Or there's a great story and and I, I, every year this becomes less and less likely to still be true. But in 2005, um mozilla, there was a change made to firefox and it broke mlbcom, and this change was pushed out on day one of the world series. So for years, as long as I knew, there was a smoke test in Firefox that was test MLBcom. You know now those are okay, cause those are just. They exist. No one. They're not part of your. They don't slow you down, they're not part of this.

Matty: 21:52

But how much do we have? You know, it's also the other side and they're supposed to be a part of the socio-technical. One of my friends used to say never be the reason a policy is created, but so many of the ways we work are built around things that may no longer be relevant, but they're in our operational DNA and we do need to pull them. It's really it is important to look back and to say why do we do this? But it's not from a oh well, that's wrong. It's just more like let's understand it, but none of this is oh, the way you've been working has been wrong and you're bad. It's just like with the information you had and the world the way it was at the time, that was the right decision.

Eyvonne: 22:39

Things change but let's adjust. I gave an internal talk just a week or so ago where we were unpacking these things. You know, let's understand an organization and where they come from. What was their mission statement? What are they in the world to do and accomplish? And what they're in the world to do and accomplish shapes not just their product but who they are as an organization and in order to impact change at that organization, you have to understand those things.

Eyvonne: 23:06

I feel like sometimes we have to become almost an organizational forensic archaeologist and unpack some of those things and say this is why this thing exists. Now let's talk about what's new and different. Now that gives us permission to change that thing. Right, because a lot of people don't understand that and they and and and people have different fear and risk thresholds, and some folks are like you know, burn it all down, I don't care, like we'll just go make it all new. There's value in that kind of mentality. But then there are also people that like these things exist for a reason and we might break something if we change it, and you, you need both of those mentalities to come together and solve the problem and determine what the way forward is, and and we also have to be patient with one another through that process right If things don't turn on a dime.

Matty: 24:04

I always people ask me and say what's the most important DevOps book I should read, and I say go read Freakonomics, understand incentive and you can do DevOps right. And it comes back to I think about, like you know, pagerduty, and I would teach these workshops on incident response and incident command and one of the things we would teach is, say, during an incident, one of the things is we'd say don't litigate severity, which means during the incident it's not. Oh, is it a really? It came in as a SEV1, you know well, maybe it's really a SEV2, blah, blah, blah. So you know, and I would constantly have folks who come on and say we do this all the time, this happens all the time, and they said I bet that you have an executive that measures success by how many SEV1 incidents exist, and they're like yep, and I'm like, so there you go, right, so that's the thing. So why? We say then the most important metric on a team then is not mean time to resolution, it's mean time to innocence. Right, how quickly can you prove that you didn't break it? It wasn't a sub one, and none of that serves restoring service or getting the stuff done and this goes all into.

Matty: 25:06

You know you talked about Dr Forsgren and you know she's a big believer and leverages the Western model so much. Right, and it's kind of funny because I, my, my fiance, works for a small business and she's having some not in tech and I think she's an operations manager for a heating and air place and she would, we actually. I I told her about the western model the other day because, talking about things it happens in a company of 20 people, I'm like pathological and she's like, yep. Yep, that's because I was like it was funny, she's describing certain things. Yep, that's because I was like it was funny, she's describing certain things. And I was like, oh, these are literally in the West room. So it's, it happens. And those are the hardest things. The technology is easy. We figure that out because it does what we say, it acts the way we expect it to mostly, and if it doesn't, our expectation was probably wrong. You know people don't, although you know, if you kind of understand how people work, it doesn't mean you can manipulate them, it doesn't mean that you'll always get it right, but if you know what's important to them.

Matty: 26:05

I have an old talk called the five love languages of DevOps and it's fundamentally about change. And I think it comes to mind with anything when you're trying to affect change, because the thing that resonates to you about why this thing is important is not going to be the same as the person you're trying to convince. And I had, you know, a fellow was on my podcast early on and he's not a tech person, he's a. He's actually a leader, a leadership coach, and he's the one I got all my management training from and I've told him I was talking to him the other you know recently and I said Billy got all my management training from. And I've told him I was talking to him the other you know recently and I said, billy, you don't understand how many large organizations have heard your name when I'm talking about DevOps, because I talk about. He talks about committed versus compliant people in a change, and that's a different conversation. But also he's like if you're trying to convince somebody of something and you find yourself saying, you know, william just doesn't get it, that's on you, that's not William, right? That's I'm not speaking your language.

Matty: 26:59

And you know, the thing that resonates to me is, maybe, as the system engineer is like, okay, if we have all this much better automation, this is easier, or whatever. Well, you're maybe on the finance side of it, so you're going to resonate more with the reliability. You know different. You have to. You have to understand the drivers of the people you're trying to make change with and it it feels like falling in deaf ears. And I've I've also, you know, on the.

Matty: 27:21

You know I've spent a lot of my career recently working in developer relations and I came to this conclusion over the last year or so, cause a lot of folks in Deverell be like oh, the business just doesn't understand what Deverell does, they don't understand why we're about. The business doesn't understand De rel, and I've come to the conclusion that it's the opposite problem. The problem is dev rel doesn't understand business. Right, you know, it's like and and this is true for, uh, you know, uh, it's funny because developer I'm not. I'm not, you know, slamming on developer relations people, that's me, but dev rels are the only place.

Matty: 27:51

And if you talk to anybody else in any other kind of job and said I don't feel like I should show the business value of what I do, they would say what? And the only other people who would say that are software engineers, by the way, you know guilty from you know the new King makers, like screw the whole thing up. Right, we got into this whole. I remember early on in Arrested DevOps we had someone on and she said the value of DevOps was that it protected the time of your most valuable people in your organization, your developers. And I was like I want to be like get off my show, right, you know like that's not what this is at all, but and yeah, like I'm kind of giving me so many one liners for shorts video shorts that I can make just out of that sort of blurb right there.

William: 28:39

So I'm loving this back and forth between you two, like that was I'm loving this. It's almost like I'm in the audience right now and I'm loving every second of it. I want to propose something and I want to hear you both talk about it now. So one thing that I've noticed in the past like it's really become real to me, like over the past two years, I used to think that organization or team size didn't matter so much. It wasn't that big of a deal.

William: 29:03

And something I realized when I went from large enterprise to startup is I've seen small teams scale huge things, but I've also seen the more hands you have in that cookie jar, for some reason or another, tends to slow things down.

William: 29:21

So, like when I worked in enterprise, there would be this thing where they would say okay, we're just going to hire an army of DevOps people and we're going to pair you up and we're going to hire a partner to come in, give it a steroid injection. You have 40 people at your disposal, we'll taper them off. And the more people that we threw into these big initiatives, the slower, the more complicated, like it would completely nerf the whole. It would be a waste of money, pretty much like why you're light your money on fire. But even in the startup space, like having like a very small agile team like the, the, the amazon, you know two pizza box thing, was like a really sweet spot if you did it right. And then as you start injecting more people and more opinions, you find that it starts becoming problematic and I don't really even know at this point how to think about this problem. Like, does team size matter? What does it mean?

Matty: 30:16

Well, the first problem with the two pizza team thing is that I'm a two pizza team. You know I can eat two pizzas by myself. So you know it's all relative. But I don't think there's like a magic number right. There's not this magic Dunbar like. At this size it's too big. But you're 100 percent right, because I mean, just even think about if you've got you know, know how hard is it to figure out where to go for dinner if you're deciding between four people or a dozen people. There's there's more opinions.

Matty: 30:43

I like to think about ways that you can leverage. You know, and a lot of it, the size of the team you can handle to be able to move quickly depends a lot on how that team works and what your organizational culture is, because if you have high levels of psychological safety, that team can be bigger. Now, that being said, the bigger the team, the harder psychological safety is, because you're mixing more ways of thinking, you know so, getting alignment, but you get too small, you get too aligned, and then you don't have any voices of dissent, you don't have any other pieces of that. I think you have to if you want to be able to make a larger team work better. You have to have a little more rigor that you can abandon when you come to how you do things. You can say we have a certain amount. You have to be able to say we will bike shut on this only so long, but eventually a decision has to be made. I like to think about voice but no vote. Sometimes in certain types of decisions where we're like I want to hear what everybody has to say, but ultimately this is decided by these two people. Because if it's complete, again the consensus protocol right, the bigger, the more people, the harder it is to get to consensus. And I think there's lessons you can learn and also how you rotate that, because what you don't want to do is have the same people always be the vote people and the same people are the voice people. But maybe you say okay for this one.

Matty: 32:05

At the end of the day, the buck stops with Yvonne and William. We want to hear from everybody. Okay, but the next time. Maybe it's not the next time, but you have different ways you could structure it. Okay, this is your place where you make the ultimate decision, but you can bring so.

Matty: 32:17

So the bigger the team, the harder it is to do it, but I think just to say, well, then we'll just do a really small team there's, there's. You lose something from that because now you're going to get way more homogenous. You know, because even if you don't enter that team that way, that will tend to happen over the course of you know, four or five people are going to tend to eventually is how we do this. Part of that so part of it is also maybe even just maybe, smaller teams that are more ephemeral as well, like if you. That's. The one thing about it too, is like you can have a bigger team on the thing, but you're like, for this problem, we have this squad that exists, it comes up to do this and then it's done, done and next time it's a different squad. So that way, if we have to make fast decisions, we can do that, but we're not always the same people making the decisions and we're able to distribute it and bring that in I love that.

William: 33:09

so you're basically saying that's one more thing I want, sorry, okay. So you're saying basically, like, instead of like decision by committee, like that you would see at a large company, you basically have the few people that are like, in charge of this one thing and, yeah, they're going to take feedback from all sorts of different sources, but whether they use that feedback, they're just using it to inform their decision that they get that final decision.

Matty: 33:30

The bug has to stop somewhere because otherwise you get into analysis paralysis. Same thing happens in incident response Analysis paralysis same thing happens in incident response. That's why you have to have you know, if you think about it in the proper incident command system, the incident commander, whoever that happens to be, in a proper incident command system it's a different person. All the time they outrank everybody on the call, even the CEO, because someone's got to be able to be like. This is the decision, because the right decision is the wrong decision is better than no decision Right Oftentimes is better than no decision right oftentimes so.

Matty: 33:59

But I think you have to distribute that, though similar, like innocent command can't always be the same people that are the decision makers even around the same thing, because people will eventually turn into then why am I even involved? Because I'll undo. All I am is an opinion. But if it's like, hey and again, I wouldn't necessarily say you'd structure it that it changes every week I'm using this as an analog. But it's well, it's not my turn this week, but I know I'll get my turn, and that also if you eventually and then also brings in that making the decisions is hard, what do you think about?

William: 34:25

it.

Eyvonne: 34:26

Well, there's God, there's so much. I mean because there's team size and I do think, like beyond a certain size for particular tasks, like it. Just teams don't work once you get past a certain size. If they're, if they're all working together on a discrete task, you know, and and in my mind that's somewhere around 10 or 15. At the same time you've got organization size, which is a completely different thing. Because here's the thing the bigger organization, the more discrete teams you need. And then you've got to figure out how are all those teams going to coordinate with one another and work together.

Eyvonne: 35:03

And when you start with a very small company, it's a non-problem, because you know, and I think what we know, is that a group can really only ever be as big as around 150 people, like that's how many people we can know well and keep track of In a startup. That's perfect. But once you start growing to thousands or tens of thousands of people, you need a certain degree of organizational scaffolding to make all that work. And the analogy that I use is like if you're building a single family home that can be framed up with wood, with lumber, but eventually, if you're building a multi-story apartment complex, you've got to have some steel underneath to keep that thing from collapsing in on itself. And then the question becomes how much scaffolding do you need? And then how much room do you need to give these teams the ability to bend and flex, to be able to work together?

Eyvonne: 36:04

And it's actually a very difficult problem because, depending on the organization, depending on its mission, depending on its culture, depending on its skill level, that's going to look different. But I think that the team problem is one problem. But then part of your question, william, was like organizational size, and that's a completely different thing, because at some point an organization has to scale beyond. Like we just hire smart people and let them do their thing, okay, but if you have five smart people all working on the same thing in different ways, then you're working against yourself, right? So it's a much bigger, more complex problem than a lot of times we want to give it credit, for it seems like it should be simple. It is not simple.

Matty: 36:55

To take an analogy, that's probably going to fall apart halfway through it. But it's kind of like, you know, when you're talking about, like in that start of that small team, when it's like, hey, we got this small group, that's a monolith and there's great things about a monolith, right, you can move fast with that because there's one thing or whatever. But then as you distribute out, so the large organization, that's your microservice. So if you're going to have a microservices architecture, there's a whole lot of different kinds of not just technical but just procedural infrastructure you have to have in place and you have to have well-defined APIs, right where that comes in. And I've really pushed that metaphor in maybe terrible ways before. But you know, teaching my team in my last role would say you have to learn the marketing API, you have to learn the sales API, like how do you communicate? And I have a talk I gave. I just forgot the name of it, I'll get to it in a second.

Matty: 37:49

Oh, it said zero trust is for networks, not for teams. But it's kind of this idea of that. If you think about a service-oriented architecture, it's like what happens inside the team is the black box. But you can only do that when the same thing with a server, like nobody cares how that service executes as long as its contract is fulfilled. So if you say this is how we work with the larger part of that, and we agree that nobody cares how you do it inside there, so to speak, you know it's like that's fine because this thing goes in, I get this result, that's the thing that happens and that's so.

Matty: 38:19

But just like we said, you know the advantage of a monolith is you got one repo, you got one thing. You know it's easy, it's fine, you know. But you know. So you can't just suddenly say we're going from a monolith to microservices but we're not going to build all the right things in place to let these services we're not going to have a service mesh, we're not going to build all the right things in place to let these services.

Eyvonne: 38:39

We're not going to have a service mesh. We're not going to have a contract, yeah, yeah.

William: 38:43

That's a great analogy. That didn't fall apart at all. It's perfect, great thought exercise. So where do you so? I know one thing in listening to a rest of DevOps here the past few months, on and off whenever I can there's so much good content out there. It's crazy. You know, platform engineering is one of those things that sort of crept up. It's almost like the new and the new and shiny thing. And I remember in enterprise, like you mentioned safe earlier, which made me cringe because I worked for a large Fortune 50 that went all in with SAFe Straight from Waterfall. We didn't even know what was happening at the time and they came in and the next thing we knew we had scrum masters, we had program managers we had project managers, we had product owners, we had more people doing process than actually doing technical work.

Matty: 39:35

SAFe is all the ritual and none of the result right.

William: 39:39

It was so hard. So do you think platform engineering is like a new? Is it really a new idea at this point?

Matty: 39:46

No, and I don't think so. I don't mean that in any kind of a dismissive way. That's actually the good thing. It's where we're in a position that we can. It's interesting when you see things and you're like we've actually wanted to do this for so long and we were not capable because the technology was different, the places were different. I think you know it's what is it?

Matty: 40:08

James Governor said you know, everyone's just out there trying to replicate Heroku. You know, and if you think about it, this was the idea, right, you think about. You know I've been talking about for years, about treating your platform, treating your infrastructure as a product internally, and that's real I almost said real platform engineering here. Now I'm going to be this purist, but if you kind of look at platform engineering from that perspective and the places that do it successfully, that's exactly how that works. Right, it is. This platform that we build is an internal product and we have to treat it like a product. You know, in terms of we have customers that have to use it, so there has to be. So DX matters, right, you know, giving the features that that are. You know whether your customers are one that's directly or indirectly giving you money, and I think that's something that has been sorely needed on the infrastructure side and we haven't really been in a good way to do that. I think there's also, just like you know, well, I'm going to DevOps. I'm just going to, you know, throw some Terraform at stuff and now I'm DevOps. Right, there's a lot of you know, platform engineering isn't just like go install Backstage and now you're a platform engineer. Right, it is a modality, it's a way of of working and I think it is an evolution of some of the things we talked about in devops. I think they go, I think they're very closely aligned.

Matty: 41:26

I think having, if you, if you espouse platform engineering principles, devops as a principle and a way of working, it's easier, right. And also, I think where it came out of was a little bit like yvonne's point early, very early on, which said okay, you can have these ideas and they work. And then you start to hit certain types of scale and you know michael deuce used to say, you know, I mean, silos are good, right, like silos like keep things from blowing up a lot of times on farms. Right, like there are reasons we have these things, but for them to work well, they have to be friendly silos and the models of platform engineering allow for that. This says we can have because the scale is so big that it can't just be everybody does everything. We just sort of talked about this in team side, but you have the way that it's like serving the right thing versus you know.

Matty: 42:12

So it's not kind of the the wall of confusion stuff that andrew schaefer would talk about when you're talking about interacting with a platform team, because you're interacting with it like a product, not just like here you go, you know, and you can, you can kind of take some of those or socio part of the socio technical and kind of codify it and productize it a little bit. I think that's where I think. But just like every other time, we've tried to make things better in this industry, people. You know I again I'm really I'm giving you all the things, but one of my, my favorite little off the cuff ones, is you can't buy DevOps but I sure can sell it to you, you know Well and one of the one of the unfortunate side effects of the DevOps movement was that there was this mentality that grew out of it.

Eyvonne: 42:56

was that there was this mentality that grew out of it that, like your developers could do the ops and they could do all the things right, and so it was this bastard it was the dumbest word.

Matty: 43:08

It should not have been called that. You know, and Shea will tell you that.

Eyvonne: 43:12

It was a bastardization of that ideology, right? And so I think what platform engineering is doing is coming back and saying, okay, the platform is a thing that developers need and we need people who specialize in maintaining that platform and delivering it right. Because what we whitewashed away as we started talking about DevOps is that very important job of the operations people and the folks who maintain all the things that everybody needs to work. Because what the bean counters heard was oh, it's the full stack engineer, we just need fewer, better people, right, and it's. It's kind of like that's, that's not how any of this works, right?

Matty: 44:04

So I it's like ops is a skill right you can't just and it's funny because there's, you know, this happens and this is a lot of this is human and organizationally human. You know, I remember I had a friend that I worked with many, many years ago when I was a webmaster. That'll date the conversation in the first place, but it was for a computer catalog company and you know so I worked very closely with the person who did design the graphic designer, who did the catalog, and she would say, you know, like people come and be like hey, tammy, can you teach me Photoshop this afternoon? And it's like this diminishing of what you do into like well, it must be easy, I can just do it. The only reason I don't do it is I don't want to. And that was a lot of that to your point of a lot of that was to your point. A lot of that was okay, well, if we just teach the devs and then they can do what that was we're like.

Matty: 44:59

No, we're not saying that. We're not saying everybody, it's just one persona. What we're saying is break down the walls, like devs should be thinking about how this will be operationalized, but not, they do it. And same thing and we. The same damn thing happened with shift left, right, like the idea of shifting security left was not get rid of the security people and make all the security be done. That we're just saying move some of the thinking about that earlier in the process, right, but you still have to have. You can still have software engineers thinking about testing and thinking about security and stuff, but that doesn't mean you don't have security people. You know it doesn't. But it means that you're, you're, we're just sort of making it more holistic and that's harder to sell, cause, like you said, cause same thing is how many things have been misinterpreted as this will save you money. And if that's your cloud, if your thought was cloud was going to save you money, you're doing it wrong. All cloud did was move CapEx to OpEx. Right, there's no magic thing. Automation does not. Automation is not a headcount reduction exercise. Automation is a reprioritizing of effort exercise. It's getting more value out of the people that you have. But it's not like, oh cool, we'll bring in Puppet and now we can fire all the sysadmins. You're like no platform engineering. I agree with you 100% that that's a way of saying, okay, we can have it in this way, but you still have this domain expertise. That has to reason about that and it doesn't work. It doesn't scale one-to-one, and I think that's why you need to have that, because you are always going to have more developers than ops for lots of reasons.

Matty: 46:49

And so I remember at apartmentscom when we, when we took on agile, it was the idea was okay, so now we moved into these different agile teams and we said, okay, we want to get sysadmin like integrated. So we said, okay, cool, so there would be on every feature team. You know the team would made up oh, there were the testers, and there was the scrum master and all the developers and like, and then there was the sysadmin. There were like seven feature teams. I had three sysadmins, which meant everybody was on multiple teams. And we actually hit a point with the rituals where my team came to me and they said do the math, let's do the math.

Matty: 47:21

I literally have no time to actually do any work because my entire day is taken up by Agile ritual, because I go to three standups a day, three sprint reviews every time, you know, and all of this stuff, because you're never going to. It doesn't scale organizationally. So you kind of build those and that's what I'm saying where everything old is new again, because that's how you would approach it. From infrastructure. You said, okay, well, there was infrastructure as a service different kind of as a service than cloud type thing. We said you consume it because we have to be able to scale behind it. But that created a really big wall and I think at least with PlatEng we are better equipped to be able to treat it in the way that we can use a lot of those great things. But I'm going to remember what you said about it being a pushback to that. I think that's spot on, spot on.

William: 48:11

This was. I think we need to start wrapping it up at some point. This has been an amazing conversation. It makes me so we were supposed to have, uh, tim banks on at some point in the future. We should have both of you on and do like a future of platform engineering or the future of devops, one of those sort of round table things. I think having both of your personalities in here at the same time would be a riot it might be a multi-episode conversation.

Matty: 48:34

We might have to chunk it up into series yeah, I was thinking, william, when you were talking about the one-liners for the shorts. Um, we, uh, I started early on in arrested devops just sort of randomly doing cold opens where I would, you know, just pull a, a quote kind of out of context and throw it before the opening music. And then we, uh, we had, uh, my podcast co-host, her husband, was doing our audio editing for me and, my favorite thing, when I would get the, the edit back from him, I'm like, ooh, I wonder what, what, what Joe is going to pull out for the cold open. You know, and, and it was uh, it was always. And then sometimes you sit there and then, as you're doing the episode, you're like, all right, oh, yep, I just said it, that's the one.

William: 49:15

So you got to keep things entertaining, got to keep it fresh. Got to, you know, throw some curveballs in there, right yeah.

Eyvonne: 49:21

We're all about the one-liners and platitudes.

William: 49:24

And I wasn't going to ask this before I just got to. I have to know. So I was in, I was presenting it. Um, it was, uh, it was. Aws community day, like two years ago, is in Chicago, and I'm pretty sure I met you there, tim says there as well, valtieri runs that she did DevOps days Chicago with me many years.

Matty: 49:43

Tim Tim banks was actually at that one too. Yeah, um, which I didn't realize, and that was part of what. Yeah, and it was one of the. I think when I came it was a very like last minute, like, oh, I don't have anything going on today, let me come in and hang out. But I think you're right. Yeah, that's crazy. When you said I would be like it wasn't two years ago, it was like no, it probably was three years ago it was in the morning star building.

William: 50:01

It was that month okay, great.

Matty: 50:04

Yes, hey, I know that, I remember that part.

William: 50:07

Yeah, it was. It was very nice. Okay, that's good. And you so. One thing too you motivate me. So I've been on like sort of trend, kind of like in between X and blue sky. At this point, and you're no shame, you're posting those leg day updates, and leg days are like the worst days for me. I've hated them forever, since I was in high school, but I'm starting to love them as I get older because I've had these. So I play competitive ice hockey that's my one hobby and I've had these hip issues over the years and like doing legs the right way and starting to build up that strength with hip core glutes and all that has been a life changer for me because I sit so much. That's a problem.

Matty: 50:48

Very, not to go into the whole thing, but it was like what, what turned it around with me as I started like I was sort of messing around with programming when I was starting the lift. I did, I did like, oh, I'll do a PP, I'll do whatever, I'll do, like day. And then I was. I did a couple of months where I did the five by five, and five by five is every day, is is squats and so, and that was what drove, that's really what would push my squats way up as I was doing them three days, you know, and I don't do that anymore but like, yeah, it's, uh, it's, it's a good three days. And I can tell the difference for myself, just how I'm getting older and that's a big part of doing this is you get around differently. You feel more comfortable doing certain things, you know, and, um, it's, yeah, it's, yeah, it's uh, they're, they're not fun, but but then they become a little fun, because then you're like, okay, you know and when you see the gains and you see the benefits.

William: 51:36

that is when it's like real and you're like you will suffer through it. Wake up early, do whatever you have to do, but like how? How often do you do legs out of curiosity?

Matty: 51:44

Well, I mean, so I do. Um. It's funny too, cause I was just about to change my program and I'm like no, I'm going to stick with it. So I do a four day. So I'm probably doing, I'm doing something with legs every day, so it's either my main lift or my second lift. So it's either so like on one day, like for example today, I'll do my, my heavy lift will be squats and my secondary lift will be bench press, so like more reps but less weight, and then tomorrow it's going to be overhead press is my main lift and then deadlift for volume, and then on thursday it will be bench for volume, and then I do bulgarians, because they're terrible. You know I'm gonna hate myself, but there's still a squat, and then on, and then on friday I'll do deadlift heavy and overhead lighter. So so there's something posterior chain every day, um, and and that also makes it because the same thing if you have quote-unquote leg day, then you can hate leg day, but if every day is leg day somehow but I also like doing it that way, because there's other variants I've done where, instead of spilling them up, it would be like like the you know, um, five, three, one. It could be like, okay, you do heavy squats for, and then you do, you do lighter ones and do that all in the same day. I'm like I like to mix them up because then it's not like then you'd also don't have that day. You just like you do something different.

Matty: 53:01

And you know, I told my son, my, my one of my 15 year old son. He started lifting with me for the beginning of the school year. Now he's doing it for baseball and he would talk about and he sees all this stuff on YouTube and he's like all this stuff, and I said Joey, I said we are novice lifters. I said again, I've been doing this for a year and I'm still a massive beginner with a lot of stuff. You know, since I was doing it, however, long ago and to be like so far, when it's like how do I optimize and make sure I'm doing exactly the right thing? You know it's uh, it's just like fundamentally, the best way to get gains is just like do something with intentionality all the time or show up and make yourself sweat right, you know, discipline over motivation so um love that, but it's, it's, it's fun stuff so awesome, motivation so um love that, but it's, it's, it's fun stuff, so Awesome.

William: 53:49

Well, it's been great having you, having you on the show. We've got to do this again with Tim. Yeah, it would be awesome. Uh, where can? So? Where can folks find you? You've got to ask.

Matty: 53:57

Um, yeah, you know that's a really good question. Where can you find? I mean it's, this is the bizarre thing. You can find me on LinkedIn is a thing I never really would have thought, but I was talking to someone recently about how all the tech content is over on LinkedIn. You can find me at BlueSky, on maddiewtf on BlueSky. Just look up Mattie Stratton on LinkedIn. Those are probably the main things. Follow us, DevOps. We'll get some more episodes out of there. I'm excited Just having this conversation. I'm like all right, you got a podcast again, although it's way more fun. Maybe we'll get you guys to come on my show, because being a guest on a podcast is even more fun, because you just show up and talk. You know I know what people listening don't understand is you know William especially. You know you guys are sitting back there going, ok, what's the time and and OK, got to wrap up this part or whatever. Then you come on and you just run your mouth. So we'll do a crossover episode. It'll be great.