Choosing The Right Quality Approaches & Boosting Engineering Efficiency (with Bernardo Guerreiro from Auth0)

Read The Transcript:

In digital reality, evolution over revolution prevails. The QA approaches, and techniques that worked yesterday will fail you tomorrow, so free your mind. The automation cyborg has been sent back in time. TED speaker Jonathon Wright's mission is to help you save the future from bad software.

Jonathon Wright This podcast is brought to you by Eggplant. Eggplant helps businesses to test, monitor, and analyze their end to end customer experience and continuously improve their business outcomes. 

Hey, welcome to the show. Today, I have a very special guest. Also, a recent article writer for theqalead.com. You have to check out his quality engineering tips for remote teams. So but not how's it going today and how is the lockdown in London? 

Bernardo Guerreiro Hi. So, yeah, everything's OK. Seems like it's starting to get a little bit lifted. Of course, for me, because I was already working remotely. It wasn't as much of a change as I imagined. To some people. 

Jonathon Wright Yeah, I noticed that you referred to from your you'll be working at the moment of Auth0. You know, you've got this great article about kind of seven years of tips. and I guess you read in the lucky position where you've kind of been used to remote working. 

Bernardo Guerreiro Yeah, I think I chose my time right. Because I joined Auth0 about six months ago. I've worked remotely before. But this is the first time that I work for a company that was full, kind of full remodeler globally distributed where, you know, we have I think it's in engineering, it's like 90 percent is remote. So that kind of culture remote-first really plays a big role in how your day to day goes. 

Jonathon Wright Absolutely. Yeah, I noticed that with this week that we were in my sixty-ninth day in lockdown because, you know, companies like Twitter and some of the big names in Silicon Valley have said that actually, they don't mind staff working remotely, indefinitely. So the idea is that maybe this is the new norm. So I know you've got a great article with some tips around remote working for those listeners that are not vetted yet. Could be kind of a highlight of what you cover today in the book. 

Bernardo Guerreiro Yeah, I think if John Miles starts with what I think it's probably like beyond all the very specific tips I gave to this kind of role, I think one of the things that you have to get used to from the beginning is the concept of asynchronous communication. You have to go away from that mindset that you might have in your office or where you can just, you know, call someone more or, you know, look inside to your desk mate and ask a quick question. You have to realize that people are working essentially in different time zones, but certainly with very different schedules, especially since a lot of people are working at home with kids in right now. So, you know, you have to get used to the fact that you won't directly get back the response you might need. So I think that's like a bit of a shift that you have to do in your mindset. And I think it's one of the last tips I give in the article. But probably one of the most useful ones is you should kind of learn how to make use of that time where you might be blocked or waiting for someone, because if you use it, if if you don't use it efficiently, you might end up feeling like you're actually being less productive. And it's the fault of the remote situation when really it's more about getting the right team tools and mindset and. 

Jonathon Wright Yeah. Something I've always kind of personally struggled with. Never. I've worked for a lot of investment banks and they kind of cover the globe and you kind of face this handoff between, you know, Singapore or New York and, you know, the time zone, eight hours. It kind of splits you. 

You've got you to know, you sometimes have that challenge where, you know, you're waiting for some other team in Moscow or somewhere else to come online before, you know, an action can then get resolved, which, as you said, he's a blocker. You know, it's just this really changes how software G Damian is done because, you know, there was this kind of, you know, you taking a task if you're doing Kanbanize something along with the board and, you know, maybe you've got to have other activities you can work on in parallel. 

Bernardo Guerreiro Yeah, I think so. You know, the maturity of where your software engineering is at is really selling, in my opinion, of how well you're going to adapt to remote working, because if the culture around your company still has a lot of like micromanagement and a lot of deliverables that get shipped between teams as opposed to a team kind of honing that process, then you might struggle a bit. The other example is just aware I think the culture is important is your team might not have both the right tools as well as the right mindset in place already for the remote-first culture. And an example of that is like I mentioned, I used to work in another company. I think it's almost three years back now. And I was actually working from Portugal at the time and they were in London based, but I was one of the few people in the company to do that. And the only one in my team to be remote and different. I see in how our calls go now where everyone is remote and, you know, the company as a whole has to solve that problem versus back then where, you know, only I was in that situation. It changes things a lot with, you know, things as small as the audio quality or the video quality or the kind of collaborative tools that you're using which might not be as well adapted if you don't have that remote first mindset. So, I think where companies now need to take a step back is first invest in those in those remote-first approaches. But also it might require a little bit of redesigning how you ship software, how you move those deliverables across different teams, or how even the teams are structured in some cases.

Jonathon Wright Yeah, I find it fascinating. I was on a really interesting call with some of the M.I.T. guys and I talking about gamification and, you know, it was interesting because, you know, halfway through, one of the speakers who were involved in the call was kept on breaking up, that the quality was Zoom really difficult stuff. They were only in Italy, but they were still having real problems with bandwidth. And actually, it became discrimination in a kind of a poor quality connection, discrimination, because they kind of said, well, actually, we're going to have to carry on without them in the discussion. And, you know, that's you know, that's quite hard for some people. 

And I guess, you know, it's watching a great keynote from Gartner yesterday who they're kind of saying what the new norms are. And they were kind of saying, you know, the security and home office, you know, you've got to deal with things like I know you've got the kind of performance experience, so we'll get onto that. But, you know, kind of things like people sharing the bandwidth that, you know, your home. So people watch Disney plus and, you know, you start eating into that connectivity stuff. So, you know, that's a real different challenge with, you know, people who've got families and maybe their home infrastructure is just not up to the job. 

Bernardo Guerreiro Yeah. And I think, like going back to your point about it being a little discriminatory, I remember in my previous company, not the one I was working remotely, but just one. Whereas here in London we had another team that wasn't mine who had a lot of people who were remote in that particular team, and that I really liked their approach because they used to organize. They were and platform or SRE team. So they used to organize this monthly meeting where they told some news and got some feedback from the back and engineering community as a whole. Just part of and that meeting used to be in person because they started getting so many people who were remote and they started to feel that it was discriminatory. They actually moved even though a lot of them were in the office. They actually moved that meeting to be fully remote so that everyone could be on the same experience level. I think that's an example of like a team or even a company adapting to that and kind of avoiding some of those pitfalls. 

Jonathon Wright I think to know these challenges have come for quite a few years now. You know, I remember working for a company, a commodity platform that was there. They insisted on having virtual conferences all the time. So you'd have it just your money, you for screens. So even if you, you know, VPN in one of your monitors would have all the rest of your teammates. And, you know, they used to have a lot of fun with it. You know, people used to put funny comments on people's screens, but it was this kind of, you know, even though you were literally next to each other in there in the office. It was always on. You could dip in and out and pull the right people in. But again, that requires the kind of the right equipment. And also, it's that kind of mentality as well. And, you know, I think there's a lot of companies that are struggling around with, you know, trying to understand policies and what works well for them, for their staff. And, you know, but some of us who are so used to remote working and like yourself, who's got this, you know, this sex, all these experiences, because these are the pitfalls that, you know, people quite easily will just kind of go, oh, well, you know, I have no idea how to work with a team in Lisbon when we're in the UK And I worked for a large piece of the company who all their resources were in Lisbon because there's a massive .NET community out there. And, you know, it's a perfect the center of excellence. And they used to, you know, bring in special equipment to have, you know, screens with a team connected all the time. And, you know, it works so well. But obviously they have to get it working right. And that took a bit of time. And most people don't have that learning curve to kind of get those lessons learned. 

Bernardo Guerreiro Yeah, yeah. I guess it's a little bit of a Darwinism model right here. You have to adapt to live on to. This is kind of the new reality, at least for the time being. And might presumably even change how we do a lot of this in the future, even if things return back to normal. 

Jonathon Wright And it's interesting because I remember sort of go round some great engineering companies, and one of my favorite things I ever saw was a little traffic light system Next to people's. And they could set the traffic light to say, you know, red, I'm busy. Don't disturb me. You know, green, I'm okay. It was a great idea. Was it? But, you know, part of it is now you're on your Slack. You're on your teams, you know. Yes. You may have set, you know, a status away or something like that. But, you know, since embassies message you well, you know, we don't really have this opportunity. Can I say prioritize my office, communicate your stuff, this kind of thing where it says, you know, if an important priority message came through, interrupt me, but otherwise, you know, automatically send a response, say, look, I'm in the middle of a meeting, you know? You know, can I get back to you? He is my current schedule. You know, it will be finished in 40 minutes, you know. Do you see that kind of coming through that maybe there's a virtual version of yourself that is bouncing off the standard questions that you can answer and you sleep and then, you know, it connects you what it really needs you? 

Bernardo Guerreiro Yeah. I think you touched upon an important point, a couple of them, actually. So one is before I mentioned how you have to change your mindsets, but that's both true as the person asking for information as well as the person that is potentially delivering that information. So, for example, like myself, I work in a team that actually integrates with a lot of different teams of kind of like quality advocates. We build infrastructure for a lot of teams. So I might need to interface with a lot of these teams because the company has that remote mindset. We all have the mutual understanding that both me as a person than might be asking something from one of those teams and then asking us something. There has to be that concept of asynchronous communication. You know, people understand if you don't respond directly. And I think that brings me to my second point that I think is important. 

You mention a personal calendar. I think it's one of the things about remote working is you have to be a little bit more verbose and transparent about what you're doing. It doesn't mean that you need to have your schedule fully online. But those blocks of time that you really are busy with the team, they should be publicly accessible to other people, too, so they can easily see whether they can contact you or not. And also, you know that there are formal systems, at least in our company, of asking if you have a formal request for a specific team. We've got engineering service desk tickets, for example. If it's something that takes them more than a little bit, it's something that they can take their own time to go and pick it up. 

I've never dealt with anyone in our company who I felt wasn't as. As soon as they could help that they didn't want to so that they were purposely avoiding. And similarly, I've never come to this situation where alone ignoring someone. But I have had to delegate my time and say, OK, I'm not going to answer this right now because it's going to take me a little bit of time. And I would prefer to give them a better answer when I have full time for it, as opposed to like, how fast. And so. 

Jonathon Wright I think you're raising a really interesting problem here, which, you know, I don't think anyone's really addressed, is that this is kind of lead time to the correct answer and doing the right thing, giving them the right information. And there's this, I guess, knee jerk reaction to respond back because people have been waiting. And I don't love the solution integrators for many years now. I've had this kind of email at the bottom of the email. They say something like, you know, we encourage flexible working. So, you know, we may send you emails out of .NET normal working hours. We do not expect to put any pressure that you respond to, you know. And I think, you know, that's the kind of thing because I get it all the time, you know, I guess it's this anxiety of, you know, some pings you on Slack. You know, you have to feel like you have to give them what they need. Otherwise, you know, they're blocked. And, you know, I guess that's the thing. If you could have that push back to say, look, I don't have the answers for you right now. Right. You know, I'm going to go off. I'm going to investigate. And I'm going to keep you kind of up today with, you know, how long I estimate it's going to take me to work on that. And, you know, that was one of the problems we were always bad at engineering was an estimation. Right. How how long is it going to take to do that? You know, do you find that you know, that's a problem when, you know, like you said, you're working between lots of different, you know, third parties, you know, how do you do those handshakes? Work is down to an account management program, management, or even, you know, product management. What's the key behind it? 

So to clarify, even though we do integrate with third parties, I mean, my particular role, they are third parties, but within the company. So there are third parties in the sense that there are outside teams from myself. I'm not directly in their software development lifecycle, but we are like one big team rowing in the same direction. So from that sense, there, there is that mutual understanding. To answer your question more specifically, I think one of the things that help, as I mentioned, is the service desk. But the ability to informally say I need the help of this team and the team part is really important there because you gave you were mentioning before, OK, like someone messages you and you might feel like you're blocking them if you don't respond quickly. But maybe that person could have messaged the entire team enough if you are as a company fighting for not segregating information. This is why I said it's really important to actually be a little bit more verbose and have a lot of information online and also to not have just too many subject matter experts. 

Bernardo Guerreiro You need them, but you don't want them to control all the knowledge. You want to share that knowledge. So that one single person isn't the cause of blocking someone between daps the service tickets or equivalent as well as, you know, maybe some formal arrangements that we're starting to build that some of the teams that we're going to be interfacing more with where you have some responsible people with within our team as well as within each of their teams to touchpoint between those things. I think it's it's quite manageable, but I'm not going to lie. It's a challenge, especially if you're not coming from a situation where you've had to deal with these problems. But it's solvable, in my opinion, solvable.

Jonathon Wright Absolutely. And I guess it looks at new metrics and measurements as well. So, you know, people talk about cycle time. And meantime, to recovery, you know, maybe it is this meantime to resolution. But, you know, you're tracking the potential, you know, where you've got something like you said, the service BASH, you've maybe got some kind of first-line chat Ops kind of capability to kind of understand how to categorize it. Then you're using up to BC the ability for, you know, some standard kind of service management, KC kind of capabilities. And then it goes into some kind of team pool where people can actually, you know, all jump in and track this all the way through to its resolution. And, you know, maybe those metrics will be important in the future. 

Bernardo Guerreiro Yeah. And, you know, just make you know, especially if you have a process like a service that's made sure it's a first-class citizen within your project. We as a team also have our own service desk. And one of the things we've done recently is to make sure that as part of our daily stand up, the first thing we do is actually look at the engineering service desk board, which we've kind of integrated within our project, and make sure that there is nothing new that we might have missed, as well as talk about any anything there that was going to pick it up or is it done yet? Kind of thing. It's a first-class citizen at the moment. In our team, and I know another team is so. 

Jonathon Wright And, you know, I find that really interesting. You know, one of my pet peeves, we've always been first and second line support. You know, how does it get that kind of prioritization within the organization? And, you know, see the value of just the importance of the relationship of their customers and how quickly they can do that, but also the prioritization. So even if they're using something like service now is how that feeds back into the SDLC, the resourcing capability you've got, which is obviously finite from it, from a development and engineering perspective. And, you know, I remember a recent role that I was where they used to have every day. They used to have someone they used to classes, Batman and Batman, and was responsible for saving the day. And, you know, it was a great idea. But at the same time, you know, it kind of felt like it was the disconnect for kind of DevOps was all about, you know, the idea that there was you know, it wasn't just, you know, throw over the fence and that its operations problem now and it was more. Now you're delegating someone to take on the first line, second line SAPOL on a rota basis. And, you know, they have to solve the problems and the tickets. And, you know, maybe that's still not really what we envisioned when we start to talk about kind of DevOps to kind of know OPs kind of approach. 

Bernardo Guerreiro Yeah, I definitely agree with you and. I mean, part of it is a cultural change to make sure that it's really integrated into our software delivery lifecycle. On one of the things that Ashley tried and our team was so we had a service desk, but you kind of had to go to a different place and Djura for a different board and project. So, you know, using the Jitter filters and such, I made sure that we also had a board that pulled in from that filter within our project. So really, you know, it's very visible. And like I said, it's the first thing we do in up that it must be something that we tackle. And it's definitely not something that a specific person will tackle. We will tackle it as a team. Of course, there might be some people that are more catered to solving a certain problem because maybe they have a little bit more information about that, or they were the cause of it or they were already working with those people. But you know, we work as a team. And like you said, I think that's kind of one of the things with DevOps, you don't want to be throwing things over the fence.

Jonathon Wright Absolutely. And I want to take a dig a bit deeper. I know you'll go. You're incredibly technical. And, you know, it'd be great to kind of get an idea of, you know, some of the challenges you're doing. You know, some of the kind of testing you've been involved with. And, you know, just generally, you know, how you get you to know, how you do quality engineering. 

Bernardo Guerreiro Sure. So obviously, that's a very abrupt topic. I'll try and actually answer that from the perspective of both my recent company. So my current one on Syria in the previous one result they are kind of different. They're similar in the sense that they're both exceptionally well-doing startups that are, you know, doing extremely well in their market and have had very fast growth. And they're basically like here to stay. 

But they're engineering culture was slightly different. And even some of the maturity of some of the engineering practices are a bit different out of, you know, some of the pain points of growth of the companies and then sometimes the field that it is. And I find that that distinction is important because each when you go into a new company and you come with a tool belt to things that you've done before and potentially things that you'll learn, they're. 

When you go and you tried to apply that, I think you have to be careful not to try and apply everything, you know, because it won't necessarily fit the maturity or the stage or the even just the tech that that company is using. And the reason I'm saying this is, for example, one of the things I've been very involved in at the Zone doesn't quite fit, Auth0, which is social contract testing. 

Jonathon Wright You mentioned contract-type testing. What is contract testing? 

Bernardo Guerreiro So contract testing. So deep. So when you have an application or a set of applications, generally, you will have maybe a front-end and maybe a back end. And in the most crews or even more, you most use the example of what a contract is in this situation is the integration between that front end and the back end, which basically means, you know, the front end is expecting the back end to return some shape of data that then the front end needs to be able to deal with. So this is in its most simple form. What is a contract now as well as engineering has changed over time? And we now have things like microservices, architectures for the backend. We've had some interesting challenges when it comes to testing, because whilst before you might have had a single back in Monolith, now you might have that monolith, which is like a huge piece of code with a lot of logic, might have been down into separate, separate, separate surfaces. 

Bernardo Guerreiro So instead of a single service or a couple of services, you suddenly might have dozens or hundreds at services and you want to test the integration between the service. So it's Service A talks to Service B, for example, A Service A is a user service that talks to a billing service. The billing the user service needs to understand the shape of the data that comes from the billing service. 

So if it comes with a field that says a currency that the user might be using in their account, it's important that you test that integration between not a challenge there when you're dealing with dozens or hundreds of these services. Is that in order to test all these integrations, the problem doesn't really grow linearly? And also, suddenly you might have two walls. Instead of spinning up just one single service, one monolith, you might have to be spending up a lot of services, which you can kind of orchestrate with Docker, for example. But it's still a hard problem to solve. That's one part of it. It's just how hard it is to do this. 

And the other part of the problem is that when you are testing the integration between the services, you're making some assumptions. You're making some assumptions about how the other service is going to respond. So as I'm testing the integration on service, say, I'm kind of assuming that the shape of the data on the billing service, for example, is still the one I thought it was at the time. But in this microservices architecture, the whole point is that these services can move and evolve very fast. So over time, it's not guaranteed that your contract. That integration that you're trying to test is still valid. Contract testing aims to solve these problems when it kind of does it in like almost a synchronous integration testing where you don't need to spin up your whole environment. 

You're going to make some assumptions still. But you're going to verify those assumptions over time with exactly the services that you're consuming. So I know it's a long answer, but it's a complicated topic. 

Jonathon Wright I guess you need to get more top complex when you talk about things like nano services and the next, you know, the serverless kind of approach to these kinds of technologies. You know, I think there's always a generation of I love what you have to say about using those blueprints and patterns that you've used to other places and how you've got to make sure that, you know, sometimes the technology doesn't always align north of maturity, always aligns, you know, when it comes to the stuff that you're doing. You know, How do you choose the kind of the right approach to going to support it, like a performance or security or something that you're helping with a customer. 

Bernardo Guerreiro So if you don't mind, I'll continue with the current example. So when I got to Auth0 was the zone was very much already all at the place that they had all these micro-services. And even there, it was complicated to introduce contract testing. And it took time. When I got to Auth0 there, their models, their architecture model was very different. They are working towards having some separation concerns with services. But some of the some of it is still not split into microservices. So perhaps contract testing didn't really fit that. And it's important that I think it was important for me to get their insight. I think I had just been accepted to give a workshop when I joined us here to give a workshop on contract testing. So obviously, you know, contract testing was on my mind. It's the last thing I was doing heavily on design. You know, I was submitting talks for it, etc. So it is the thing I was doing at the time. 

It was important for me to get into that new company and stop a little bit my excitement for it when it applied to a work as much as I would want to work to a place where we have that, I think there are other more fundamental things that we have to tackle before we get to a point where it's even stable to run contract testing because the problem of trying to retrofit these cool technologies on to any architecture is that you might end up just creating something that's not very stable. And what I find in testing, especially in the kind of team that I am where I'm providing like tooling and infrastructure and advice to several teams on testing, you have to be really careful to make sure that you're providing value so that the other teams don't kind of get discouraged for testing because whereas are in our current company, developers are responsible for testing in their teams. And we are kind of like a quality advocacy team that they'll stooling infrastructure and gets advice. So it's important for us that day they are still keen to work with us. And part of that is, you know, not having flaky things in the pipeline. And a lot of that comes down to that choice in the beginning. You have an idea. Sure. But think it through. Does it really apply here? And does it solve the real pain points of the company? And I think that's where low testing will come in if we talk about it at all. Auth0. That one of the pain points at the moment, at the moment that I joined Auth0 is more around performance testing. It was a much bigger thing to tackle and a much more important thing than something like contract this. 

Jonathon Wright You can definitely say that again. I've been doing the, every Wednesday we doing the Performance Advisory Council meetups and with people like Mark Tomlinson and Scott Moore and Wilson Mar. And, you know, I've always fascinated by performance engineering and, you know, just how complex those ecosystems are. And I guess with, you know, just like you said, being quite a lean startup, you know. You know, engineering to that kind of scale. And then things like SRE and all those other things coming through. How did you approach doing the low testing for Auth0? 

Bernardo Guerreiro Yeah. So with us here as use case day, there were quite big already. Like they grew quite organically. So by the time I drawing, we're talking already about probably. Half a thousand engineers, maybe. More so it's already quite a sizable company. And like I said, we've got several different product teams. There was already some work that was being done by our colleagues and very good work at Auth0 before I joined on that changing the frameworks. And I think that was a very important decision point at the time. They were using J. Meter and J meters. Not that I'm not saying it's BI, but it's kind of like a little bit behind its time. And it was the developer experience for writing those performance tests was not the greatest for developers. 

And again, in a company like ours where we run developers to feel like they are responsible for quality too, I feel like the developer experience in the stability of what you're providing is tooling is really important. They have already made the switch to artillery. Now, it just so happens that I have worked at the Zone with the creator of Artillery. So it's kind of like a good match there. And, you know, I had some some some thoughts about it, about how to use the tool to the maximum of its capacity. And I think a tool like an artillery was perfect in this kind of system because it's written. You write the tests both in Yamal and in and the logic in JavaScript. So I recommend that one, and in case six, for example, are both really good in this JavaScript ecosystem. And keeping the tooling kind of close to the ecosystem of the development is, I think, also a really good bonus point for developer experience, because then, you know, your developers are going to write tests in the language they are comfortable with already and they're using every day in terms of like how to tackle for the whole company, One of the approaches that we took that I think the team, in general, did a very good job in focusing on that was to be very data-driven. We as Auth0, provide identity as a service. And we have a lot of customers that use our tool to integrate into their products. So we are a software as a service company. 

As a result, we have a lot of data from different usage patterns of our product and we wanted to basically analyze that data and try to understand and categorize how our different customers behaved and try to group that a little bit into how they use the environment, because some people use it very lightweight, in a very lightweight model, while others sometimes hammered the environment, as well as the kind of like the size of the data that they had in the environment. This is all was also an important consideration in taking that as learning to drive which endpoints we want to hit, which ones maybe if we noticed that our databases were was a problem, we're a problem. One of the challenges maybe we would exercise endpoints that we know hit certain collections in those databases more because maybe those collections have created incidents in the past, know things like this. There are very data-driven, I think are kind of key to performance engineering in general. 

Jonathon Wright Yeah. I think artillery, you know, is it is a great product for that, especially if you're on, you know, the aid of U.S. stock, you know, it logically makes sense. And, you know, they use a kind of a serverless kind of architecture as well. You know, I guess, you know, as you said, you're picking the right tool whether or not you're using your terraforming or you to use conformation or something else. You know, part of it is you've got to choose the right tool that's going to do the right thing for you. And also, we got to support the developers with their challenges, which they have. One of the interesting things I always find, you know, is around things like time series data. And, you know, I've come across things like the ticks that could inflict DB or how did you collate those metrics that you got from, like you said, one customer that's likely using the environment and another which is, you know, hammering it and you've got different volumetrics profiles that you could have to do. And obviously that influences your rate of us and it's, you know, configuration and infrastructure. 

Bernardo Guerreiro Yeah. So we heavily use data for more life, life performance, and metrics of what's going on. Our observability teams are investing a lot in giving us more tooling to dig deeper into some of the problems and try to trace back. I think two key things to answer your question was that we keep a lot of their data, obviously not sensitive information, but about the usage patterns of the people using our product. And because we store that data in with an internal solution, we are able to query back that data in the ways that matter to us. So we were able to profile based on the endpoints that were most useful, the size of the tenants, as we call them, the people using our environment, the customers, the sizes of that meaning, you know, how much data they were using as well as, you know, how those endpoints affect collection. So that's one side of the thing. The other side is incident analysis incidents. If we're doing that right and we're keeping track of the data that we get from the incidents, as a general rule, they over time give us a good idea of what are the parts of the product that might be performing or badly more often. And we had a couple of incidents that were from related sources. And that was, for example, on our last order as a team, one of our goals was to focus on building some low tests for those kinds of points that we saw were more heavily used and maybe had gone behind some of the incidents that we had. So I think it just goes back to making sure that as a company you are keeping that data. So adding that observability is very important. And then that you're making use of that data because if it's just there and you're not using the data to make decisions, informed decisions, then you're just wasting money and keeping the data.

Jonathon Wright Absolutely. And I think, you know, it is quite interesting you are saying about the data dog. You know, I've been using a tool called Orange Data, which was a strange day for a data visualization tool. But so many of those tools coming through now that help provide a visualization of what's actually happening. And I think that's really important that people can understand the data. And, you know, one of the common mistakes I used to make was around, you know, people looking at like average response times that I hit on the real data. And, you know, I think Tableau and those kinds of tools that can actually visualize all the data I can give you, you can look at it from different viewpoints. Really helps to kind of give you an overall view of the system health and how to take it to get better at what you're doing. And I guess that was also that quick feedback that you gave to your to the developers, that if they're owning quality and that quality advocates, you know, that that must be really important for them. 

Yeah. And, you know, part of it is keeping the data and making it possible to get it back, but also understanding how to how things flow through the system. We were talking before about the complexity of distributed systems. And even though we're not fully there at all, sure we have a pretty complex system, And being able to trace the calls to our endpoints as they go through the different parts of the system so we can dig deeper is very important. So, for example, you know, when we run a low. And if we do find a problem, we want to be able to dig deep and find out where that problem's coming from. That's kind of the point. And if you don't have that set up in this setup, doesn't directly, but it's not directly the purview of your low testing framework or anything. It's something rooted much deeper in your development toolkit, in what kind of investment you're making in observability with the traceability and monitoring, etc.. 

Jonathon Wright No, that's great. I'm so great. I know you've got some fantastic tips on this and you know, I love the blog, so I recommend to everybody who's listening today. Check. How theqalead.com. And look for quality engineering remote teams. And you'll pick up. But it's a great article. As far as kind of some kind of best ways to kind of get in touch with you, you know, reach out, you know. And also, you know, the best way to get hold of you. What would that be? 

Bernardo Guerreiro So actually, that's an interesting question in terms of the timing. I've been racking up some personal stuff, but once I'm done with that, one of the things that I plan to do is be a little bit more involved in the community. I'm already on a couple of Slack like the one for Pact, that contract testing tool. But you can reach out to me on Twitter or medium @bernardobridge. Well, one of the things that I actually really want to do, because it was an unfinished thing that I didn't get to do more of in my previous company was some mentoring opportunities. And one of the things that I'm considering doing is taking some people who either want to get into testing or even software development as a whole. And it's kind of hard to start or people who haven't. Maybe you want to do more on the automation side of testing and perhaps doing some, you know, some free mentoring sessions there where I can point it in the right direction and, you know, give a few of the baselines, like some of the things like we discussed, like, you know, what's contract testing or what's talking about a little bit about distributive systems or even the, you know, the whole set of the software development lifecycle, those things that they're kind of like nowadays, a little bit prerequisites for being in tech. 

So, yeah, if anyone wants to reach out to me about that, I'd be happy to help out. And we can. We can do something. 

Jonathon Wright Yeah. I think they'll be absolutely awesome. I know a lot of people may take you up on that. You know, I think the ability to find a mentor, you know, is really important for people who are kind of starting in this career. And, you know, I also refer you on we're doing as part of the BCS, the British Computer Society. 

We're actually doing a mentoring system for four testers for the spot specialist interest group. So, you know, I'll put your name forward. I think you'd be absolutely fantastic. A perfect fit. So, yeah, it's been an absolute pleasure today. And, you know, we'll have to get you back on the show in a couple of months once we're all out of lockdown and see how you've got on. 

Bernardo Guerreiro Yes. Thank you so much for having me on the show. It's been a real pleasure too.