Next Generation Mobile Testing (with Eran Kinsbruner from Perfecto)

Read The Transcript:

We’re trying out transcribing our podcasts using a software program. Please forgive any typos as the bot isn’t correct 100% of the time.

Jonathon Wright In the digital reality, evolution over revolution prevails. The QA approaches and techniques that worked yesterday will fail you tomorrow. So free your mind. The automation cyborg has been sent back in time. Ted speaker, Jonathon Wright's mission is to help you save the future from bad software. 

Hey and welcome to theQAlead.com. Today, I have a very special guest, Eran from Perfecto. We're going to talk about what we covered back in December for the virtual community days event and how exciting the next generation of mobile testing. So don't miss out on this great episode.

I've just literally this week and I know you have as well, but this week, you know, I've just seen with the PlayStation®5 launch and Xbox. I've seen mobile apps just crumble, right? Just absolutely, just being destroyed and they're all impacted by performance, right? But we're at the back of our mind, we don't really think about mobile performance and how, what that looks like. 

So, so what's your viewpoint, you know, this is, to me, this is a, a taster of what black Fridays and yes, you know, an Xbox only comes out every seven years or something, but, you know, for mobile apps to fail so badly during this time, you know, is there a lesson that we've got to learn from that?

Eran Kinsbruner Uh, that definitely. And it's not just about, uh, looking at the shift-left, shift-right performance testing availability and coverage. You know, I talked a lot over the past,I would say even decade about test automation coverage as well as performance. And when people talk about coverage, sometimes they fail to look at, uh, you know, just the test cases or maybe just the platforms on the tests, because, you know, I will tell you what, uh, just this week, Apple released a new, uh, iOS 14.2.1, just for these iPhone 12 series.

So there is already kind of a little, uh, bottle of iOS 14 that is only fixing a specific bug on a specific set of devices. It relates to coverage because there are going to be impacted applications running on this new series of iPhones. Me and you own this new iPhone, so you got this update, uh, as well.

Uh, but it has implicate implications on mobile application testing as well. So that's one thing. Second, and what I see becoming the new normal is this new modern application types. I don't know if you've seen Jonathon, you know, uh, the APKs rising from Android and the iOS app clips, okay? I just actually played a game on an app clip on my iPhone without really going through the app store.

It's such a small game, you know, typically when you think about the game and we talked earlier about gaming, right? You download the huge, you know, IPA or an APK and it takes time. It takes resources and it performs slowly. But no, I didn't even install the app on my iPhone. Just click on a link for my Safari browser on my iPhone.

I played two or three levels. And then I got the option to download the full app. Why is that important? A - it saves resources. B - it gives you the taste of the look and feel, the performance, response time, everything. And this is kind of the new normal for developers. If you get it right. Uh, with this subset of application and the end user loves the UI UX, UI performance, everything, then even go and download the game.

You will embrace it and you will get more coverage. It will be more spread. What we see in up until now is that, you know, you just flow batch of code, huge, massive application, which as you mentioned earlier, it doesn't really get tested even for minor change on all the different platforms. And then it blows up.

In Black Friday, pre-Black Friday and stuff like that. So to summarize, I think that the world is moving to a more, uh, you know, in the DevOps world, we talk about Microservices. Here I'm talking about Microapps, okay? You're going to see a Microapp, uh, APKs for Android, app for iOS, PWA - Progressive Web for web.

So we are going to consume a smaller degree of applications, which will eventually also give you better user experience. And it will also be easier to test. 

Jonathon Wright Yeah, it makes so much sense, right? And I know we've, we've talked and we've covered so many topics in the past, but you know, part of something like react, React Native, right?

You know, part of it is it's it's, it's mature enough. And of course itchanges, you know, it's changing. Facebook are probably updating it hourly, daily, every minute. So you obviously cannot compile an APK IP, uh, a file with a version of react, right? React native. And, you know, part of it is you're restricting your, your app to work within those confines.

Whereas we know the app, the phone's got that capability to run a little bit like, you know, direct techs or anything else is the idea is it becomes a standard where you can pull libraries and those libraries are updated in real time. And therefore the code, well is actually compiled on top of that is much smaller.

And that makes a lot more sense because also it means that you'll get a certain level of abstraction because you no longer reading through react code, which is just included because it has to be included. Like my, my favorite part of React Native is the experiment code where you can literally say, you know, if you are a flagged user, which is a VIP, then, you know, imp, you know, do this, if not, catch that. So, you know, the idea is you start building in experiments into that front end. You've got, you know, one of the things I was supposed to be talking about for Eurostar this week was dark canary, this kind of combination of dark launching and canary rollout.

Well, why do we do that exactly what you said, you know, a new, uh, update comes up. Does everyone have to update the iOS to do that? Or is it just those specific devices? And then is it those specific dices of devices that are on a particular hardware batch that are slightly different? You know, we knew during this, this 14, uh, sorry, the 12 pro maxes that actually they were pulling out chips from the, uh, the iPad pros because they'd run out a chips, so they were reusing it. 

So there was difference in batches, right? We know the same kind of thing with hardware is always at the start, it's a little bit uh, challenging, right? And I think there's been this ig— we've ignored hardware, and we've virtualized it, you know, the idea with Java and everything else is that we have a fine out of access to direct access to the hardware level.

And that's why generations of like the Xbox once series, which we're, we're talking about at the moment. And the, and the PS5 is they've got more access to two, and that's why there's so much quicker. And we know these devices are getting superior. And I think it's really interesting because I finally replaced my phone.

Like you said, 12 pro, very excited. And so I expected a different experience. I somehow expected my apps to load faster. They didn't. I expected the apps, which I play to not be jittery and, uh, and have all these glitches in them. And they still did. So even though I was using the flagship device, they still lagged, they still had problems moving around the screen.

And I think I kind of asked myself, well, why is this so difficult? And it is, you know, it's partly is this, um, everyone writing their own proprietary code, which we always knew was going to be the problem. And, and, and, you know, I, I, I must admit I [00:08:00] confused mobile gap Gavin as this kind of, you know, the end of, of this, this plethora of apps, but actually it was all around app to optimization for Google.

And this idea that Google could needs to start taking into consideration, good apps and bad apps. And start looking at the like you said the, uh, uh, the apps, the reactive design apps and saying, okay, well this is, you know, this has got a good score. It's easy. It's following accessibility, all these basic things which we take for granted.

And actually this is the sign of a good app. And I genuinely think from 0 to 10, which I'm going to kind of call just onboarding is the most difficult challenges. It's also where you lose all your users, right? Because you, they they're excited. They download an app. First, they've got to wait for the app to download.

So sometimes, you know, talking up to a gig, which fair enough on a 5g network, isn't that isn't that difficult, but then once you've got that app, it then starts patching. So it starts pulling down libraries that potentially needs and configuration changes that didn't require a repackaging of the mobile app to then you wait and again for these apps to do.

And then there's this kind of this phase where it's connecting and you're kind of thinking to yourself, Isn't it already connected, but now you've got this kind of, okay, now I'm going to handshake with the server and then potentially the servers up, the servers down, the servers are unreachable and there are all these touch points along, is this, if we were looking at you at good UX design is where people can have a negative or a positive experience. You know, waiting for something, you know, is, is a very difficult, uh, experience for most people. And I don't think people have got that down yet. And, and we we've got, we've been doing this with the guys at MIT.

You know, onboarding for us was always our most challenging because it was the part where we said enable exposure notifications. And for most people, exposed to the ex exposure notifications was something that was new and they were kind of going, well, what does this mean? You know, and it, re again, redirecting them into the system level going, do you want to turn on exposure notifications without any real context of what does it over look like or overlay or read them something kind of guiding you or explaining to you what it is because, yes, it's fine for us.

We kind of understand exposure notifications, a Bluetooth beacon that exchanges keys, but for somebody who's either older or younger, you know, they might not get the whole point of this and, and therefore they never successfully onboard because they skip that step because like the first thing that every mobile app does is it pushes you that notification.

Cause it's kind of a bit lazy and it's not using anything like Firebase messaging or something clever to kind of say. Um, you know, how do we do your push notifications in a more sophisticated way? It literally says, come on, let me have access to send you annoying messages. And it's all all in the wrong way.

Cause you're not ready to commit. You've only just downloaded that mobile app, but again, X, the notifications is an externally facing kind of service, which they can push those notifications to you, which means they're time sensitive. It might be, you know, like the Amazon truck, you know, they pushed the notification, went to get people to actually build up, uh, some, uh, excitement about a product launch or a product offer, or it could literally just be you know, uh, we're missing you, come back into the app and stuff. 

And I don't think people have got notification testing in mobile apps because we're doing time-based, we're doing UI functionality. We're not doing experience, which people have every day, whether or not they're driving their car and it's, uh, notifications coming up, um, or on their Apple CarPlay or something.

So, you know, how do you think people are going to get better than this? How are they going to start understanding these digital experiences, these, you know, over a number of different devices, over a number of different end points. You know what that experience should look like between apps and also between mobile and web?

Eran Kinsbruner So I think, uh, there is no magic solution here other than really virtualizing as much closer to the reality, the environments, meaning, you know, if you want to understand what an end user in Australia is going to experience, when you push on it, push notification, when it's close to something that can offer it, uh, context specific offer, uh, you know, to the app that he's using. You know, you need to be able to mimic this experience in your own lab environment that developers, the testers need to understand. How does it work and what are the implications? 

Um, I am now building a new product, uh, within Perfecto and I'm going to serve so many different customers. So when I started doing some validation, like any product manager would do, I spoke with few customers and I was blown away because, you know, I have my own product requirements, but guess what? My, uh, my customers, uh, and the end users are actually, you know, I'm just giving you an example, Cypress. 

Cypress is a new, a rising star in the cause based test automation, right? It's a JavaScript framework, but how many real Cypress developers out there that are building Cypress scripts on top of BDD, okay? On top of cucumber, because it's a front-end development environment, developers are not really into this, uh, given then when, uh, and then I found out, okay, someone actually build kind of a Cypress plus cucumber.

Um, I wasn't, I wasn't expecting that, okay? Uh, and then suddenly you need to build this kind of environment inside the dev team because they need to test against that. Um, so it's the chicken and eggyolk, but without really virtualizing and really, uh, developing, I'm not talking about testing, developing from the end-user experience.

And you talked earlier about shift, right? And, you know, bringing the production, uh, into the development workstation, you know, that's the only way, uh, to really, uh, mitigate risks, to really delight, you know, the customers in this digital world where we are living in. Uh, so that's my, uh, gold, uh, advice to anyone who is listening, try to virtualize, try to think like your end-users, uh, build these sophisticated environments, do negative testing. Negative testing is something that you don't often hear about, uh, back or back in my days, you know, I was working at Sun. I worked at general electric and I was trying to really build a product which does the actual opposite of what my product was trying to do, just to see, you know, what's what it looks like when you fail.

And this is how you build this, you know, non-happy path test cases. It requires creativity and many more. So, uh, I think that, you know, if we are marching, uh, and we are, uh, into a very brave new world of digital reality. Uh, the other day I was at Best Buy, you know, when you go to the Best Buy store today, especially in the US, you see a lot a loud sticker on the door, uh, we are always open at bestbuy.com, okay? 

Because in these days it can be closed, right? But their website they claim, you know, and they put it very clearly in your face. They're always open, you know, the website is always on, you can always get this delightful user experience, but, uh, you know, they put themselves in a situation that they need to commit.

They need to, uh, ensure that you get this performance that you talked about earlier. The user experience that it works on Edge and on Firefox and on Safari. And that that can consume all these end-user journeys on their website, either on a mobile device or web or anything else. So it's not easy. I'm not saying it's easy, but few of the things that I've mentioned I'll, you know, towards the either action together.

Jonathon Wright Absolutely. And I liked that, I loved the best buy kind of example as well. Um, because you know, to me it's it's, it's like what you get with washing machines and stuff. Cause you should be able to get this little rating that says A for efficiency, B for whatever, and it should be independent, right? A bit like, um, you know, a governing body of apps, which kind of tells you how actually easy and easy is to use. How, um, um, you know, how reliable it actually is. And, you know, also to externally getting feedback from, from people's viewpoints to, to kind of balance that out a bit like a Metacritic score, right? Um—

Eran Kinsbruner Independent benchmark or something. Yeah.

Jonathon Wright Absolutely. And it should be based on, you know, what, how they implemented the technology, right? How easy it is to use from a UX perspective. Um, another one, which I think is, is brilliant, which is this, you know, I, I, from your research, cause I, I always, I love your, uh, your, your mobile index, which you send out all the time, but you know, this multi, uh, people move between devices.

You know, they may use their mobile device to browse, right? They might use their iPad to browse through products that add them to their basket, which is one of the behaviors we talked about last time. And then they'll complete it by using the browser, because there's a concept that they believe that it's more secure or something, or maybe it's got the PayPal data they're already all set up or their card details.

So people are using these multiple platforms and they should flow between. Now we both know that typically there's a divide between the mobile app development company, a team and the website e-commerce thing. And yes, they're both going to the same e-commerce centralized platform, but the web front-end and the mobile front-end sometimes isn't going to be the same, right? 

So therefore the experience is going to be different. Also the release cadence. So feature X off. Okay, you can add, uh, now you can, uh, over a hundred items into your basket maybe existing on the mobile website. Cause they can do it quicker and it's a smaller change and it might be on the backlog for two weeks to be released to them, push to a mobile app.

So, you know, you, you start seeing these differences between platforms where, what people are consuming the same as those notifications, same as the look and feel we've we've, you know, like you said, we've not come far with, with this kind of side of things, but things like Best Buy, yes. We know Black Friday is coming up.

People are pushing hard. You know, like I mentioned about this, I don't, we've talked about before. I, you know, I'm still in shock that mobile apps just can't deal with a non-connected state, right? You know, they've got this, this they're brilliant when they're connected to a network, but as soon as eight, you get this lack of network, they, they didn't catch it.

They don't catch it like a web browser catches it, which is unfortunately, a very bad experience and it kind of says page not found or something, or it just hangs. Mobile apps don't as gracefully switched between an online mode of connectivity to an offline mode. And even the react native kind of platforms don't support that.

And I think that's a real problem because, you know, so by, I, I, you might've seen my tweet this week, um, about buying a PS5. So I, I kind of, uh, I pretty out, and it was really nice actually, because I got someone from about five people connected to me on LinkedIn, not, not Twitter came to me and said, I loved your tweet.

And I, you know, I do get concerned about these things and, you know, I, I think things are becoming quite negative and, and it's not a positive experience now with mobile. We're just, you know, we're not treat we're tracking the users are treated as secondhand citizens, right? They're not getting the experience that they deserve.

And so I'd gone on to John Lewis. John Lewis had said, okay, we're going to, um, we're going to release a whole sack of stock for PS5. So I got the stock notification. I've got the mobile apps, I've got the standalone mobile app added to my basket. Fantastic. And then between kind of adding to my basket and checkout, which is obviously when typically it switches between Stripe elements or whatever it's using for the payment processing and then the lookup engine that it's pushing out to find out what your address is and, and work out your account details is another system.

But just between those two kind of areas, it got stuck. And I know I'm going between the mobile app going back. Oh no, this page is not accessible. Try again. Okay. And then I'm getting in a circle of loop, right? And I spent about an hour and a half in this circle from hell. Um, and then I got finally got through to the page that I wanted, which was my delivery page.

So I'm a static, right? And it says delivery not available, pickup not available. So somehow it, the API is that had done those calls too, had failed and give no responses. And it's best way to say is, yes, you have the product in your basket. You can buy it, we can't pick where it's delivered and you can't pick it up from anywhere.

So I was then at not in a scenario where I couldn't check it out, I could pay for it. I couldn't get it delivered or specified and something like that, which is a very simple experience. You know, I put it out there and I kind of said, look, you know, and I know Paul Smith, who used to be the head of performance for, um, for John Lewis.

And I spent a lot of time, uh, in different countries speaking at different events with them. And I remember sitting down to talking about performance and Black Friday, and I said, you know, what did you do? And he kind of said, well, what we do is we, I optimize for the highest percentage of, of things that are, um, searched for on John Lewis.

And then we cashed them and, and we, so when there's a big iPhone launch, it's all cashed. It's all good to go. So don't matter really what's happening at the back-end, the front-end, uh, kind of keeps things going for a bit, right? Um, And they did the two biggest categories of technology and laundry, which I always thought was really funny because I said to him, well, that's not the full spread of what John Lewis sells.

They sell lots of things, but like Macy's, right? So why aren't you search? And he said, well, actually the happy path, it, 20 odd tests are all always technology or laundry. That's all we ever searched using the testing thing. So straight away, they're at a bit of a disadvantage because there's all these different products that they want to be looking at.

And then they're obviously then optimizing it for these scenarios where they're expecting sale days, people to hit certain products, sections harder. And, and that was how they approached it. And I kind of, you know, I do raise is no longer there, but so I don't, I don't blame me anymore, but you know, part of it was, you know, we can't just optimize an experience for one particular type of customer, right? All customers should be equal. 

And I think this is this new digital revolution that we're talking about is that people should be able to have the experience that they expect or require no matter what they're looking for. And the MIT app really shows us. And it, you know, we couldn't have done this without Perfecto, which has literally has changed our life, especially that few weeks, right? 

So we rolled it out to Puerto Rico. We rolled out to, uh, another number of different languages. So I had, um, Dorothy Gray. I'm on a call this week and that's there, and she's crowd testing it for us. And she's, so she, we showed her how to, uh, go into the debug menu and, and, and for say a exposure notification.

So she didn't have to have two mobile phones. So like you said, these deeper kind of capabilities that you should be building into your app. Um, and then she's going through and changing the languages to these different obscure languages that are supported in different countries, because language is really important because that's how they look on the screen. 

So if you put, you know, straight away things, things like French or German with a lot, slightly longer words, they, the buttons are slightly too long for the screen factor. And again, back to exactly what you said, you know, depending on whether you're using a 12. And you've got the real estate on a 12 pro max versus using, uh, an SE. Your real estate starts changing and, you know, react just doesn't react to it, so to speak.

And so therefore multilanguage in, uh, testing is really important. And again, it's one of these golden gooses and it, and where people don't think about that, right? They don't think that actually there's different languages that are required. So the great example is in the UK, the NHSX app, which we've talked about before, which failed miserably it's in use at the moment, but a couple of weeks ago, they, there was a whole stack of people who had their phone settings at, in French or Spanish.

And they couldn't actually report that they got COVID because it just wouldn't let it process to the next page. So they tested it in English, which is the normal way of doing things, but not with language settings set up differently at an iOS level. And I think this is, to me, this is a real problem. And everybody I know loves Perfecto because they that's the first thing they do.

They go into a device and they change the actual language off the device. Something that you don't, you don't always feel comfortable doing with your own phone, right? Um, and then change the language within the app as well. So you've got kind of two tiers, which potentially means the ISO characters and sets and stuff has slightly changed. I'm guessing there's other discrete things. Um, the best example we've had this week with the, uh, from, from Ronald who wrote the book Leading Quality was around Indonesia, not requiring a surname, you know, 40% of people in Indonesia don't have surnames and we make surname a mandatory field in every single one of our mobile apps, which I'm I'm to blame for as well. 

Well, we need their surname. Well, we don't, if they live in Indonesia, you know. What's your advice when it starts coming into these kinds of complexities of, I don't know if the, the mobile index talks about this stuff, but, you know, language languages and, you know, screen resolutions and contrast levels and accessibility and, uh, you know, uh, all these other things. What, what's your, what's your viewpoint and recommendations to kind of people listening? 

Eran Kinsbruner Sure. Uh, so, uh, a back going back to the MIT and the react native application. So actually I work with the army at Mackenzie, uh, on a bug that, uh, you know, was very surprising.

You know, text was being, uh, in English. I'm not even talking about different languages was simply disappearing from the screen, uh, on a one plus 70 device on a Motorola and other Android devices it was showing perfectly well. On a one plus a device, a Chinese manufacturer device, uh, the text was simply disappearing and, you know, uh, it was testing it on the Perfecto and he was, uh, talking to me and say, listen, you have an issue, uh, in the Perfecto lab because I see them as a wall.

It looks good. It looks, so listen, if it looks good on one device, it doesn't look on the other. It's not, it's not our platform. It's always something, you know, in the app versus the device. That's, you know, the metrics that, the metrics that you need to build. [00:27:00] So to your question, I want to give this as an example, because that's exactly my response to your question.

Um, when you test for a mobile application, whether it's react, uh, or, you know, a native application written in Java or whatever, uh, it's about, uh, you know, you cannot cover everything. It's clear. That's, that's why I hide this index reports because I'm giving you the different configurations to mitigate risk as much as possible, because we know, any Samsung device, you can take three generations that are running different operating systems. Even if it's Android 10, it's Android 10, uh, in a version that is called One UI 2, okay? If it's unloading nine, it's One UI 1. It can be even touch ways if you go back to the S7 of the world, right? 

So A - you need to look at the device and the operating system families. Then you need to look at, like, you talked about the iPhone SE and the iPhone 12. You need to look at the, uh, um, I would say screen size resolution pixels per inch density. These are all things that matter, especially when you talk about, uh, different, uh, UI, different languages, strings, popup notifications, security alerts, everything, okay? 

Then you need to look at the UI, UX because you know, uh, the new iPhone 12 mini just got an update because, uh, 8 has a smaller battery, okay? It has a smaller battery capacity. It consumes much, uh, legal resources compared to the iPhone 12 four max. But you, from an application development perspective, you are going to give just one MIT application adding on to these devices.

And I will tell you that that's not the only application running on these devices. So this race condition is something critical for mobile testing, you cannot. And that's what I see almost in any account that I go. They think about the app as the universe, but that's, you know, your app is maybe 1% of what's going on on the device at the given moment.

You aren't this MIT application and it's, uh, uh, you know, uh, subscribe to push notification. It is connected to the location services, APIs, and, uh, you know, the same time you have applications still, you know, you have ways and you have Google Maps or Apple Maps and you have the camera and, uh, you know, the browser is running.

So many things are happening. Uh, so going back to the visualization or, uh, I want, uh, do marketing for Perfecto, but you know, you want to have this environment for testing the 3d matches all these different complex scenarios and it cannot be just the device and just your app, okay? That's not a test plan.

That's kind of being naive. That's the epi path that you talked about earlier, Jonathon. You need to think really wild, okay? It's not even Kelsey engineering, it's just reality. The reality is that in a given time you're getting at least a dozen notifications from so many apps with so many access to different permissions. The camera, the GPS, uh, other sensors.

So either you test for the real world or you kind of a wait and play for, for things to work for you, but we know that it usually doesn't work that way. 

Jonathon Wright No. Absolutely. And I think the app store talks for itself, right? There's so many angry people there. And I, you know, who are talking about combinations of things and, you know, I'm always blown away by some of them because I, I feel like I need to hire them, but you know, part of it is, is your testing in the real world is the only real world, right? 

We know, we know, we kind of say in that now is if you test your app against, um, an emulator, you are truly liberating yourself, you know, and I think the only way is physical devices. Uh, and, and, and, you know, we've, we've had conversations before about things like, you know, GPX and how it poles on the battery.

And I know what I, what I, I got the iPhone, the, the assistant said, oh, you do realize that, you know, it goes the iPhone. You can turn the 5g off to save battery. And I say, well, what's the point of having the 5g if you're not going to use it. And then of course, if you read the articles, it Apple say, well, actually they do this really cool energy saving aspect where they turn it off when it's not really necessary in the background.

And you're thinking a bit like Bluetooth, low, low power mode. Is that what you're certainly getting all these limitations by having these power saving capabilities and you know, we've got it now with the exposure notifications. The reason why all these people are saying I'm getting false positives is the calibration of the Bluetooth beacon is going to be different based on different hardware providers or even different versions of the hardware that's been installed within that device. So, you know, that then is kind of a, you've got, you've got to tweak that a bit to get them at the correct range because each one of them is going to be different.

So certainly the complexities and more exactly what you said, Australia, you know, you've got, uh, you've got location sensors. I worked for Westpac when I was over in that, right? And they have their app allowed you to find the nearest ATM, which is great. So you need GPX information, but you know, at what point do you need it in the background to say that you're just walking past one and it says, oh, by the way, you know, you can go and, you know, do you can go and get some money out or, you know, at what point does it trigger it?

And at what point is that running in the background? A lot of these things people don't think about. I think, like you said, a lot of it's happy path and I think we can, we can move leaps and bounds by bounced by that, you know. One of the things, what I loved about your WindJonathon Wright is joined by Eran Kinsbruner, Chief Evangelist at Perfecto. Eran is a recognized mobile testing influencer and thought leader, as well as an experienced speaker in major software engineering conferences. Listen to learn more about the next generation mobile testing. Tunnel app, you know, was this idea of giving them personas and starting to think of, well, what are those user journeys, right? 

What is a user X look like for in Australia versus Indonesia? You know, what is the paths that are different in those ones? You know, the same as you do for your web, you go, okay, well, actually 10% of our users are speaking this language or in this country, we need to facilitate at least 10% of the tests around them to support that, right?

The same with device, you know, the whole point why you bring this fantastic index out is because, in Indonesia, they're using different devices with different versions than what they are in Canada, because the frequencies for the cell towers are slightly different. People still use BlackBerries in Canada is very odd, right? 

But we didn't look at it like that. The Walmart guys or the Best Buy guys, I sat there going. Everyone's in the UK, in the US and everyone's using an iPhone and everyone's got the latest versions of the iPhone iOS. They've updated it as soon as it comes out. And they're always updated the latest version of the mobile app.

So we've all, we don't have to worry about all that other stuff when actually. Just that one to 3% of people who were getting failed transactions could potentially talk millions of dollars of extra income that would justify the work to be done mobile, mobile level, right? I think that is the use case now is actually, don't emulate. You've got to use the real thing. And if you want to see every single device in the world, fine, you're going to need a very big lab or you have to leverage Perfecto, right? And you've always been at the forefront of this. And I think going forward, things are going to get more complex.

They're not gonna get easier. You know, we're not even going to talk about that. The, the AR capabilities and the pro max with the new radar thing, you know, these kind of challenges are going to get they're coming through now, right? You know, we've seen—

Eran Kinsbruner Just the other day, uh, one of my clients was testing accessibility on a fall of a smartphone in the cloud, which just descenders on its own seems okay. So far I had no, you know, you have so many different brands that already selling and such a device, uh, cost about 2000 US dollars. You're paying so much money, okay? And if it doesn't walk or if it's not accessible, you know, old by such a device, either kind of a gadget freaks like me and you, or, uh, someone who really needs this big screen from accessibility perspective.

And, you know, I work with IBM and they are doing accessibility research for these foldable devices and they were doing it on Perfecto. And I tell you that when it was in a folded mod, it looked fine when you unfold it, suddenly you start to see so many, uh, uh, audit failures on the accessibility, uh, spec, which is, you know, I wouldn't say it's the future.

It's the reality. So to your point, yeah, we are just, uh, you know, in a, uh, junction of a new digital reality with APKs, App Clips, progressive web foldables, 5g scenarios, and, uh, AOL, which you need. You need to have good base that already today for your naive applications. So you have a good start to these new innovative scenarios, because if you're failing today on the basic level, you have no chance of, uh, you know, making it through 2021 and beyond.

Jonathon Wright Okay. That's I think that's kind of suddenly up and I do think we need this standard. We need to standard, which gives us this kind of index of, you know, what, what level it's being checked. And I do again, back to accessibility, it's the same thing, right? You know, accessibility is going to have a massive impact screen brightness.

You know, contrast levels, all that kind of stuff, which we take for consideration certain generations, uh, from a, uh, from a demographics perspective are using apps in different ways. They've got screen size and fonts at certain levels. There's so much complexity there, which we still need to get right from a basics level before we even think about what's coming down the line with these next generation, uh, of devices. 

Um, and I think, you know, partly as a, kind of a, as, as a leaving kind of topic of where we're, we're kind of going and where, where we need to, um, we need to kind of stand up and really take in this kind of getting the fundamentals right. I think he's, is absolutely essential. I think, you know, you've got to, you've got to have a good test plan, like you said, uh, you've got to know what you wanting to do.

And you've also got to have access to the necessary devices that your customers are using, right? And you've got to understand how they're using it. Um, and this is, you know, this, isn't just another digital strategy kind of thing. This is, is going to be a lifeblood, right? It's going to be whether or not you actually, uh, you make revenue or you don't make revenue or you die, right? 

Because we're all in a situation now with, with, uh, with COVID and, you know, I keep on reading articles where people say they're buying the, the, the, uh, 12 mini because they're at home now. They don't need the large real estate because they're not commuting and watching movies.

They're wanting to just move between different rooms. So people's re uh, uh, requirements are changing and, and that seeing them, people like us who want the bigger estate, they want the new technology. 

Eran Kinsbruner Yeah. 

Jonathon Wright And I think the foldables is one of these big kind of unknowns going forwards is, well, what does it look like? What does, which screens go where? And I think there's also a, another big one, which I'm going to kind of finish on, which was this, this, this Gen Alpha and Beta, where we excited to see different behaviors. So this week, something I've been talking about for absolutely ages was, um, the Instagram Shopping, right?

So Instagram or Facebook shopping, uh, has been on the cards for a long time. And, you know, part of that experience now is they see a picture. Like we, we see each other's Facebook profiles and you've got, um, some, some new barbecue in the background and I can click it and buy that barbecue, right? And you know, that, that we talked last time about the company that I, I T I worked with with computer vision that do this on Amazon and, and Stephan part of it is people now are wanting that simplicity, right?

They didn't want to shop around for the best barbecues at bestbarbecue.com. They literally just see the barbecue you've got. You've had a, just a chat with them on, on, on, on Zoom or any other platform. And you've got, yeah, it's fantastic. You should get one. People will just click on it and buy it and buy it within the same app.

They shouldn't need to leave the app. They should need to put their delivery details in because they know where they are. They know what their payments are, cause they've already got Apple Pay and stuff set up. They should be able to do the whole thing with three clicks interaction to a pitcher. 

Now that's going to get worse once we talk about AR next time round with, you know, people just, you know, hovering it, seeing what the new sofa's going to look like. Then think, oh, okay. I'm going to order that and do it all through that kind of, uh, AR platform, which we're going to start seeing with the Google Glass coming through and I mean, sort of the Apple glasses coming through, and you're going to start seeing this augmented reality going to be taking over. People are going to go well, I want to see what it looks like beforehand.

Okay. Well, I use my, my AR device, which I have in my hand, which is my phone. And so this new era of shopping, which isn't the, you know, the way that we typically shop. We've got to support those generations and they're just generations differences. So I think we've got so much to, to start thinking about, and we can't dismiss them because we believe that that isn't our target audience anymore.

Anybody is our target audience, whether they've got disabilities, whether they've got, uh, uh, issues with sight. They all need to be taken into consideration and everyone should be equal citizenships when they're digital citizens, right? And I think this is where the new mobile revolution will go. And this is where Perfecto is going to support customers in an entire new, brave new world.

So, obviously, I talk a lot around the shift-right aspect of how do you create a digital twin or some kind of on the right hand side, some kind of blueprint based on Telemetry data to give you to create your scripts based on a, on an existing system. Uh, and that's kind of, my viewpoint is that's kind of the future and that's where AIOps suddenly blends in as far as, you know, I, I'm talking to a lot of adolescent companies at the moment and, um, uh, and research and investment companies that I'm trying to put, facilitate that capability to deal, move past synthetic tests to more realistic tests in production. Test is that test should be in production and they should be testing in production and they should be implemented as kind of a code thing.

So, you know, we don't have this software development life cycle because it just doesn't make sense. On the left hand side, we have test running continuously and, and self-healing and things like that, which just, you know, kind makes sense, right? Um, and you know, part of what I'm seeing, I'm seeing today, You know, someone kind of halfway through this afternoon, literally two hours ago turned around and said, well, who tested this?

And I kind of thought to myself, well, Yeah. That's the whole point is someone says to you, oh, can you make this DNS change or make this, add this to the five gateway and you just go and do it. That'll do and you're kind of, it's just part of what you're expected to do, but no one's saying to you, you must build a test in there that kind of says, okay, what happens if something goes wrong?

And I think that's what the, the mentality will be is we don't make changes anymore. We make tests that rapid change, like the behavior driven testing. And then what happens is, that doesn't happen if the DNS starts causing problems or some behavior or anomaly starts happening in a way that the behavior wasn't expected. Then it stops, it kind of rolls back and self heals, which I guess is kind of where your, um, talk with what you were talking about for your continuous testing kind of landscape is, is to me, is that, that moves a little bit to the right-hand side.

Um, So I'm quite excited about that kind of aspect of, of synthetic monitoring, but synthetic, uh. My viewpoint is synthetic tests aren't enough. They're just to the hat to happy path, right? So all your app D Dynatrace, all those Splunk and stuff, they're all happy with happy path. Very simple. All the APIs are the API POS down, not okay.

The kind of stuff with chaos engineering, where you've kind of got that kind of, okay, well, what is. What does systemic failure look like? And what does that look like if that's, you know, geo-based? Um, so, yeah. So the, the idea of, you know, all these things starting to come together is, is really interesting.

I don't know if you saw, I did a podcast with the CEO of, of Grambling software. Um, and it was, um, he was, it was really interested in that kind of stuff for Dixie done Chaos Monkey at Amazon and, um, uh, Netflix. And it was taking some of that kind of knowledge across. And it was, it was really interesting to see what he was, you know, he was, he was talking, he was, he's talking about, and I think.

To me somewhat like Perfecto and also pretty much every other vendor that I think of in my head is, is that they would eventually become part of this open test API. They'd become part of an ape APM where those tests live on in, in the right-hand side and even the servers and stuff, um, you know, ended up being, um, you know, sorry, the, the, the, the, the physical devices are testing scenario as well, just, you know, uh, production is, and then you are, you know, you're kind of taking them to say, well, they're not actually these aren't production transactions. You can, you know, um, and there's lots of ways you can do that with the experiments and stuff. 

So I think to me, that's what I keep on thinking about when I think of this, this kind of new future.

Eran Kinsbruner Uh, Jonathon, you, you, you have to smile, uh, honestly, uh, you know, if you talk to any average test automation engineer, you know, and I talk to them almost every day within the enterprises that we work with. They're so far away, uh, from a mindset perspective, from what you envision, which is perfect, you know. That should make sense because in any case they are, they are not successful in what they are doing today.

They are failing, okay? It's flaky. It doesn't cover enough. It's not really looking at what's coming from production monitoring. They're not even doing production monitoring, okay? They're not even doing single user performance testing, you know, the basic of, you know, uh, response time availability. So, um, you know, the average market is far away from what we're talking about.

And that's what worries me, because I don't know if you know about this new initiative from KK, uh, X cloud-based. Uh, so he founded a company called Launchable. Launchable is not exactly what you're saying, but it's even more interesting. You know, I, I recommend you to look, uh, Launchable Inc, and what they are doing.

They're looking into the pipeline into the CI/CD and they are doing some kind of a code impact, an IC stat test impact analysis. But again, AI machine learning, all these buzzwords in there, um, that, that they're still being up, but I think they have a vision of maybe not going towards, uh, too far towards the light.

But they are looking at the smart, um, correlation between code changes, test automation, aggression, what needs to be done, uh, only the subset so that you really get this fast feedback that everyone is talking about. So, um, it's quite very, it's interesting. Yeah. I spoke with these guys a few weeks ago. 

Jonathon Wright No, no, definitely. Yeah, that'd be, that would be really interesting. So, you know, I, I like, I love this kind of concept of unbreakable pipeline, right? Is to me, this is kind of what we're talking about when we talk about continuous testing and everything else. And, um, yeah, you know, I don't know if you come across captain, which is what, um, Andy, the chief, their chief technology evangelist for Dynatrace to be working on. It's an open source project.

Um, and it's got this kind of concept of having, um, I'm going to call it kind of, uh, a blueprint or a doc, a digital um, signature off each build. And what he does is it kind of looks at every single one and looks at lots of different dimensions. And then he's trying to identify those anomalies, right? And you know, to me, things like Tableau and all those kinds of tools, which support visualizing large sets of data, right?

It starts, you start seeing things that are happening. What are the tiny little changes, right? 

Eran Kinsbruner Yep. 

Jonathon Wright Which you kind of can say, okay, well this is a good build. This is what a good build looks like. This is what a bad build looks like, right? And then you've got this kind of predictive, I know we've talked about predictive ALM over and over again.

Um, but what I thought was really interesting is we kind of explored it and there's a lot, there's loads of interest in MIT libraries that do this, which do these kind of model the code, right? And then they, they, they, these code maps kind of give you a viewpoint of complexity. And it was always, you know, one of those things that I remember, I never can pronounce psychometric complexity or whatever it is, but you know, it got to a point where you kind of looked at certain parts of the code and the dependencies of that code on other parts of the system. 

And then you could kind of say, well, they're actually the most, um, you know, potential pain points by looking at the underlining code. Um, and we kind of had this with IntelliTrace and stuff was the idea is executable code versus debugging logs. And this, again, this test is open test API, which kind of allows you to turn flags off, to see what the code is doing, whether or not it's healthy, et cetera, et cetera. 

And, and I think underneath that is there is something to it in the sense of, you know, what we were looking at was, okay, it's this huge spider kind of diagram, really complex systems, all interconnection. And if someone makes a quick change to one of these areas and, um, you know, it's committed and it may be, it doesn't go through the same peer review process that it should have done.

And you know, the developer who normally maintains this functionality wasn't it there. And somebody else is picking it up and then checks it in. The risk level doesn't really share. If you look at the get-ops approach is it's another check-in, you know, you look at it, it's a small check-in. But the impact is huge, right? And that, and then the code impact analysis, which is what these kind of guys are doing, is this starts using the AI to say, okay, well, what's the risk level based on the committed change. Okay, actually that we need to either wrap more around this to work out whether or not this is good or it's bad.

And I think that DNA kind of viewpoint is you just going to start seeing very complex models being produced off software development life cycle, and this unbreakable pipeline will be the outcome of that. And that's great, but you know, that's maybe too late, you know, part of it is what happens in the right-hand side.

And, and I think, you know, partly this right-hand side is also, you know, um, there's a couple of things that messages are coming through. There's obviously this, this temptation, which we have around scrapping things that are old and replacing them with something new, right? So there's this, you've got a finite amount of software development life cycle resource, and you want to utilize that pipe all the time.

You can't just turn it off and remove those people. So part of it is they refactor and look at replacing or, uh, rearchitecture in sort of those, there's things, but then the underlining business rules that might have been written in COBOL, or might've been written in some of the lavish smart people did that, right? 

Smart people worked out that, um, that blueprint for what, uh, derivatives or whatever else they're trying to do from a business perspective and they express express state in a language, was that language sophisticated enough to deal with what we're seeing now with kind of CAFCA and everything else?

Probably not, right? But you know, whether or no they were using TIBCO and it was a, you know, an enterprise service bus and it was maybe overkill and it costs a lot of money for licensing the architecture, you know, but the underlining logic was good. I think part of it is how do you go, how do you transition from not a throwaway society and to what we're seeing now with kind of recycling and social responsibility is, you know, don't throw away all that hardware that's been developed, you know, abstracted, but don't abstract in the language which it was written in.

But with the business rules. So, you know, to me, that's why I love the right-hand side so much. And you know, and it's, it's everything, it's the overnight bachelors, it's, it's all the stuff which like ETL and stuff, what people don't really think about. I'm really excited about, you know, your talk and kind of, kind of what you guys are working on.

Um, and I'm looking forward to you guys. You've been the opening keynote on the 4th of December. 

Eran Kinsbruner Yes, I'm looking forward to that, uh, you know. This going to be an epic event I've seen, uh, you and the other speakers, so I'm truly excited, looking amazing. Great, great agenda. 

Jonathon Wright Wonderful. Well, it's been a pleasure as always, uh, around, uh, we're going to have to continue this on another podcast, uh, with the next generation, maybe the post, uh, Black Friday this year we'll we'll, we'll come back and see who failed and who succeeded at, um, you know, surviving that period.

So, thanks again so much. 

Eran Kinsbruner Thank you, Jonathon. My gut feeling is that we'll have plenty of good examples after Black Friday.

Keep on learning and gain the insights of an internationally renowned, award-winning software engineering consultant, author, and coach when you read this article: LEADERSHIP IN TEST: SERVICE TESTING