How To Test Complex GraphQL Knowledge Graphs (with Mike Harris from Geckoboard)

Jonathon Wright This podcast is brought to you by eggplant, eggplant, how to test, monitor, and analyze the end to end customer experience and continuously improve that business outcome. 

Welcome to the show. Mike Harris, he's a fellow BCS committee member. He's been doing testing since the 90s. You know, he's also been doing some latest greatest testing against things like GraphQL, you know, I'm facing that challenge. And if it wasn't Field Ministry of Testing blog, you know, helping people understand the importance of maybe exploratory, but, you know, this is a different will. When did you start? 

Mike Harris With brought or with testing. 

Jonathon Wright Well, with testing in general. 

Mike Harris Oh, OK. So I graduated with an investment in the late 90s and I joined Shell to work on the data warehouse projects. So trying to just take some time ago. The testers are at one end of the room, the developers at the other end third weren't encouraged to talk. So a lot of my testing really since every few months. I learned an awful lot. There were some really good people there that are testing to me should learn about manual testing. My six conferences.

I think that back then. So yeah. As a control. Sure they're full of that. It actually works in British Gas and they run a small says to send in sort of 40000 people who were laid off in the late 90s. So this is a second crash. I think a lot of testers and that's what it's shown, that they will ask teams out to start a calculator. I was running an offshore testing team based in India. So then for a while, but I went to Kentish Town where initially I was the only tester.

We got the growth curve, but roads grew to test. We moved from Waterfall to Lean and Agile we moved from monthly releases to continuous delivery. I learned tremendous amounts in the process. So a lot of Russian sports and they still in touch with the people there. So which is nice. Well, I was that to Geckoboard on the tester to Geckoboard folks did automation and testing So since I think that the development team said. 

Jonathon Wright It's fascinating because you get that kind of exact dashboard aspect, it's like the Holy Grail, what Geckoboard does. I've kind of giving people KPIs that actually being matched to that business. Right. But, you know, tough testing that that was my first hardest thing I ever found in automation was I went for a company and they made a graph come out at the back of it.

And I was thinking, how do I know that the graph actually has the blocks in it in the right position? All that pie chart is correct. You know, there's a lot of challenges with that visualization of KPIs, you know, so. So what kind of approaches do you do? I know you were going to refer to your blog and, you know, the exploratory work you'd be doing to kind of, you know, catch things earlier and before they're actually visualized using, you know, graph query time, which I'm so, you know, what tips do you have for people who are kind of dealing with, you know, that kind of complex challenges? 

Mike Harris I think from my letters that concern data that was granted to a system is having lots of different data. Sources support a large number of integrations in Geckoboard, are actually having data that I can put in disintegrations that note that he's saving the front-end. Well, that was very important. And I enjoyed the Barcroft. QL was said actually had a lot of control because you are in the middle of the application and.

I love the steam GraphQL because I could use that as a reference point and could go back and you could see all the mutations and the queries that were big money growth jobs. Oh, yes. I got a test for that one. Oh, I have to go to that one. OK. That's good. I've got to do. This was a really nasty night out for me as a tester sort of trade rather than having to wade through this will cut coterie person in one place.

And then once you got a basic cut you to build up the coverage to all those different queries and mutations are also criss-cross Pierro. So they're a great help as a test. This is why I wrote the article I wrote from you on it because we were caught early. At that point, tools like Postman didn't support GraphQL. They do now. So we have to do things for ourselves. It was so fun that it was successful. That's right. 

Jonathon Wright Yeah. I'd miss a bit. You know, I struggled when I first got when I started using things like Neo4j and GraphDB because, you know, it added an extra layer of complexity. You know, people just kind of got their head around. No sequel. And, you know, that kind of. Well, you know what? Now you've got this whole node relationship node which has multiple relationships that spiral out into these hugely complex, the very visual.

So it's a tester you know, the example we were working on was for Amazon. Right. So the relationship between data and category. So, you know, in the US they have XXL. We don't have an equivalent size, you know, these different sized mappings based on different locales, you know. Plus, you know, then there's the relationship between products and categories. Say, you know, a baby car, Check is in the category of automotive vehicles and also baby and child. Right. So you've got all these different relationships which, you know, you're testing.

Right. You kind of thinking, I've got to test all these different permutations. You know, I actually had an interview today with Hugh Price around test data and generate and test data for these kinds of systems, you know? How did you go about kind of we said lots of different input sources from integrations? But how do you control that? You know, they did you mock and stub those endpoints out so you could control the flow of data, or was it real, real data coming from these third-party applications? 

Mike Harris The first piece of GraphQL, for us, was actually on Lightweights Integrations, so I'm supporting a register to already being messy, it's different versions of them. So for me, there was no research in finding that boundary values on the different types of data. So that was interesting to them. So other tests. I had a colleague, friend, Jon Normington, who created some tests where he was creating lots of random values that were firing at the system and saying what results in these produced.

And that was really very helpful. Administration development because it could run trouble test cases and we could be found. Lots of interesting things happening. There is the test of the Chancer. I was writing to Keep the bounty values aware of that become part of our regression sites. So there were different approaches taken to get different results and to explore different parts of the application. 

Jonathon Wright Yeah, and I noticed you'd been using the GitHub explorer you mentioned about, you know, Postman not supporting the language languages search. You know, the great thing about Facebook bringing in new technology to is, you know, how do I get started and start writing some some some queries that actually are going to be tested as you would do, you know, back in the day of sequel and Oracle and that kind of, you know how did you get. How did you learn? I know, you know, the reference to really good Elizabeth Hendrickson for things like Explorer, you know, you've got, you know, very strong technical testing background. But, you know, how did you go often and learn this? Did you learn it online or did you read through the manuals or where was your source for getting up to speed with everything? 

Mike Harris I was GraphQL. There was this, there was some stuff online. But I was speaking to the developer's creative queries and mutations actually kept copies of some because actually, she hasn't run a mutation before. Can take a while to create some and actually became a reference point to that for us to use now.

So there no experimentation, doing some research, collaborating with my colleagues in this building up the Shirley shared library that we could use. So I had this one cause of problems. There it is. 

Jonathon Wright And how much of that was data? You know, as far as you know, you talked about the mutations. But obviously, you have to control your test data. Do you have to? Did you generate that? You know, where did you how did you create that and those input sources? 

Mike Harris So two types of to go to my colleague, Jon. He wrote some code, which is generating the random code that veterans data for myself. Said it was researched and looking for boundary values. There was one issue I found with our TV address that would allow a larger user, a particular type of numbers and graph here. And we had to handle that situation.

That's the sort of thing we look for as a test. So there were different approaches. So how we do all this research as I was just generating random data. 

Jonathon Wright And getting it sounds like you really kind of sharing collaborative kind of environment. 

And, you know, did you kind of document what you were doing on on like a Wiki or, you know, how did you share that information between the teams and the developers that you were working with? 

Mike Harris Yeah, we have a techtron as a Wiki, so we're very keen on using that to share information. Because it said it's a central core to the team. Everyone has access to it. Everyone can search for stories. That's where you start. I think that's extremely useful.

Jonathon Wright It's interesting, I guess because I went to my last conference, I went to Civil and talked about a new role as a data engineer, a test, and they'll create a new role every week. But, you know, I did find when I was going through and learning this stuff that actually, you know, there's a lot of debate. We had low today.

Scientists who were really keen to get the information do really cool stuff for their very academics coming out of university. But then, you know, getting the data into the right format, getting it in, you know, understanding the, you know, where, what you're testing with such a big, massive challenge for us. So we ended up with a data engineer who, you know, in our days, we may have called it, you know, let the ETL testing enhance transform the globe.

You know, part of it is we need to understand all the transformers, all the mutations that we're going to do, and exhaustively test the system with, you know, different data sets and different sizes. You know, I'm guessing when it comes to that front and Geckoboard dashboard, there's a whole stack of calculations going on there. And then pretty well they. Real-Time or do they the query running the GraphDB you out running every kind of few minutes to update the dashboards. 

Mike Harris That's different refresh rates and different types of widgets. Yet some of its roots on some of it is stored there for me is testing. It's important to note the ROI input corresponds with what happens is a deterministic system in that way. So actually that makes it that much easier.

But if there is a method in creating test data to get started. That means testing different boundaries on the issues that we're concerned about. 

So, yeah, that's some of the things I've done, actually, looking at where I could store that title online. I started using it just to keep some data in as well. I'm using so that's there are certain things results for the REST team. 

Jonathon Wright Yeah, we had it. We had a massive challenge. I did realize when you're loading the data into something like Neo4j, it's just a raw file. Right? And it's just a set of data. And, you know, part of it is obvious, if you do any mutations, you go to the date data set to change. Right. So we were obviously in a similar kind of day to rolling back.

You know, we had different processes for this because the data was so big. You know, we looked at things like virtual testing and management. So being able to replicate, you know, only change the delts is what changed during the mutations and allow us to run tests, you know, clear them down and then build the environment again, you know, from scratch.

Do you find that you're doing a lot more of that kind of continuous testing and deployment now that you're in, you know, as kind of a DevOps pipeline? 

Mike Harris I feel I haven't actually had to put down test. I think that further the regression tests, it's actually been good to get a set of data that can stay up, that can be used by the whole team that represents covers a lot of bases. 

And what I was looking for is of particular different types of numbers. And what I found was a fantastic table produced by NASA, not tickets for the different planets because it has different densities, gravities, whatever. So you get large numbers, small numbers, floating-point numbers, integers, negative numbers. I was able to use that as the basis of what I wanted to show, that there were some very large numbers that weren't included, solar systems, planets.

So I had to create some time. So I've got some Klingon planets with their names, which is particularly large numbers. This guy's a set of data that the whole development team could use. And if they didn't want the large numbers, then they could exclude the table. It's just tougher for the solar system so that that was the thing. I think it's a total contribution to the data that we used with yet another piece of code that Jon wrote actually was just random and it ran every time.

It was different every time. So that was the case of pulling anything down. It was just an automated process that ran every time. We don't run those tests that they were useful for the creation of the interface, but actually, now the interface is established. So the data set, which also is based on NASA's type of planets. That's actually what we're actually seeing. It's very useful. 

Jonathon Wright So that's super smart because the good thing is that you also know that you can go and test what, the database. Right. So you can go off and say, well, how far is the moon from this planet? And you'll get a number and you're built to Check that that number is right. And that was what we were finding is a real challenge with the traditional way of doing an obsession and trying to look for value.

And like you said, if you're putting in randomized data, it's been like Monte Carlo algorithm. You put the same data. You get different outputs each time. So part of it you should look for those mutations, all those patterns that you're expecting and then you're validating those, which pretty does make it really hard for things like automation.

Do you find that your being more exploratory with the book from a data engineer is kind of high on than you are creating, you know, repeatable automation assets, or is it a balance? 

Mike Harris I think this is a balance that I think that I find is are doing exploration or actually a tool to make that expression and find something, actually, it's broken. Can I control to make that when it comes back and it's fixed? I will. And that way you can build up your tests. I think that there's a mixture there. I think that's also what I find interesting about it is that we tend to think of exploratory testing is being done to you.

Right. Actually, do something with Henderson. We chose to in her book is in fact, you can use the same techniques on an API. So actually, if you look at the API, you look at your GraphQL query on mutation, it took certain inputs. It should give certain outputs that are tested. You can then work out what you would do with those inputs and what you expect to get back. So what did writing the tests in the graph show levels? That's how we're working.

I found that interesting. I quite enjoyed working with different inputs and outputs, different types of numbers. Say what kind of what comes out should get back. OK. I've got that test that's useful that covers these biases a little to make that one and that'll go into the regression test. So you can get coverage and be learning at the same time. 

Jonathon Wright That's very clever, I think. Did you find, you know, obviously collaborating quite closely with the development team, you know? Were you saving these kinds of these queries that you created as code in GitHub or, you know? Were they running as part of the build processing Jenkins or something? Well, how did the regression pack evolve? 

Mike Harris Of the regression, the pack ran out and three tails and circled. And certains integrated and built up over time and the story. Get help. So, yeah, that no says I process, which enabled us to move quickly with continuous deployments. That's great. It's good to see how quickly a team commits. 

Jonathon Wright And as far as you know, you talked about the fact that's your polling data set. Right. Depending on the type of widget you know. Was there any other kind of nonfunctional kind of challenges that you had, like the performance? You know, if it was trying to read the data from a source too often or, you know, what kind of challenges you come with, you know, from a non-functional perspective. 

Mike Harris Yeah. The issues at the convention. How quickly do you recover? How could it refresh? Well, so the thought was duplicating. 

We have got automated tests that run of production regularly. So please give us feedback on your run through a series of steps on the floor and see how quickly it happens. Yeah. So I actually thought that's an ongoing process that we have. Well, I'll say it's very important as a test or we know it doesn't look good. Just well, it's just got to work in a way that the customer finds acceptable and useful system. It's refreshing to particularize we get to create the right response to their actions quickly enough. 

Jonathon Wright So do you find you do things like contract testing and say that this is one of the things we were kind of doing because we had you know, we're injecting data into an API. REST API was then going on to Kafka the consumers and producers, depending on how we could see which you had, would take that data, stick it into Neo4j. It dueled the mutations and eventually someone at the front, the UI would do a GraphQL to see if the values change.

Right. But from the point where you rejected the data, which I guess is where the third party is providing some kind of data to the point where it appears on the screen, you know, do you do the kind of the end to end testing and measure that as an as a full contract? Or is a do you break it down into each one of the components and test them in isolation through build and CI? And then you do that end to end with a real, you know, real, real stream oh. How do you do that? 

Mike Harris We've got some integration engineers who are involved in that side of, I've written some tests of the performance of something just really looking at healthy forms and make sure that you continue to work as the consulate too.

But actually, the integration engineers will be doing the things that you suggest, looking at how each individual component works and actually gets our end to end process. So I'm looking forward to that. 

Jonathon Wright And as you're a customer that they have. Does it have a mobile app as well as a dashboard that they can put on a screen or what? What is typically the usage of your own customers? 

Mike Harris Well, we do have a mobile app as well. So you can look at seeing if we can share information on the farm catalyst they put out on the table and production at the. People are using the mobile app Trommel. So it's working on a Slack integration at the moment as well. So we'll be able to send your successful Check to your Slack channel. 

Jonathon Wright That's very clever. So, you know, part of, I guess it's ChatOps that you're talking about here is, you know, you potentially your boss flash widget can then say, you know, show me today, you know, all the orders that you've made today and then old post that information into Slack. And of course, you could probably explore tech you could share internally from an executive perspective. I know you mentioned there that there's been a switch between the kind of, you know, obviously coexisting.

You know, I love organizations. I'm always fascinated when I go into an organization. We've got a dashboard in the back. Right. And I worked for a pizza company once based near where you're talking in Hertfordshire Bedford, so kind of that direction and they used to have how many pizzas coming through. Pizza orders. And I always use the word lookup. Go and look at it and go, oh this is really interesting.

It was things like they spent a huge amount of time testing the x box version of the pizza ordering platform. But only two orders would ever come through that. You know, part of the other was the mobile was huge. And then, you know, they got in the states, you could order a pizza on Twitter, you could order a pizza, through the TV. You know, these all these different channels that they're doing. So I'm guessing you're kind of your custom.

That's the kind of stuff that your customers have in. You know, the accuracy is really important because there's a trust in your kind of transparency of data. So do you find that you add, you know, like the end of month or end of day kind of calculations like you would do with an overnight batch to kind of say yes or all the data that we've shown is correct and was accurate? Or is that kind of not necessary because you've already tested it? 

Mike Harris

We got a way to test run Check that the numbers you see in the front end are correct for detector inputs. So a widget that people used to display their data can use a whole range of different day insights or day periods. So we would need to test those as well. So there is into sort of a patch at the end of the day, a small data is there and wants to see it today.

So last week, last month, last quarter, or some type that you particularly data that you're interested in. So you need to be able to make sure you've got the right numbers and achieve at officialized in the way that you choose from. She was that she's about products, pretty much make making communicating visually. 

Jonathon Wright Yes. Yes. It's so interesting because, you know, you might be looking at a period of time and that could mean something to the business. You know, that could be, you know, orders of pizzas from, you know, seven when they say open. This is, what, in the afternoon. If there are more pizzas getting noted or evenings or after nine o'clock on a Friday, you know all sorts of interesting data that the customer needs to make decisions about how they sell their product.

And, you know, I find this really fascinating, this kind of testing and production idea of, you know, having synthetic tests running to Check things are working at Check that, you know, data that's being, you know, is correct. You know, do you have much involvement when it comes to this kind of DevOps lifecycle off the operational side of things about, you know, usage patterns of your products, you know, how people are going through, how they're using the different features, which widgets with the most, you know, sought-after, you know, is that kind of stuff that you look at as well.

Mike Harris Which widgets? REST is definitely which integration to most students, which can specialization several things that we're interested in because well, and we were largely ignored the most heavily dos and don'ts in designing tests that are automated for ensuring that those things continue to work as what you expect them to. Yeah, absolutely. 

Jonathon Wright That's pretty cool. So, you know, take take it a step back from Gecko at the moment. I know you do loads of voluntary work. You know, you're doing stuff. You help the British Computer Society specialist interest group for software testing. What do you what kind of what's your role there and what kind of things do you do outside of your day role of Tester? 

Mike Harris I guess the second stuff, 50 programs, Gates' program Check between us. We're involved in networking events, the six-foot. So we are within a couple of weeks ago with Lisa Chrispin pulping. Com and in-house. We're now working on the next event, actually got a committee meeting tomorrow night, was working out a little something at the morning meeting.

Our equipment out there since they hadn't planned to be a whole face to face up to that now. So we're having to look at how we can change the trends to make sure that we continue to have Web analysis to engage people. 

Jonathon Wright Because I guess it's really important to be able to get to. Are you finding with only the stuff that you're doing for work? But you know, the stuff you're doing for your voluntary status stuff as well, is that your use a lot more Zoom, a lot more kind of, you know, new technology for streaming, you know? Is that something that you've seen a change over that, say, the last day, five days? 

Mike Harris I think technology hasn't changed. I think that we've changed, actually. I think that's one of the ways that we use this stuff, because it's every day, it's how we work. It's a long term, actually. You know, you're looking to see that person in the office this week. So any conversations you want to have has to go through Slack, or Zoom, or Teeple. What if I told you using? So I think that we've changed in that way. I was on a call with someone with a set of companies.

They worked to they used to have a culture of people turning off the video cameras. But actually, now everyone's at home, everyone's turned them on because they want to see everybody. And I think they were all sort of hungry for the social context in that way. So I think that maybe you think it's different and the way that we're blending work and other life is different.

But one of I set up shop Chromium test automation and by radishes started to come through in the go. So actually this led does work and profit washing. So. So it's new. I think lots of organizations are getting it. So for the first time, I'm involved in a cricket club on a voluntary coach. This week we're starting coaching sessions like this in which you can be running the kids from rooms. 

Jonathon Wright Yeah. I noticed your daughter's part of the cricket club. So, you know, is it. Say, what kind of things are you going to do? You get to set that up in people's gardens so you can work on that. That that technique, you know. Do you know this is a quite exciting new medium for coaching, isn't it? 

Mike Harris Yeah. It's been some interesting things that people have done that of innovation, innovative ways of working with kids at home. So the different routines you can have for batting or bowling or you're catching for fielding. And these sessions are going to start this weekend to be bound by the strictures cricket because we've got a challenge running at the moment.

Yesterday you had to say how many times you compensable and the fact that you've got a point. Three fronts today, up to seven meters apart in the baton, in the hole or whatever, and see how quickly you can run the change, if you want, between the wickets. And so there's a whole range of things that people can do it. It's great seeing people being imaginative and actually learning new technologies. 

Jonathon Wright And that and the fun aspect, if it comes back to the. And I think, you know, I was speaking to Bob Cruz today and, you know, he was doing lots of funny things like wearing a dinosaur or how his video conferences, you know, they make, you know, their employees would get a Uber Eats voucher and all having lunch together as you know, you would go down to the canteen, you know, they were doing, which I'm doing tonight, which is why I've got this in the background that day, the

Performance Advisory Council, you know, and yeah, we have a beer. And do you have a chat kind of all together in a social as you would do on a meetup? You know, we've seen these changes happen. And yes, some of them will be a great new platform to get more people involved and things like cricket, you know. I reckon this would be a big trip. I know you're a keen comfort speaker where people are going to be telling the lessons that I learned from COVID in a positive way of how that's changed them and that digital skills that they've let out of the back of it. You know, do see that as something that is a big positive that can come out of this. 

Mike Harris I think it's going to be really interesting, the world before it's going to be very different. I think how we collaborate is going to change. I think looking at how you collaborate from one way, you're not going to see people from week to week and the whole company is like that. There's going to be lots of issues that are going to be different. And I think that's a lot of things to share lots of lessons from with how you engage with people on Slack communication. So we had a fancy dress that spent a few meetings on Friday.

We have a cross club study on Slack. We started out the cooking channels, so it's just like having the golfing channel tomorrow. And so I think that's really good to engage people and have a conversation and those little conversations around where you interact, I think they're very important. And so that's good. When you start to stand up and your body facilitated stand out this morning, actually small conversations as people start, I think, are actually very important as well. Yeah, I think there are lots of lessons. I think that sir, we're still learning those still adapting because situations are dynamic. And sometimes in people's families, this is not going to happen slowly. In space and support. 

Jonathon Wright This fantastic advice and, you know, so you've given us some great tips, you know, for those people who are listening to the podcast. You know, what's the best way to kind of reach out to you or get in touch with you or, you know, find to be an old material, all that material, you know, give you give the listeners a bit of how to best get in touch with you. 

Mike Harris OK. Oh, it's so great to get in touch is made with Twitter every day. And that would be really good. I have got a British Computer Society email address, which is mikeharris@bcs.org.uk. People could contact me there. Twitter is probably the easiest way. My handle is @TestAndAnalysis. 

Jonathon Wright Yeah, I spotted that. That is great. Definitely a great Twitter handle. And, you know, I'm looking forward to your next blog. Maybe not GraphQL. You've done that now. But, you know, maybe something late to the lessons from cricket and at combines with growing your own testing. Oh, so fit like that and be really excited to say so. Thanks so much, Mike. 

I really look forward to working with you at the BCS and also seeing some of these online conferences that you're going to be putting together. 

Mike Harris Okay. Thank you very much.