The Future of Software Test Engineers and Codeless Tools

A few months ago I was chatting with Evgeny Kim about some of the reservations I had while exploring a new codeless test automation tool. He was also exploring some codeless tool options and so he invited me onto his podcast to talk about it. We chatted about a wide range of things such as challenges faced by software test engineers, the role of codeless tools and hiring problems.

It was a fun podcast with some interesting topics so I had the video translated. Watch the video or read the transcript whichever you prefer. (Sorry in advance for any mistakes.)

Transcript:

EVGENY KIM: All right, guys. Thank you for joining for today’s podcast session. We have a guest, Chris Kenst. He’s a QA Engineering Manager at a company, Promenade Group. Also, he’s a president of Association for Software Testing. And we’re going to talk today about the future of software test engineers. Thank you, Chris. Hi.

CHRIS KENST: Hey, thanks for having me.

EVGENY KIM: Chris, let’s talk about software testing in general, but before that can you tell a little bit about yourself?

CHRIS KENST: So I’ve basically been in software testing and quality for over 15 years. I’ve worked for large and small companies. I mainly like working for startups and mostly as an individual contributor. More recently, I’ve moved into a management role, so hiring people, managing people, trying to manage the test process across a bunch of different business, all kinds of exciting things. So, I’ve kind of run the gamut from being an automation engineer solely to mostly exploratory testing, kind of all across.

EVGENY KIM: Awesome. And can you tell a little bit about software test automation, what you’ve done before and where are you right now?

CHRIS KENST: Yeah. Depending on the company that I’ve worked at, I’ve built numerous tools. Most of the stuff that I’ve built had been just like some old scripts, but just to kind of help me, like, I don’t know, design a test or compare something, that kind of thing. I’ve also, depending on the company that I’ve been at, I’ve worked for Ruby shops. So, I’ve built some automation in Ruby. I’m currently building in JavaScripts right now. I’ve more recently decided to start, this is kind of interesting, so as an automation engineer and as a QA engineer, you try and build whatever tools that you need to solve whatever problems you have.

I think more recently, and this is new to me because I’ve mostly been in small companies where you kind of just use open source tools wherever you can or you write a script to do something. Recently, as I’ve become a manager, I started looking at different tools, but also buying tools. And this may seem like really obvious but just kind at from an individual contributor to a manager role, kind of taking a step back and going like, I can actually now make a decision on whether to buy a tool or build a tool.

At Promenade Group, we actually run a few different tools for automation, I’m talking about UI automation now. We have WebdriverIO, which is JavaScript UI automation based on Selenium. We use Cypress, which again is UI automation but a different architecture. And then, we’ve started using a new tool called Reflect.run, it’s codeless automation tool, which makes it easier to just basically just record UI steps and that kind of thing. So, now I’m kind of attempting to broaden and locate different options for our toolsets.

EVGENY KIM: Can you tell a little bit about each tool, for which purposes you guys are using, especially, I’m interested in WebdriverIO, Cypress, they’re kind of similar, right?

CHRIS KENST: They’re similar but different. I think I originally started using WebdriverIO when I joined the company over three years ago and again, it was just to automate some checkout flows. We’re an e-commerce company. We have a bunch of different businesses now, but I wanted something that in theory I could run multiple browsers on, browsers being kind of the important thing. I want to be able to have Chrome and Safari and a bunch of different browsers that we have customers on. So, just to kind of do those checkout flows.

Over time, we ended up building React application. I wanted something that was a bit easier for my developers to use. Something that would perhaps be a bit more integrated into the code base. So, our main applications are in Magento and PHP, but I still built a UI automation framework out of JavaScript because, one, I didn’t want to build it in PHP and two, JavaScript was our next most used language. And so, when the React application was being built out, it was like, well, you know, talking to my developers like what would you like to do. And then, we kind of ended up settling on Cypress because we didn’t really care about having different browsers and the availability of that. I particularly liked how well they kind of integrate into a codebase, especially, a JavaScript codebase and how you can listen to different requests and be able to build things kind of internally, and it felt like a really seamless way to just kind of drop a UI automation into a JavaScript code base. That was kind of the genus besides that.

And then, in terms of buying now a codeless tool, this is an experiment. I’m trying to figure out now that I have essentially three different platforms, I’m trying to figure out what I like about one, don’t like about the other and throwing a third tool in that is basically really easily record stuff and can be triggered with an API request and it has built-in recording and all that kind of thing. So, the only kind of connecting the dots between these things is that just trying to find the right tool for the right job and also kind of experimenting, running my own internal test about what I like better about one versus the other.

EVGENY KIM: Let’s talk about codeless test automation tools because recently I also got partnered with a company, TestRigger, they also do code estimation tools. And you know, I think technologies right now getting quite impressive compared to previous generation of test automation tools and like play and record and all kind of recorders, right, Chrome extensions. But at this moment, what I’m seeing, I mean, let’s take a step back, before in order to write any test estimation, you should at least have some basic knowledge of programming languages, you need to know pretty good functionality of Selenium Webdriver and it depends on how you’re going to write also API test or not, you should also know API and HTP protocols, right. Right now, to write just functional testing you basically can sign up for those code estimation tools and start writing the test right away, right? So, what do you think about it?

CHRIS KENST: Yes. So there’s this paradox because as a test engineer and an automation engineer, you are going to get paid more money if you learn how to code and you can do all those things, right, like that’s just the simple fact. And if you can use an off-the-shelf tool like TestRigger or Reflect or TestIM or any of those and you put that on your resume; you’re not going to get paid more money. People aren’t going to be like, “Oh, you know how to automate,” even though essentially you’re solving the same problem. Not to the same degree because there’s obviously differences if like you’re saying, you can combine API and UI test if you know how to code and you understand how to structure a request and that kind of thing, but it’s kind of similar.

There’s this paradox going on where you’re going like, “Okay, great, the tools are now getting better and there are more of them and they are, let’s call it a bit easier to maintain because they’ve added some, it’s called machine learning to identify hundreds of locators so that they don’t break so easily.” And yet now, you’re sitting there going like I can reasonably pay hundreds of dollars a month for a service that will help me automate, and different people can now automate it, rather than someone who needs a programming language or has to use a programming language. Plus, you don’t have to worry about them updating the framework and breaking all your tests and all these kinds of things.

So, there’s definitely a bit of a challenge if you’re looking at this. I’m even challenged myself because I’m like, I used to write, I still have to write some code because I have those frameworks that need code, but I also have codeless. So, there are certainly tradeoffs depending on what your goal is.

EVGENY KIM: Definitely. I mean from engineer’s standpoint, I think it’s got to be quite a challenge, right, because business, obviously, they want to solve their problem by just, you know, they don’t really care what is your approach will be to test, they just need to be tested, right. They just need to release the product as soon as possible with less bugs. So, from business, I would say, it looks like codeless automation would be probably the future, but what for test engineers they should do?

CHRIS KENST: Yeah. It’s a challenge. As a new manager, let’s say, I’m running into this all the time where I’m managing a group of people across a bunch of different businesses, and I don’t even get to see, I’m trying to stay hands on and see what they’re actually doing so that I can coach them on how to get better, but I know them like most managers can’t do that. And at the same time, I’m also looking at the tooling, well, you’re right like a business is probably going like is this providing value, is this tool providing value, and if so, what are the ways it’s going to provide value. And so, yeah, I mean, to a business like they’re probably the same. Assuming that the maintenance costs are kind of comparable, they’re not really going to care which one you’re going to use.

This is why this is a paradox, like what should a QA engineer do. You still need to learn how to code more than likely if you want to be able to like get paid, right, like that’s kind of the thing. But hopefully, you’re also open to just whatever the company needs to do to solve its problems. So, if you’re going into situation, if you’re joining a company and they have an existing tool, yeah, you probably should learn how to use it, but also don’t give up on figuring out how to code because some of these codeless tools will also not be able to do some things that you might need to build small scripts for or figure out to kind of work around on your own. So, even with any of the kind of the off-the-shelf tools you’ll probably still going to need some ability to manage APIs and webhooks and those kinds of things. So, even if you are a QA engineer and you’re trying to figure out which way to go, you’re still, it’s always going to be a good fall back to know how to code, at least to some degree. Wouldn’t you say that?

EVGENY KIM: Yeah, I would totally agree. You know, the trend is like more like five, ten years ago when manual testing jobs, they’re moved to offshore, right? And I think it’s very similar that those type of jobs which you can replace with code estimation, with anything which can be replaced, right, I think this type of testing, like end to end testing which can be done by even business analysts or product managers or even customer service, they can write basic end to end tests. And this type of testing will probably be gone, what do you think?

CHRIS KENST: Yes. Just because you could let a product manager, business analyst record a test, that doesn’t mean they should because more than likely, ultimately, you’re still going to have to try and think about good test design principles. And so, depending on what it is that you’re doing you should change and refactor your tests to be effective.

A good test is an effective test. Regardless, if you automate or you don’t automate, the time when it becomes cheaper to offshore that labor is when that labor is deemed to be machinable or like repeatable, right. So, if you are a QA engineer and you’re writing test cases, and by test cases I mean you’re pretending to be a machine by writing a bunch of steps, yes, somebody is just going to offshore that because you’re not showing that you are having much value, because you’re just repeating a bunch of steps anyways. And then, sure, have somebody automate it and then, give it to somebody, like basically make that the cheapest you can.

That’s what’s going to happen if you kind of make it to be similar to a manufacturing output; yes, somebody is just going to go let’s find the cheapest. If you are not doing that and instead you are focused on risk and you’re focused on risk assessment and you are showing that you have skills related to performing your job, you’re going to have a much easier time with trying to, a, get paid more and b, not worried about your job shipped off, right. So, any takeaway from this is you still need to be skilled in the job that you are doing. And if you are skilled in the job that you are doing, you’re probably going to be okay in terms of somebody just going, I’m going to offshore this or I’m going to give this to a product manager to do, right. There’s a whole lot of skill in there that hopefully people see, can recognize and reward you for.

EVGENY KIM: I would say that now I think QA engineer should be more focused on more complex stuff, right? So, coding, definitely, this is must have in order to keep up, right, in order to, I would say, to be relevant to the market.

CHRIS KENST: Yeah, just to be knowledgeable. Part of it goes with I think part of the learning to code or at least read code is just flexibility. And so, I think what happens is if you are a test engineer and all you do is you write scripts but you just write them for other people, then your flexibility in what you can work on I think is deemed to be very narrow. And so, maybe we could talk about it in terms of being like a generalist versus a specialist, right. On some level, you do want to have skills that are general and applicable around the company and to all kinds of different work, and coding actually is a very general skill that you can apply to anything, right. And so, as a test engineer, anybody in testing, you want to have a bunch of general skills that you can apply across the org.

EVGENY KIM: What are the challenges you see right now with the hiring?

CHRIS KENST: There are so many challenges with hiring right now. One of them is just finding good people who are available. And I mean there’s always a challenge of finding good people, but now with the pandemic, companies are kind of changing and kind of not changing. So, one of the upsides is there’s just this bigger talent pool, which is great, my first few hires were only in California and now, I’m hiring just across United States, and that’s the same with our company, at least for engineers, testers, product people, that kind of thing. But you still have the challenge of at least working in a startup, I’m looking for particular people, people that show value by understanding risks, understanding test design, moving quickly, and aren’t stuck in the traditional test management of, “I write test cases and I only know what a test case is and that’s all that I do.” So, there’s, there’s that challenge. Plus, you have the challenge of like, well, we need to move fast and we do kind of want to automate stuff.

I’m always looking for people who understand testing really well, but then can, in some cases but not in all cases, can actually code. So, a given for any of my projects is you have to understand API testing and you have to understand web technologies, HTPs, status codes, you need to be able to run the gamut of being technical, you need to understand DevTools and Postman and you can’t just know what a 200 and a 400, like you actually need to understand what a structure of an object looks like. But once you kind of get past those technical things it all actually comes down to just being like a pretty good engineer and like can you reverse out problems, can you make sense of things, can you adapt to rapid transformations from, you know, you’ll have a project that just constantly changes.

There’s always a challenge of finding good people, and especially in the testing space, finding somebody who just doesn’t think that testing is a bunch of test cases rules out like 80% or more of the people that I’ve interviewed in the past, which is both sad and at the same time really thrilling when you finally come across somebody that understands what testing is supposed to be. Because frankly, you have to understand what testing is to understand what are you going to automate, because I don’t want somebody that’s just like, it always scares when I hear of companies that are like, “Oh, yeah, we have a bunch of manual testers and they write test cases and then, we hand them over to automation engineers or SDETs and they automate them.” And I’m like, that seems like you’re just duplicating work, like, hmm, why would you do that? In the example of codeless tools, just give the manual testers a codeless tool and ask them to do that and then, actually don’t have them write the manual test cases just have them record them, and then, you could probably save money. So, that’s a little bit of a rant at the end there, but, yes, so there’s lots of challenges for hiring just kind of across the board, mostly just finding good people though who understand what testing is.

EVGENY KIM: Okay. You also mentioned that you don’t really require for the hiring to know the coding, right, or to some sort of degree of knowledge.

CHRIS KENST: The challenge of interviewing people and understanding what it is you want is ultimately like, so I used to ask people coding questions and I used to ask people kind of a wide gamut of things and then, just hundreds of interviews have taught me that you can ask all these things and then, kind of grade people based on what you get, but if the person that I’m hiring is not going to spend a good deal of time coding when they actually get in the job, which the people that I’ve hired so far have not done that because they’re leads, they’re dealing with constantly changing requirements, most of our businesses are very young, and the goal is just to kind of build out minimum viable product and kind of get that on the roadmap, then the actual amount of coding they would do is less than 1% of their job. And so, it just didn’t make sense for me to ask people and kind of filter people based on coding if in the first year they’re not going to do it. So, part of it is just figuring out what you need, right, and then, focusing on those things rather than a wish list of, you know, I want somebody who’s a senior software engineer, but then I want them to be a software tester and then not to code anything for a year, that sounds horrible, you know. It’s terrible setting expectations. So, yeah, in my case, that’s not the way I want.

EVGENY KIM: Got it. So you are looking first at the skills as a tester, I would say, right?

CHRIS KENST: Yeah. I mean the challenge of hiring is you have to ultimately, and I’ve consulted with other companies on this when they start hiring in hiring software testers and I’m like, “What are you going to have them do for the first 6 to 12 months, and then, work your way backwards on what those skills would be to accomplish that.” And when you do that, you start going, oh, right. So, all of those wish lists and things that you have, like it would be great if this person knew this particular tool, like are you going to use that tool? No. No, not really, but it’d be great if they knew it; yeah, but like would you not hire them because you didn’t have it? So, you have to focus yourself on what it is, the skills that you need and then, that’s kind of how you hire. That’s my suggestion, not everybody does that, unfortunately.

EVGENY KIM: Yeah, I’m 100%, I mean, if you’re not going to use it why are you asking that, right? All right, but tell me like a little bit about, because I’m very curious, so you’re not setting expectation from, I mean, not hard expectation from engineers to know coding, but at the same time in your team you’re already using WebdriverIO and Cypress, right? Who’s handling this? That would be the first one question. And the second would be like how you guys got onboard the whole team for that?

CHRIS KENST: Yes. So who’s handling the automation? Right now, it’s mostly me and my developers. So, on the React side, most of my developers are building out the test for that, which is actually great. If I had to choose between having myself or one of the tests engineers build out the automated test or my developers, I would have my developers. If it was up to me, I would have my developers writing more of the automated tests. Both, obviously, unit level tests, API and UI tests, I would still have my developers write it. So, I think you and I were talking about this, there’s the book Accelerate by Nicole Forsgren, Jez Humble and Gene Kim and they’re based on a DevOps reports that they do every year. Nicole Forsgren and others went out and they surveyed thousands of companies and look for traits of high performing companies. And one of the traits of high performing companies is that devs own all test automation. And if you think about it, it kind of makes sense, right. If you’re building an application and you’re constantly changing that application devs are already going to be updating unit tests that you can also update any of the UI tests.

So, my ultimate goal is kind of trying to help at least internally empower my developers to be able to update all our automation. That’s not the case right now, but that is the ultimate goal, which again it kind of conflicts with what we are talking about earlier because I was like, well, I’m trying a codeless tool. Will I be able to get my developers to use a codeless tool? Maybe, but I don’t know if that’s correlated to success because you would do that after everything is done, right. So, I have this conflicting thoughts in my head about like how well is this going to work, because, yeah, that’s great I have this codeless tool solution that I can use and it’s pretty easy and I can give it to somebody who doesn’t know how to code. But at the same time, I would prefer to give a tool to somebody who knows how to code because the feedback loop is much smaller and they can build things out the way they need to be able to test the things they need to. So, there are all kinds of ways to skin this cat, but what I’m trying to figure out now is where it makes sense and who I have resources that can build out tests. The ultimate goal is to just kind of build out more tests and make sure that we have a useful feedback loop, and I care less about who does the actual work, whether it’s the testers or the developers themselves. So, that’s a very inconclusive answer is kind of where I’m at right now. What are your thoughts on this too, right?

EVGENY KIM: Honestly, I think I already said that, but I mean codeless tool will be used by manual testers, right. In general, the quality is the responsibility of the whole team. I mean I’ve been in many places and most of them they treated the whole team as the one unit who is responsible for the quality, it’s not just like the tester. But, yeah, I’ve been in companies where like, okay, testers missed the bug, but that’s not the case. Anyway, so what I’m thinking that, yeah, we would like developers test their code and to write and do testing as well, right. But in reality, they don’t really like it, I would say.

CHRIS KENST: Yeah. And like the thing is like why don’t they really like it. I mean this might have been a while ago, but I’m just curious, do you know why your developers, maybe you’re not speaking from experience, but do you know why they didn’t like it, was it the tooling around it?

EVGENY KIM: For unit test, no problem. For end to end test, I mean they think that it’s a tester’s job.

CHRIS KENST: Oh, interesting, okay.

EVGENY KIM: You see?

CHRIS KENST: Yeah. So it’s a mentality thing more than a tooling thing.

EVGENY KIM: I think it depends on the companies, right. Again, like in some startups, developers, they should test, they should write test. In a little bit bigger companies where it’s all divided, yeah, I agree, it’s a mentality thing, it’s not a tooling thing, right. I mean if that would be just a tooling thing that could be easily resolved.

CHRIS KENST: Yeah and I think you’re right, it’s cultural and it’s a mentality thing, like it’s based on the organization and how they delineate who like the rules, right. And so I think in larger companies, you’re right, you tend to have people that are given responsibility of like certain segments of, you know, like you have this whole team who is supposed to be responsible for quality but they don’t really control quality, they just receive it, right. And you’re right, and I could see this being even more complicated if you have offshore development, right, where you’re like, we have offshore development, the code they produce is okay, I wouldn’t want them writing the desks, so build a whole suite of tests that catch how bad their code is. And you’re like, well, you can solve both problems by just fixing the original problem, but companies don’t always do that, they kind of layer on Band-Aids across of their problems. Yeah, so there’s no silver bullet or one thing that works well across all the companies.

EVGENY KIM: You know what, also, one of my previous clients, they have a QA engineers, they have QA engineering department and they also have a customer service or customer representatives. So, their customer service in their job description, in their job requirements was also knowledge of QA and they actually performed some type of testings, you know, this is kind of interesting thing. And also, I was kind of representing, confident that they would like that their customer representative or customer service would have write those type of basic end to end testing, you know. They would write a ticket, like this functionality is not working, and then, customer representative would write, okay, this is what it’s not working and send it back to where it should be doing, logging the file and then, engineers would be picking it up.

CHRIS KENST: And there’s a couple of things to kind of think about that, right, where you have on the one hand that like where a cord tool allows you to branch out your stakeholders from just be on the engineering team, right. So, like there’s a downside to letting customer service or asking customer service to test because that is not the thing they know how to get skilled on and so it’s going to be very shallow stuff. But on the other hand, it might be like really customer impacting stuff where they’re like, listen, like 80% of my complaints are around these things, so I would automate those. That’s not a bad idea, even if the test ends up not well-written, like I wouldn’t want to have to debug that test, but maybe that’s useful for them, right.

EVGENY KIM: Yeah, interesting.

CHRIS KENST: Maybe it allows like a broader set of people that do some degree of testing and broaden our stakeholders a bit so that more people feel empowered to automate some forms of feedback. I think that would be fine. I just don’t know how that plays out for engineering and engineering productivity.

EVGENY KIM: Also, I mean, in this case, I think that people who is interacting with customers, they actually know better their behavior, right, because QA engineers, they usually test, what, they usually test against the requirements, against the user stores.

CHRIS KENST: Yeah. And this is the whole push behind data and observability and all those things, it’s to remind testers that they actually don’t know how the customers interact with the application, right? So if you think that you’re testing some requirements, remember, written requirements are a very small part of what you’re supposed to test, you’re supposed to test also all these unwritten requirements. And unless you’re using data to actually understand what your customers are doing in real life, then there’s probably this big disparity between what you’re doing and what you should be doing.

EVGENY KIM: Yeah, definitely, 100%. All right, Chris, we shifted a little bit off the topic, but I think that’s great talk. Let’s summarize about the future of software test engineers. What software engineers should be focused on based on what we have right now, right, based on those codeless automation tools and so on?

CHRIS KENST: Well, we just talked about a bunch of things, right? So, one of them is, regardless, if you’re going to need it for the current job or the next job, you still want to learn how to code and you want to be code literate. You want to be able to, you know, we just touched on a little bit, like you should understand some data and what is going on in your organization so that you can help test better. And I’d see those becoming just more and more important. If you haven’t been looking at logs and you’re not looking at analytics data and you’re just making wild guesses about what you should test, you’re probably not doing a very good job. And that mean that’ll be your fault, it may be an organizational problem, but that’s only going to become more and more important, like everybody is gathering more and more data, everyone is trying to build in-house collections, data likes, that kind of thing to be able to better understand what’s going on. So, the world is only going to get more and more technical. What we talked about is there are going to be some applications where it’s going to be easier to buy stuff off the shelf. I don’t think test engineers should count on that, but if it’s available, they should use it. So, you’re always going to be better as a test engineer if you build your own skills and then, if something happens, you’re dropping your lamp, that’s really easy, like the things we’ve been talking about, great, that’s gravy, but you’re still better off learning as many skills as you can to become like the best engineer you can.

EVGENY KIM: Yeah, absolutely. Don’t stay on the same path, you just move on and learn as much as you can.

CHRIS KENST: Yeah, always be learning.

EVGENY KIM: All right. Thank you very much.

CHRIS KENST: Thanks, Evgeny. This is great. Thanks for having me.
EVGENY KIM: It was awesome talk. Thank you and I’ll see you later.
CHRIS KENST: All right, and I’ll see you later.
EVGENY KIM: Bye.