QAPaaS update: Continuous Integration and Continuous Build
QAPaaS update: Continuous Integration and Continuous Build
Hello, good morning. Good afternoon. Good evening. I’m an esteemed and important boy. I’m a professional services here at Sir Boy. With me as always, it’s Sean Devlin, party on Sean. Party on Steve. Never gets old. And today we are going to be doing a webinar describing Sir Boy’s QA pause, which stands for quality assurance platform as a service. This is something we’ve been working out for quite a while. And I believe this is the first time that we’re revealing it in a webinar. So I am going to hand it over to Sean who understands much more about this than I do. And as always, if you have questions, please, please enter them as we go along in the questions panel and we will do our best to answer them as they come up or at the end of the webinar. So take it away, Sean. Thank you, Steve. Actually, we launched the QA pause late last year. And we did have a webinar to announce it. However, because a quality assurance platform as a service is a difficult thing to visualize. I think that as time went on, a lot of the community kind of forgot about it and maybe doesn’t know what it is or if it’s for them or how it might help. So I wanted to do another webinar that sort of really just shows it and most of the webinars in this series are very demo focused. So I went to our DevOps people that work on the QA pause and said, give me something that I can that I can show that I can demo so that people people can understand and visualize what it does for them. So that’s what I hope to achieve in this webinar today. We just do a few short demos. One to demonstrate continuous building. One to demonstrate how we make UI testing easier. And then also a bit about what sort of analytics are available. Then we’ll do of course, after a short demo, we’ll do some overview of slides to kind of fill in the gaps and take time for questions. All right. The first thing that I want to show is I’m going to my lovely command prompt here to show you how I can run an automated UI test. So I’m going to run a command here. This is on my local machine. And you can see that it’s starting up Chrome. You’ll see some clicking around here, but I am not doing the clicking. This is being automatically driven by a script. And it’s gone to the demo. Serveway.com website. We’re testing out the cryptography page. Now moving on to the example searching, searching for all orders for coffee and Germany, updating the sales rep. Again, this is all scripted. I’m not doing any clicking. We go in and test some other components here like the calendar field, the password field. And then shortly after this, the browser will close because the script is ended. You can see that it terminates in the terminal there. So let’s take a look at the source code of that. I forgot one thing. Before I showed the UI testing, I want to start off something for continuous build because it takes some time to build so that it will be available at the end of the demo. Hopefully. So park that thought for just a minute. I come into the serveway IDE. I have loaded here the project for the sample gallery. I’m going to put one small label on the form. I’m going to type hello. Maybe I should make it white to show up. I’m being lazy here and not using CSS like I should. I meant to do foreground color. Well, whatever. I’m going to save that. I’m going to go over to my git client. And in sample gallery, I’m going to stage this and then commit it and give it a comment. So now I’m committing and pushing this to a repository. And it’s going to be kicking off an automatic build because of the change. Skip that. Okay. So that’s committed. In a few minutes in the serveway QA posts in the hosted environment, it’s going to begin building my project because I made a change and I committed it. So pardon the distraction. But now we go back to the UI testing. So let’s take a look under the hood at the UI testing script. You saw that it was driving the web browser, clicking on buttons and entering text in the field. So what we’ve done is we’ve created a sort of macro language to allow someone to very easily script out what they could do in a test flow. So you can see in the comments here that we go to the website. We test the cryptography and that the language is very sort of natural language based. So it’s a lot of when and then statements. Down at the very bottom, you can see that there’s some of the substitutions that we put in there. What I’m going to do is change one of the substitutions from coffee to or from Germany to USA. So we would be looking for orders of coffee in the USA instead of Germany. And I’m going to save that. And I’m going to go back to my terminal here and run it. So again, it kicks off Chrome and starts walking through the script. Of course, we go past the cryptography sample to get to the searching where we made the small change. So now you can see there it is coffee USA. We get a bit of a different result. And the test continues. Well, that continues to run. I’m going to show you some output from the test because not only are we telling the browser what to do, but we can also verify that certain certain states have been reached. So I’m going to open up a report here in the ID or in the browser. And you can see that it’s the E2E result report. And it was built a few seconds ago. And I can expand one of the tests here. And I can see all the steps. And all of the steps have passed, including wanting to validate that the data that I put in the password field was what I had put in. So let’s deliberately go in and break that validation. So instead of substituting that value back in, I’ll just put in a dummy value and save this. And now I go back to the terminal and rerun the test. So the test that I’m doing here, everything is running local, although it is pointing to remote site. And this is a way to quickly build and test your flows. However, most of our customers are doing this in an automated sense so that they write the tests and write the flows. And once they’ve tested them and they like them, they commit them under revision control just like source code. And then every time a project builds that the test is run. Okay, now there’s going to be a little bit of a pause because there’s a timeout period when there’s a validation and when the validation fails. So we’ll see that once it fails, it’ll the process will be killed and we’ll go and we’ll check the report. In the meantime, let’s check on the commit that I did to see if the process is building. I’m going to go to my Jenkins dashboard. And you can see that it’s running, it’s running right now. So the commit that I had made earlier where I dragged a little label on and changed the color. That’s still building on the QA past server. I think our, yeah, our flow is done. So I’m going to reload this report. And you can see that there was a failed test here when I expand it and scroll down. I get the failure here on the verification of the password field with the value that I had put in. And you can see that when there’s a failed test that even has a screenshot of the browser during that state. So this is just kind of a small example. Obviously, there will be lots of tests. And so you get really a big list here in report. And again, this could be automated. I’m just running everything locally here. I’m going to check on the build one more time. I think this will take a few minutes to build the last time I ran it took about 10 minutes, I think. So in the meantime, what I’d like to do is go do some overview and then we’ll check on it at the end. Before I get into sort of the nuts and bolts of this, maybe I should back up and talk about why server is offering the QA past. We did a lot of research. And we found that a lot of our customers, small to mid-size software vendors and IT departments and corporates, we’re really spending a large amount of their time on DevOps, on maintaining source code, on just coordinating development between developers, managing deployments, very little automated tests, very little quality assurance other than kind of responding to bugs. So despite a large amount of effort on sort of DevOps, we were seeing still problems persisting with quality and also really slow release cycles. So instead of releasing new builds once a month or something, twice a year or something like that. So part of the reason server way exists is because we feel that software vendors and people who are building software have a lot of common problems. And this is one of those problems that where we think we can help and then everyone can benefit. So the server quality assurance platform as a service is really a complete managed service. It’s not a set of of tools where you kind of do it yourself and stitch it together. It really is something where it’s hosted and it provides everything integrated. Not that our customer base isn’t capable of doing this sort of thing, we’ve seen it. We’ve seen a lot of our customers build their own quality assurance pipelines. It’s just that it takes a lot of effort and it’s a lot of duplicate effort across the community. So that’s why I think it’s important that we can sort of consolidate that and share in the benefit. So what’s provided in the platform is of course revision control system source code management. Also case management because you can get to linking source code issues to particular cases and move those straight through back to the development cycle. Continuous build as we’re going to see in a minute when that build finishes that as developers are making changes and committing them that the application is being built and deployed onto test server. It can be promoted to UAT development at their discretion. Sorry that means user acceptance testing. And of course when the build happens test can be run and that can include unit tests which I didn’t show here. Something that’s been in the product for a really long time is the ability to write unit tests. So those can be easily automated and also the UATests that I showed you that I’m doing manually and locally. The reality is those get those get run in the cloud every on a trigger after every commit. The output of this allows you to monitor the quality of the application and to analyze how your application is changing. So for continuous build it’s really linked to revision control systems or school management systems and that can be done as a trigger. So every commit it could be run, it could also be done on a frequency so you know you could do it once an hour or something like that. The application is fully built and it’s deployed to a server and so you can developers can go and verify a change that they made you know functions in the test environment. And that same build script could be modified to promote that build to a user acceptance environment. So if for example I’m fixing a bug and I commit the code and then I go and verify in the test environment that it’s fixed and then if all the tests pass also then I might promote the build to an acceptance environment for quality testers to go and review. The tests are run automatically and continuously and failed tests can generate notifications, they could send emails to developers or managers. They can also generate cases in a case management system so that developers immediately get new tasks to work on for fixing issues so that cycle gets really shortened. And these are issues that might take a long time for someone to manually find or maybe it’s already in production and gets reported as a bug and then it really slows down when it can be fixed because it has to be coordinated with a production cycle. The output of all of this is that we get some monitoring and we get some analysis. Again staff can be notified when there’s a problem but we can see code quality analysis. The result of tests, we saw some of the output from the UI tests. I’ll show you a bit more of that in a minute. Also you can monitor changes over time so you can say that you know you wrote a flow and you can see that after a certain commit the flow is now running slower and you can go and analyze why that is. So it gives you the ability to do some analysis. The UI testing suite is something that we spent a lot of energy on in part because we found that it was as time consuming to write a flow as it was to build it. So you say you build an interface that a user can change their password and then you go and you write a flow for Selenium or something to go and test that functionality while you’re spending double the effort to test the test it or to write a flow for it. So we came up with a macro language based on some tools we were using so that even non-programmers could write flows and the flows could be written and modified very quickly. I only learned the syntax of yesterday I think and I was able to write a flow. So if I can do it you can do it. Again someone could a non-developer could write these flows and just run through them locally and they don’t have to have a developer environment. They don’t have to know how to program so it could be really a software tester that’s doing it and the flows can be tracked into revision control. So that even as you make changes to your to your own test scripts those get run on a trigger just like changes to the actual application. Then we get into the cloud and where this is really kind of going to scale is that this becomes automated it can run in the QA boss on a grid so then we can scale that out to test multiple browser and operating system combinations multiple versions of Servoy and also multiple versions of your code so maybe building off different branches so the actual build scripts can be extended and remember this is done as a service this is not some expertise that you need to bring in house but it can be extended so that you could even test unreleased versions of Servoy so that you know it’s safe to upgrade to version 8.3.3 for example. You can also test a branch before you release it so that as you’re working on a feature you’re running tests on that before you maybe even merge it back into the rest of your application. So what’s next I was able to show you part of this which is visual but one area we’re working on is a unified UI so kind of a manager’s console where you can log in and get a complete picture of everything because right now that is still lacking and that’s something that we have at work in progress right now I’ve seen some some prototypes of it and it’s coming along nicely. Another thing that that we’re pretty far into is an actual recorder so you could you know record clicks to generate the flows the script that you saw me modify that could be generated from someone just walking through a scenario on the application so that would definitely make it easier to write these flows. Also we don’t currently have Mac OS support when we run tests so we’re looking to see if we can expand to be able to verify that an application runs on Mac OS in the test environment. Let’s take a pause here and check on our build. Yeah it looks like it’s cleaning up so we’re just about done. Yeah well that was good good timing now it’s done.