DevOps & Quality Automation

So, how does DevOps work in the testing arena? Extremely well for Borland Silk customers, as Archie Roboostoff explains in this blog.

DevOps means different things across the IT and development landscape. Take quality. The bottom line is to maintain a positive end-user experience while making the process more efficient from both speed and cost perspectives. The top line is to constantly monitor your application across all end-user endpoints, for both functional and performance needs.

The old-school method required a combination of record/replay or manual tests being run across large virtual instances of platform and browser combinations. This was great for eliminating functional and performance defects, but as more browsers and devices came to market, testing across these variants became more demanding. This lead to organizations making the tradeoff of delivery speed vs adequate testing. As many of their customers will testify, it’s a compromise too many – and one that is not required now.

For example?

One of our large financial customers was spending more time building infrastructures than testing their applications. They had almost doubled the size of their quality teams while reducing their overall test effectiveness – as the missed deadlines and budget over-runs confirmed. Their products were falling behind competitors, the end-user experience was poor and their ability to deliver new products was very unpredictable.

It doesn’t have to be this way. This blog post is a walk-through of using Silk to achieve an automated quality infrastructure for DevOps in just a few short steps. It is quick and simple. This process works for any operating environment, test desktop and web application across any combination of platform and device.

Step 1 – Create the tests

The continuous monitoring of quality in a DevOps context is about taking any development changes such as nightly builds, hotfixes, updates and full releases, and rapidly scaling out the testing environment to improve the end-user experience. This swiftly identifies any device- or browser-specific issues caused by the changes quickly, exactly where the issue occurred. Many organizations using manual application testing do so because they feel automation is too difficult to or requires a different skillset. With Silk, less technically-adept users create scripts either visually while coders work from within the IDE.

Step 2 – Verify and Catalog the newly created tests

To confirm that our test works, we need to set a context that makes collaboration easier and test reuse simpler.

Verifying the test runs is straightforward – create your test and select one or more target browsers or devices to run the test on. It doesn’t matter where or how the test was created; if the test was recorded using Internet Explorer, it can be verified to work across any browser or device. To select the test target just point and click.

Making it work for DevOps

Our test works, so let’s add some context to the test for better collaboration and reuse in the larger DevOps infrastructure. A unique Silk feature, ‘Keyword Driven Tests’, enables cataloging and providing business/end user context to a large number of tests. In this case, we will provide a keyword for our test. Keywords can be assembled to reflect a series of transactions or use cases. For example, adding keywords like ‘verifyLogin’ and ‘checkoutShoppingCart will create tests to check the robustness of the login and shopping cart checkout.

We are now able to better collaborate with business stakeholders and in the DevOps context, enabling the creation of a variety of use cases that get to the root of any post-deployment issues. Keywords are also able to take parameters and pass them to the tests in question making automation even easier.

Step 3 – Create an execution environment

How do we get to a point where continuous deployment and integration is overcoming the challenges of today’s software market? The key is to set up an environment that can replicate these real world conditions. As we have said, previously this would require a number of development/test boxes or virtual machines. These were expensive to maintain and difficult to set up. For DevOps, Silk can manage and deploy an ‘on demand’ test environment either in a public cloud, private cloud, or on-premise within the confines of the enterprise data center.

With Silk, setting this up is easy and straightforward. So let’s do it. In this example, we will setup an execution environment with the Amazon AWS infrastructure. Even though this is in a public environment, all tests and information access are secure. This environment can be set up in a more private cloud setting, on premise within the firewall –even on the tester’s individual machine. Whatever the environment parameters, Silk has you covered.

The Amazon test case

We are about to connect Silk to an Amazon instance using AWS credentials given to us by Amazon. Silk will set up the browser and device combinations so that we can rapidly deploy our applications for testing. Rather than manually setting up a range of VMs to test different versions of Chrome, Firefox, IE, Edge, etc, Silk will spin up the instances, configure them, deploy the application, run the tests, gather the results, and then close them out. It is in this context that we start to really take advantage of some key DevOps practices and principles.

Silk will take the tests we have created and pass them to the environment you have set up. Do you have applications inside the firewall that need to be tested from an external environment? No problem. Silk’s secure technology can tunnel through.

Step 5 – Run the tests

Now we’ve created the tests, created the context and set up the execution environment, we must determine when – and how – we want these tests to run. In most DevOps environments, tests are triggered to run from a continuous integration environment managed with Jenkins, Hudson, Cloudbees, etc. Whatever the preferred solution, Silk will execute tests from any of these providers.

When the tests are executed, depending on the selected configurations, Silk will run through the tests and provide a detailed analysis across all browsers, devices, keywords, and tests. Remember – the more the better, as you want to use what your end-users will be using. Better safe than sorry; along with the analysis, screen shots of each test display the evolving trends for that application. This represents visual confirmation that the test was either successful or highlighted an issue – especially important in responsive and dynamic web designs.

Each test is outlined across the platform/browser/device combination and the end user benefits from a visual representation along with detailed results analysis. Management appreciate the high level dashboard summary showing trends across targets.

Image 8 – Dashboard of runs over time.

Step 6 – Test Early, Test Often

Now that the test? environment has been established and connected to the build environment, ongoing testing across any number of environments is completely automated. Now, instead of setting up testing environments, quality teams can focus on building better tests and improving collaboration with business stakeholders. Here is where Silk delivers true DevOps value to an organization. New browsers, such as Microsoft Edge can easily be added to the configuration environment. There’s no need recreate tests; just point the tests that are already there at the new environment.

Step 7 – Performance Tune

Along with each functional automation piece, Silk can test from both a ‘load’ and ‘functional’ perspective. When testing applications under load, Silk determines average response times, performance bottlenecks, mobile latency, and anything related to a massive amount of load generated on a system. Taking a functional perspective, Silk runs a smaller number of virtual users across different configurations of browsers that outline where things can be tuned. And this is key information. For example, Silk can determine that FireFox your application runs 15% slower on Firefox than Chrome on Android. Adjusting the stylesheets or javascript may be all that is required to performance tune your application. Testing for responsive web design is crucial to keeping user experience sentiments high in a DevOps context.

Using Silk’s technology and running these tests over time will track the trends. This, along with details analytics from data within Silk and sources like Google PageSpeed, will illustrate where your applications will benefit from being fine-tuned across browsers and devices.

In conclusion

DevOps is a slightly nebuous phrase that means different things to people. But when it comes to testing, the value is pretty clear. Aligned with the right software, it will ensure your applications perform as expected across any device/browser/platform. In addition, using Silk will ensure that your apps are:

• delivered on time and within budget
• constantly improving
• responsive
• built and tested collaboratively.
• feeding trend data on responsiveness and quality to a central location
• successful with end users/consumers.

So if this sounds like something you can use, then we should talk about Silk. It’s the only tool that can reach your quality goals today and will continue to innovate to help you eliminate complexity and continuously improve the overall development and operations process. Now that’s DevOps.