Continuous Testing and Continuous Monitoring

The goal of Continuous Integration is to enable Continuous Delivery of the code developers’ produce out to users. But first and foremost, it is to enable ongoing test and verification of the code, and to validate that the code produced and integrated with that from other developers and with other components of the application, functions and performs as designed.

Once the application has been deployed to a production system, it is also a goal to monitor the deployed application to ensure that it functions and performs as designed in the production environment, as it is being used by end-users.

This brings the two practices of DevOps, namely Continuous Testing and Continuous Monitoring, that are enabled by Continuous Integration and Continuous Delivery. 

Continuous Integration and Delivery are both meaningless without Continuous Testing. Not having monitoring and hence, not knowing how the application is performing in production makes the whole process of DevOps moot. What good in having a streamlined Continuous Delivery process is the only way you find out about your applications’ functionality or performance being below par is via a ticket opened by a displeased user?

Continuous Integration and Continuous Delivery facilitate Continuous Testing

Individual developers work to create code – to fix defects, add new features, enhance features, or to make the code perform faster – could be one of several tasks they could be working on. When done, they deliver their code and integrate it with the work done by other developers on their team, and with unchanged code their team owns. They would along the way run Unit Tests on their own code. Once the integration is done, they would do Unit Tests on the integrated code. They may run other tests such as White box, Security tests, code performance tests, etc. This work would then be delivered to the common Integration area of the team of teams integrating the work of all the teams working on the project and all the code components that make up the service, application or system being developed. This is the essence of the process of Continuous Integration. What makes this process continuous is where the individual developers’ code is integrated with that of their team as and when they check-in the code and it gets delivered for Integration. The important point to note here is the goal of the Continuous Integration process – to validate that the code integrates at all levels without error and that all tests run by developers run without error. Continuous Testing starts right with the developers.

After validating that the complete application is built without error, the application is delivered to QA. This delivering of code from the Dev environment to the QA environment is the first major step of Continuous Delivery. There is Continuous Delivery happening as the developers’ deliver their code to their teams’ integration space and to the projects integration space, but this is limited to being within the Dev space. There is no new environment to target.

When delivering to QA, we are speaking of a complete transition from one environment to another. QA would have its own environment on which to run its suites of functional and performance tests. DevOps principles preach that this environment be production-like. In addition, QA would potentially also need new data sets for each run of the suites of tests it runs. This may be even one or more times every day as Continuous Integration leads to Continuous Delivery at a steady stream. This means that the Continuous Delivery process would not only require the processes to transition the code from Dev to QA, but to also ‘refresh’ or provision new instances of QA’s ‘Production-like’ environment, complete with the right configurations, and associated test data to run the tests against. This makes Continuous Delivery a more complex process than just ftp code transfer. The key point to note is that the goal of Continuous Delivery is to get the code ready for test. To get the application to the right environment – continuously, so that it can be tested – continuously.

If one extends the process described above to delivering the application to a staging and eventually a production environment, the process and goal remains the same. The Ops team wants to run their own set of smoke tests, acceptance tests and system stability tests before they deliver the application to the ‘must-stay-up-at-all-costs’ production environment. That is done using the Staging environment. This is a production-like environment that needs to be provisioned just like the QA Environment. It needs to have the necessary scripts and test data for acceptance and performance tests that Ops would run. Only when this last phase of Continuous Testing is complete would the application be delivered to Production. Continuous Delivery processes would perform these tasks too – provision staging and production environments and deliver the application.

Continuous Monitoring

In Production, the Ops team manages and ensures that the application is performing as desired and the environment is stable via Continuous Monitoring. While the Ops teams have their own tools to monitor their environments and systems, DevOps principles suggest that they also monitor the applications. They need to ensure that the applications are performing at optimal levels – down to levels lower than system monitoring tools would allow. This requires that Ops teams use tools that can monitor application performance and issues. It may also require that they work with Dev to build self-monitoring or analytics gathering capabilities right into the applications being built. This would allow for true end-to-end monitoring – continuously.

The Four Continuous Processes of DevOps:

It would be close enough to say that the practice of DevOps is really up of the implementation of these four basic processes:

  • Continuous Integration
  • Continuous Delivery
  • Continuous Testing
  • Continuous Monitoring

Of these, the end goal is really Continuous Testing and Continuous Monitoring.

Continuous Integration and Continuous Delivery get the code in the right state and in the right environment to enable these two processes. Testing and Monitoring are what will validate that you have built the right application, which functions and performs as designed.

Advertisements

3 thoughts on “Continuous Testing and Continuous Monitoring

Add yours

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

Up ↑

%d bloggers like this: