This is another important area of interest in DevOps.
Effective and efficient management of Non-Production Environments (NPE) driven by structured automation can deliver significant benefits along with substantial cost savings. It also increases availability of applications while speeding up the response to business and IT needs.
IT Environments Management is probably one of the most misunderstood concepts in terms of the part it plays within the IT Organization as a whole especially its contribution to the success of an IT Project in General.
Most Program, Project and Test Managers in running Projects sooner or later come across IT Environment related issues that can severely delay the delivery of their Project.
Here is my high-level attempt in defining IT Environments Managements in its entirety.
IT Environments Management service clearly contributes to improving the overall quality of IT Software development and support through the lifecycle. IT Environments Management encompasses a set of best practices proposed to provide an effective, end to end management service for test software platforms or development environments. The software test bed or development environment could consist of a client server application, Relational Database Management System (RDBMS), middleware, interfaces, daemons, customized processes (written in any software programming language), FTP utilities etc. Functional test phases such as Unit, Integration, Acceptance, all manner of performance or non-functional testing and development phases all require IT Environments.
The primary clients of an IT Environments Management Function are Software Project, QA and UAT teams.
This service caters the following:
- Manage the creation, build, upgrade and support for all test and development Application Environments.
- Clearly defining auditable processes of allocating application environments, multiple bookings or shared usage, code upgrades, service level agreement, support, decommissioning and re-allocation.
- Manage data refreshes, collating test data and assist in the sanitizing of production or other sensitive data if necessary.
- Supply, provision and manage all Application Environment Requirements from the Project and Test teams all through the software development cycle of a project.
- Assist the Project in establishing their application environment requirements, provide expert knowledge on an IT environment’s set up, connectivity and serve as a guide to the projects in using the environment in the most efficient manner.
- Review and contribute to the Project Initiation Document (PID) ensuring that the IT Environments Management function and its deliverables are clearly defined and captured.
- Create and maintain project plans to assist in managing all activities required to successfully carry out major code upgrades to all application environments.
- Provide reports on usage / utilization, availability, forward planning and schedules.
IT Environments Management is a new and emerging area that arose for the following reasons:
- The increased IT Environment requirements for many companies who have several software projects running at any point of time.
- The increased levels of interfacing and connectivity between several systems in most organizations also known as spaghetti. For example, in some companies more than thirty systems are interfaced or connected with each other exchanging files and data flows etc. and has meant that any changes to one system mostly could require a change to many others and requiring large numbers of test and development IT environments.
- Increased awareness and more commitment to carrying out rigorous software testing especially with more companies opting to use the PMI/Prince 2 methodology and ITIL Framework.
Any IT Environments Management Tool provides the following services:
- Environments bookings and allocation, manage multiple bookings and recurring bookings.
- Provide reporting on usage, availability, interconnectivity or interfacing environments, utilization and conflict reporting etc.
- Serve as a repository of all information on an IT Environment to include Host Server names, Hardware Type, Operating System, IP Address and Interfaces if any.
The ideal background for an Application Environments Management personnel could be Software Development, Application or Technical Support, Infrastructure Project Management, Configuration and Release Management etc., but must be exposed to at least the ITIL Framework, Client-Server development, System Architecture/Design, Networks, TCP/IP and Messaging systems etc.
Terminologies Defined & Explained:
Application Environment: A single test bed or development platform instance of a software application or system that can also be used for all manner of functional and non-functional testing or could be the production instance (production environment). It could also be large, medium or small which normally refers to the size of data the RDBMS will be holding depending on the type of testing it is required for.
Integrated Application environments (also known as stripes): More than one application environment connected to each other also communicating with each other and exchanging files and data flows. Connections could be via Microsoft ODBC, via FTP, TCP/IP, daemons, middleware, defined interfaces and database links etc.
Sanitizing of data: Sanitizing of data refers to the manipulation or transformation of production data held in the DB to be used in a test or development Application Environment ensuring that for example real names, addresses, date of birth, bank account details and other sensitive information or data is transformed to dummy data.
The data is transformed whilst still maintaining its defining characteristics in a DB table such as character length (Char 25 or Varchar 50) etc. to ensure its usage in testing or development is not compromised and that the integrity is maintained. For example, a valid name such as John Morrison defined as Char 13 will now be updated in the table to become possibly a unique character string XXXXXXXYYYYYY (comprising of 13 characters including the space between John and Morrison).
Pipe Cleaning: Pipe cleaning caters for the all the activities required to be carried out before a test or development environment is handed over to the Test or Project team and includes disk clear down, archiving and purging logs, importing test data, killing off rogue processes, resetting passwords, changing environment settings, end to end connectivity or integration tests to make sure everything is working okay.
Depending on the complexity of the system a checklist of activities may be required and ticked off capturing all the checks and tests that have been completed on an environment or an integrated suite of environments prior to its hand over to a Project or Test team.
Smoke Test: A smoke test describes an initial end to end test of all the integrated or even standalone environments very possibly using dummy data and carried out by the support teams who have created or built the environment or by the test team when the environment is handed over.
- Zero-tolerance with respect to contentious environments
- Risk reduction through better compliance with regulatory requirements
- Consistent and accurate environments through automated configuration updates across environments
- Self-healing proactive resolution of quality-related issues
- Optimum utilization of infrastructure assets through better license management
- Availability of environments ‘as-a-service’ model
- Elimination of errors due to manual interventions and delays stemming from dependencies
- Effective masking of test data that caters to the requirements while maintaining security
Test Environment Management (TEM)
In the testing phase, the team structure of the test team varies as per the stage of a current project. Many times, the testers working on the project in a team keep changing and the knowledge of the environments used for a particular test cycle is lost. Moreover, the test environments are continuously evolving and it becomes eminent for someone to own the test environment configuration and its usage details. This is where establishing a TEM would prove to be a boon.
The activities under the TEM function include –
- Information Management – Maintaining a central repository of test-environments in scope with their latest version and connectivity details
- Demand Management – Allocation of test environments to teams as per requirement.
- Supply Management – Creation of new test environments as per requirement.
- Environment Monitoring
- Housekeeping – Deleting / updating outdated test-environments and its details
- Incident Management – Preliminary investigation of issues on the environment and sometimes co-ordination till an issue resolution
- Problem Management – Analyzing data for environment issues, identifying trends and taking pro-active steps to resolve issues / co-coordinating for a long-term fix.
- Configuration Management Database (CMDB) Software: This tool would be required to maintain a repository of the environment components and its versions. The data in this tool would also be helpful in incident management and problem management. (Itop, OneCMDB, CMDBuild)
- Booking Tool: This tool would be required to capture the allocation of test environments and to check for the availability of the environments. Usage analysis can also be captured in it.
- Problem / Incident Management Tools: This tool would be used to capture the problem / incident data and to manage the life-cycle of the incident / problem. Reports generated based on this data would give good insights into the health of the test environments. (Jira)
Many teams use spreadsheets instead of using specific tools for the first two areas if the data is less. However, if the data is more, it is recommended to use specialized tools for it.
- Cost of maintaining a separate team for environment management would need to be justified.
- A lot of co-ordination work would need to be done by the TEMs proactively to ensure they have the latest updates available.
Automation Enabled Integrated Approach for Release & NPE Management is shown below:
Best Practices Approach of Automation Delivering Substantial Benefits at Each Milestone is shown below: