What is Performance Testing?
Software Engineering defines Performance Testing, as a part of “Software Performance Engineering”, that is performed to validate non-functional requirements of the software such as software’s response time and throughput with particular workloads and also to identify any bottlenecks that would prevent from achieving performance goals of software. Performance testing helps to find out how responsive and stable the software is when it is subjected to increased usage or at peak workloads.
NOTE: There is a common misunderstanding about Performance Testing as Load Testing or Stress Testing. Actually, Performance Testing is all about validating software quality attributes like response, throughput, reliability, scalability and efficiency in using the available hardware resources.
Why Performance Testing?
Performance testing is not intended to find functional defects but for non-functional testing when it comes to success of software. With increasing number of web applications, performance of the same matters to attract users. Users prefer to use the web application that is faster. Performance of website or software has direct impact on the business. Hence performance testing is getting inevitable. In general, a good web site responds and loads the page in 3 seconds or less. Anything more than that requires performance tuning.
Performance testing focuses on finding issues related to performance goals (response time, throughput, Scalability). Issues affecting the goals are known as bottlenecks.
Issue with Response Time – Response time is the time taken by the software to service user requests i.e. display requested page or form OR process data submitted through a form and displaying acknowledgement page. For e.g. time difference between hitting enter button on browser address bar till the time requested page loads completely in browser window, is known as response time. Response time can increase with increased number of concurrent requests. Response time is one of the key for success of software
Issue with Throughput – Throughput is the data transfer rate that the software can send back to the user. Software throughput can drop drastically as number of request is increased gradually due to bottleneck. Drop in throughput would also mean delayed response.
Issues with Scalability – Scalability is one of the software quality attribute which checks if software works as expected even when number of concurrent requests increases to anticipated peak load. In most of the cases, due to performance bottlenecks software may not be scalable i.e., software may not function as expected or may even crash. Hence scalability is taken into consideration right from the design phase of the SDLC.
What are Performance bottlenecks?
Performance bottlenecks are one or more components (code, database query, hardware, network device, bandwidth etc.,) that causes the reduction in efficiency of software partly or entirely. Objective of performance testing is to find Performance bottlenecks and retest once the issues are fixed.
Most of the bottlenecks found during performance testing would usually be result of one or more of the below mentioned issues.
Functions or APIs consuming more resources or they are not efficient
SQL Query performance issues, some of the database queries are taking longer time to execute, need fine tuning
Database tables are not indexed
Un-used Database or network connections that are not closed in timely manner.
Deadlocks due to 2 or more processes competing for same hardware or database resources.
Network performance issues, issue with Network device, network latency, network congestion or less Bandwidth provided
Hardware resources provided are not enough for running software
Disk is fragmented as a result disk reads are taking more time.
Very frequent updates to log files or database tables that are causing delay in response.
Performance Test Parameters
Below is a comprehensive list of parameters measured during performance testing or load testing.
Physical Memory Usage – This represents the available physical memory at different points of time. Physical memory usage is important to monitor, rapid or constant increase of used memory with constant load might be an indication of memory leak and a bottleneck.
Virtual Memory Usage – This represents the available virtual memory at different points of time. Rapid or constant increase of used memory can be an indicator of delayed garbage collection or could be an indication of memory leak.
CPU Utilization – This represents the percentage of CPU utilization to run active processes at different points of time.
CPU Interrupts per second – This represents the number of interrupts received by CPU
Disk Usage – This represents the hard disk usage for read and writes at different points of time.
Database connections – This represents the number of database connections opened at any given point of time.
Database Locks – The number of locks on database objects like database table or row or rows at different points of time. More the number of locks or longer durations locks are placed would hit overall performance of the software.
Network Usage – The network bytes total per second, which is number of bytes sent and received at different points of time.
Network bottleneck – The network output queue length is the number of data packets waiting in output queue to be dispatched. Bigger the size of Network output queue length implies network traffic congestion OR bandwidth available is smaller than required OR too much of network traffic than required.
Bandwidth Usage – This represents the transfer of data in bytes per second across different networked servers or web services. E.g. assuming application server and database servers are installed on different servers, now application servers connects to database and send SQL query for execution and gets back query results, all of these refer to Bandwidth usage.
Page faults per second – This is a metric to check how frequently page faults occur. More page faults would be due to more frequent virtual memory access, hence reduced response time.
No. of active sessions – This represents number of active session are different from concurrent users. Number of active sessions means the number application sessions that are still active.
Number of open connections – This refers to number of network sockets or database connections opened at a given point of time. Connection Pooling should be used to effectively utilize network or database connections. Connection pooling reduces time delay and resource overheads required for opening and closing connections.
Performance Testing Metrics
Below is the list of mostly common Performance testing metrics captured during most of the performance testing carried out.
Transactions per second (TPS) – is the measure of number of transactions handled by server per second. Transactions include one or several requests and responses.
Average Response Times – Response time is the time difference between “request to” and “response from” the software. Response times are measured as “Time to return first byte” or “Time to return last byte”. Average response time is an average of all individual response times of similar transactions.
Throughput – is the amount of data expressed in terms of kilobyte per second that is sent as part of request or received as part of response. Throughput indicates size of the data software can receive or deliver.
Peak Response Times – This metrics indicates the longest time taken for a response. However, the average response time may be still within the required levels. Peak response time indicates a potential problem one of the virtual users faced during a particular transaction. Peak response times could be result of an expensive SQL query for particular virtual user OR the size of the text content sent for a particular virtual user OR big image or other big sized file that had to be sent as response for a particular virtual user.
Error Rates – This is an important performance metric that indicates the threshold or break point of the software or server. Higher error rates means software is not able to service the most of the requests, so the other requests are getting timed out or the server is not picking up the requests beyond a particular number of requests. Lower error rates at peak loads indicate robustness and scalability of the software.
Requests per Second – This performance metric indicates the number of request sent to server or software to process, this metrics is important when it is compared in conjunction with other metric like average response time. We can plot a graph for requests per second and average response time on two co-ordinates and see how response time of the software or server is affected and the peak load that the software under test can handle i.e. at what requests per second the average response time falls below acceptable limit.
Concurrent Users – This performance metric indicates the number of virtual users that are currently active in a load scenario i.e. the virtual user has not completed its run. The number of concurrent users and requests per second are not one and the same, as some of the concurrent users may be idle due to “think time” or waiting for response for the request that was sent earlier etc.,
Types of Testing that helps improving Performance
Load Testing – Conducted more often than other types of performance testing. Objective of load testing is to check the behavior of the software at normal and anticipated peak load conditions. Load testing is performed to check if critical functions work as expected and software’s response time, throughputs are within the acceptable levels. Load testing is performed by simulating multiple users using the software concurrently. Load testing is very hard and expensive if done manually; hence they are mostly done with load testing tools.
Stress Testing – Conducted to check the behavior of the software over and above the anticipated peak load conditions. Objective of stress testing is to find sustenance threshold or failure point of software due to excess load and to confirm that the software does not crash and does not lead to data loss or data corruption at peak loads. Stress testing helps to check robustness, error handling and availability of software when subjected to excess load. This helps the enterprise architect to plan for auto load balancing based on the failure point identified.
Configuration Testing – Conducted to increase performance of software by tweaking software or hardware configuration. Many times, minor modification or tweaking in configuration can boost software performance in a big way. Configuration testing also looks at increasing performance of the application by providing more resources like load balancers, additional memory, co-processors etc.
Volume Testing – Conducted to check stability and response of the software when subjected to higher or excess volume of data, search or output data etc. E.g. Uploading a 100 Mb image, Opening or saving a 10 Mb file in MS word or performing a spell and grammar check with word file containing 100K lines of text, are some of the examples of volume testing. Unlike load testing, volume testing inputs 1 or very few concurrent users but increases workload by increasing the volume of data to be processed or data to be generated by the software.
Soak Testing – This is also known as Endurance Testing or Longevity Testing. Software is subjected to expected load conditions over longer period of time to check if software performance deteriorates due to continuous usage or continuous load conditions. Soak testing also helps to identify some of the performance issues that might not be found with load or stress testing. Soak testing is one of the tests that can help to identify defects related to memory leaks, network or database connection leaks, excessive disk usage etc.
Spike Testing – Conducted to find the ability of software to withstand sudden increase or surge or spike in load, hence the name spike testing. Spike testing is very important especially for popular web based applications like facebook, Gmail, yahoo or hotmail etc., wherein load is generated in spikes during morning and evening when most of the users would be logging in all over the world. Spikes are generated when most of the users perform an action like login or check emails etc., at same time.
Performance Test Planning
Step 1: Plan test
Understand Performance testing requirements like functionalities to be load tested, number of Virtual Users to be generated, Page responses, Transactions to be monitored etc.
Learn the functionalities for which Performance testing is required.
Identify Performance testing scenarios
Identify Performance testing tools to be used
Setup the Performance testing tool and environment e.g.: – how many load generators have to be setup? Etc.,
Identify Resource(s) to work on Performance testing.
Document all of the above in the Performance Test Plan.
Estimate the Performance Test effort.
Step 2: Prepare for testing
- Create the test scripts and virtual user scripts for functionalities identified in Step1.
Identify and create data required for testing and verify that the required data is available in test environment.
Step 3: Create scenario
Create scenario(s) that would describe the events that occur during a testing session and details on machines, scripts, and virtual users used during the scenario.
Any Performance Testing Tool can be used to create scenarios.
One can create either manual or goal-oriented scenarios. Scenarios are used to define number of Virtual Users, Load Generator machines, and % of Virtual Users to be assigned to each script.
Trial run the created script with 1 Virtual User to check if the script is working as expected.
Step 4: Execute/Run Scenario
Execution required Scenario configuration and scheduling.
One can run the entire scenario, virtual user groups, or individual virtual users based on what was planned.
Step 5: Monitoring Scenario
Monitor virtual users’ status, transaction rate, response time, web server resource, network delays, system resource, database server resources, performance monitors.
Additionally one can watch application logs for any error messages.
Step 6: Analyzing test results
Tools provide graphs and reports to view results of the run as well as and analyze the application Performance.
The results of the Scenario run are stored in a results file depending upon the tool used.
One can work with Analysis utility within a session and analysis of session should contain at least one set of scenario results.
Display information and layout settings for the active graphs are stored in file with appropriate file extension.
Performance Testing Tools
LoadRunner is a Performance testing tool by HP that is used to for detecting performance bottlenecks with wide range of monitoring and analysis tools it provides. LoadRunner has different tools, Virtual User Generator (VuGen), Load Generator, Controller, Analysis and the AJAX TruClient. LoadRunner supports wide range of technology and applications like Java, Ajax, .Net, Citrix, Silverlight, SOA, Mobile, ERP, and Mainframe etc.
Rational Performance Tester (RFT) is Performance testing tools by IBM; it is an eclipse based performance testing tool. Rational performance test scripts are not stored as code instead is represented as test tree; each branch represents a request to server or response from the server. RFT provides option to insert custom Java code to include custom logic into the script. RFT is used for performance testing of web applications or servers. Provides additional features that can help in identification of bottleneck developed using J2EE.
WebLoad is a Load and Stress Testing Tool for Web Applications by Radview. This tool can be used for Load testing any internet applications such as Ajax, Adobe Flex, Oracle Forms and much more. Through this tool, you have the ability to measure the working performance and also its response to the users.
Silk Performer is a software performance testing tool from Micro Focus International for web, mobile and enterprise applications. Silk Performer supports major Web 2.0 environments like Adobe’s Flash/Flex, Microsoft Silverlight, and HTML/AJAX. Silk Performer also supports load testing Web applications at the protocol level (HTTP).
Jprobe is an enterprise-class Java profiler from Quest Software used for intelligent code performance analysis and diagnosis. It provides Memory Leak Analysis, Thread Analysis, Coverage Analysis and Performance Analysis. It helps to identify and resolve memory leaks, object cycling, bottlenecks, deadlocks and code coverage.
Httperf is a high performance testing tool from HP for measuring and analyzing the performance of any web services and web applications. This is mainly used to the test the HTTP servers and its performance. This tool concludes the rate at which the response is sent from each server and thereby the efficiency can be calculated.
Jmeter is an open source load testing tool by apache.org. This tool has the capacity to be loaded into a server or network so as to check on its performance and analyze its working under different conditions.
OpenSTA is an open source load testing tool by opensta.org, it is a load generation tool for web servers, it supports both HTTP and HTTPS protocols, test scripts are recorded in proprietary language called as SCL (Script control language), and it also supports data parameterization. OpenSTA can generated 100s to 1000s of virtual users, also provides tools to monitor Windows Performance. This is one Open Source load testing tool worth trying.
LoadUI is an open source load testing tool by loadui.org and Sourceforge Project Site mainly for testing web services. LoadUI can be used for distributed load testing. LoadUI Pro version from SmartBear Software provides server monitoring tools.