Performance Testing

Definition

Performance testing is the system testing of an integrated, blackbox, [partial] application against its performance requirements under normal operating circumstances.

The typical goals of performance testing are to:

Identify inefficiencies and bottlenecks with regard to application performance.
Enable the underlying defects to be identified, analyzed, fixed, and prevented in the future.

The typical objectives of performance testing are to:

Partially validate the system (i.e., to determine if it fulfills its performance requirements).
Cause failures relating to performance requirements:
- Capacity (the maximum number of objects the application/databases can handle).
- Latency (the average and maximum time to complete a system operation).
- Response time (the average and the maximum application response times).
- Throughput (the maximum transaction rates that the application can handle).
Report these failures to the development teams so that the associated defects can be fixed.
Provide information that will assist in performance tuning under various workload conditions, hardware configurations, and database sizes (e.g., by helping identify performance bottlenecks).
Reduce hardware costs by providing information allowing systems engineers to:
- Identify the minimum hardware necessary to meet the performance requirements.
- Tune the application for maximum performance by identifying the optimal system configuration (e.g., by repeating the test using different configurations).
Help determine the extent to which the system is ready for launch.
Provide input to the defect trend analysis effort.

Typical examples include performance testing of an application that is:

Software only.
A system including software, hardware, and data components.
Huge (e.g., number of users, number of transactions, amount of data).
Batch with no realtime requirements.
Soft realtime (i.e., human reaction times).
Hard realtime (e.g., avionics, radar, automotive engine control).
Embeded within another system (e.g., flight-control software, cruise-control software).
Client/server or n-tier distributed.
A research prototype that will not be placed into service.
Business-critical or safety-critical.

Performance test execution can typically begin when the following preconditions hold:

The performance requirements to be tested have been specified.
The relevant software components have been implemented and have passed unit testing.
Software integration has occurred. Performance testing can begin prior to the distribution of the software components onto the hardware components to identify gross performance defects. However, performance testing must be repeated after system integration.
The relevant system components have passed system integration testing.
The independent test team is adequately staffed and trained in performance testing.
The test environment is ready.

Performance testing is typically complete when the following postconditions hold:

A test suite of test cases for every performance requirement exists.
The test suites for every scheduled performance requirement execute successfully.

Performance testing typically involves the independent test team performing the following testing tasks using the following techniques:

Performance testing is typically performed on the following environments using the following tools:

Test Environment:
- Test Harness
- Use case modeling tool
- Record and playback tool
- Performance analyzer
- Profiler

Performance testing typically results in the production of all or part of the following work products from the test work product set:

Performance testing involves the following tasks being performed during the following phases:

PHASE → TASK ↓	Business Strategy (*)	Business Optimization	Initiation	Construction	Delivery	Usage	Retirement
Test Planning	Not Applicable	Not Applicable	Completed	Optional Regression	Not Applicable	Not Applicable	Not Applicable
Test Reuse	Not Applicable	Not Applicable	Optionally Started (**)	Completed	Not Applicable	Not Applicable	Not Applicable
Test Design	Not Applicable	Not Applicable	Optionally Started (**)	Completed	Not Applicable	Optional Regression	Not Applicable
Test Implementation	Not Applicable	Not Applicable	Optionally Started (**)	Completed	Not Applicable	Optional Regression	Not Applicable
Test Execution	Not Applicable	Not Applicable	Optionally Started (**)	Completed	Not Applicable	Optional Regression	Not Applicable
Test Reporting	Not Applicable	Not Applicable	Not Applicable	Completed	Not Applicable	Optional Regression	Not Applicable

(*) Optional performance testing of COTS software components during the technology analysis and technology vendor selection tasks.
(**) Optional performance testing of the executable architecture as well as the COTS components during the vendor and tool evaluation and vendor and tool selection tasks.

A system can fulfil its operational requirements and still be a failure if it does not have adequate performance.
The iterative and incremental development cycle implies that performance testing is regularly performed in an iterative and incremental manner.
Performance testing must be automated if adequate regression testing is to occur.
Performance testing can elicit failures prior to launch.
Performance testing can begin prior to the distribution of software components onto system components in order to identify gross performance defects.
To the extent practical, reuse functional test cases as performance test cases.