The environment of performance tests

It has been mentioned that performance testing and tuning should be performed in a controlled environment. In a perfect world, this means an environment that is free of disturbance, production-like, and unchanged between tests.

Using the following three rules of thumb for your test environment, you will be as close to achieving the perfect environment as you can for your performance tests:

• No disturbances: The tests should not be disturbed by other events, such as the executions of batches, backups, unrelated network traffic, or similar factors, to ensure that measurements relates only to the system under test. In a production environment, there is likely to be external disturbances, but the origins of these are hopefully known, and the systems that generate them should have gone through separate performance tests. Simulations in performance tests of what happens to a system at the same time as an external disturbance runs might be useful for some situations, but it is seldom an exact science and is not recommended in general.
• Production-like: The test environment should also be as similar to the production environment as possible in terms of test data, configuration, resources, services, hardware, and network capabilities in order to have results that would actually be worth something as the system is deployed into the real production environment. To have a full-blown copy of the production environment available for performance testing is not always possible due to various reasons. When the test environment isn't quite up to level with its production counterpart, it is important to be aware of the differences and to be able to extrapolate any test results. Just be very careful to trust any estimates you make about the results in a different environment.
• Unchanged: The test environment must stay equal between iterations of the same test and preferably for all tests. This intertest equality of the environment is needed in order to make reliable comparisons of the results from repeated tests. The exception to this, naturally, is when some part of the environment itself is required to change as part of tuning. Then, only one thing per test run can change and it must be thoroughly documented.