Spotting performance metrics in obscure places!
Posted: June 13th, 2008 | Author: TnT Admin | Filed under: Concepts | Tags: Bottleneck | No Comments »
Performance metrics are often hidden within a seemingly “Black Box” application or component. After many successful projects I hope to open the black box and show that it is possible to validate performance testing requirements. The drive behind this white paper is improving the results gathering and reporting of Performance Testing exercise, the ideas and methods discussed are applicable to Test Engineers and Software Engineers a like. I hope to highlight methods to extend the coverage of Performance Testing to include more in depth analysis of the system under test.
1. Introduction
Can you count? Let’s distinguish between Performance Counters and Performance Monitors:
- Monitoring is referring to the current value of a known measure, such as CPU and Memory utilisation. Monitors are a superset of Counters.
- Counting is the number of times a particular event occurs with in a time interval, such as errors per second.
Performance counters are often hidden away in application log files, in this whitepaper I hope to explain how to spot a potential counter and how to utilize it in performance test results and analysis. Traditionally these types of counters were either ignored or spot checked after a performance test run. Consider when batch processing and job queues are in the equation – do you need to know (down to second intervals) how many X’s you had throughout the 3 hour test?
2. How do you decided what to counters to monitor?
There are many sources of documentation and experiences that may lead you to decide to count something, here are a few examples:
- Performance Requirements – always a good place to start.
- Quality Assurance – are you sure that the performance test is applying the load as you expect? This can, and should, lead into investigations into test scripts and existing system usage.
- Tester Instincts – often after initial runs, you just have the “feeling” that an area requires further investigation.
- Developer Instincts – speak to the people who developed (or have used) the system.
In some cases these points may be addressed by monitors that are already available to you, but quite often custom counters will be required.
3. What can a counter be created from?
This is not an exhaustive list but here goes:
- Logs – if the log (or audit trail) is date/time stamped then patterns within it can be counted.
- Files – how many files exist in the output directory throughout the duration of the performance test?
- Database – specific queries on the database can return counters. How many records or criteria Y were there in table X throughout the test?
Remember that one information source could contain multiple counters, e.g. you have a directory containing output files these files may contain a status or type. In this case can produce counters for each type of file, the total number of files.
How do I produce graphs from counter files?
Counter files should be generic – the simpler the format the better, we tend to work with CSV files which can easily be opened in spreadsheet application and graphed.
If we are working with proprietary performance test tools then there is usually a method to import the CSV files into that tool. This produces the richest results analysis where custom counter graphs can be overlaid onto standard graphs such as the number of concurrent users and CPU Utilisation of a server.
What are the pitfalls?
5.1. Logging gets de-prioritised and stops?
This affects monitors in general, UNIX based systems display this behaviour more readily. Under load a system will simply de-prioritise logging, this quite correct since the system has better things to do than write logs when under load. If your counters are based on log files then watch out for graphs stopping under peak load. Most people performance test to a level which is projected as a peak, at the same time they would expect the system to be capable of writing logs.
There are several ways forward here:
- Accept that the system is overloaded and report on the counters which you have.
- Arrange to have the logging made into a high priority process.
- Refer to the functional requirements to determine of logging is mandatory.
- Audit trails may be a high priority requirement from a security point of view, if the audit trail ceases to function under load – raise a security defect.
5.2. Buggy scripts
Are you sure you got the design correct for the script to produce counter files? Is it parsing the log file or querying the database correctly?
Like any piece of software – the scripts to produce the counter files should follow your organization’s/clients’ quality processes. Test the test!
5.3. Doesn’t counting impact performance?
If scripts are created to produce counter files and these scripts need to be executed during a performance test – they will impact the performance of the system! Monitoring of any kind, yes even the Linux RSTATD and Microsoft PerfMon, has an impact on the system it is monitoring.
This impact can be reduced by:
- Making the scripts as light weight as possible.
- Increase the sample rate or time interval – consider monitoring every 5 minutes and only recording the “per minute” average?
Effort should be made to do the counting separately to the performance test runs, log files can always be post processed. Measure the impact of the counter scripts by running two short tests, one with the counting scripts running and one without. Once results have been analysed and requirements validated, consider monitoring less and using the counter scripts in a small set of future test runs.
6. Example: ATM Service Management System
A major banking client was implementing an upgrade to their ATM Service Management System. This system was hooked into back office and infrastructure services such as a fax to email gateway. The business wanted to know facts like: “How many faxes per hour will the new system generate under the projected future peak load”. The system wrote faxes in the form of a file in a specific directory, a script was written that simply counted the number of files in that directory every minute.
Knowing the number of faxes that the system would generate meant that they were able to determine if the fax gateway could withstand the load. It was costly to have faxes queued up – a fax not being sent meant an ATM downtime was elongated.
7. Conclusions
Once you have produced graphs that no one thought possible and overlaid them onto standard performance testing output – you will wonder why you ever thought that this was difficult.
By using these methods we have proved time and again how performance requirements can contradict and invalidate each other, here are two example requirements:
- n Requirement A: The system will support 300 concurrent users.
- n Requirement B: The system will capable of outputting 20 PDF admin reports per minute.
It is easy to look at Requirement B by counting the PDF files per minute while testing Requirement A. If it is estimated that the scenario to test Requirement B requires 400,000 concurrent users – then you have to question the validity of one of these requirements!
How were you going to test Requirement B anyway? If you are involved in writing performance requirements, like any other requirement, consider if the requirement is testable.
(Source: Whitepaper by Stephen A Readman)
Leave a Reply