• What is the total user base?  What is the concurrent user base? – Now after working the maths out, do you required a load test.  Example, is a 5 concurrent user base an effective load test?  Is it worth the effort to generate a script that emulates 5 concurrent users?  Your customer may want you to run a load test for 5 users even though it is not effective to run that kind of load test.  However, if it is in the contract, you may have obliged it.  If they are willing to pay for that service, then it’s a plus point!
• Do the available load testing tools support your system in a load test? – A very strange question to ask.  Most tools are unable to support asynchronous application and map applications.  Either you may have to purchase more licenses to run the load test like in LoadRunner, find alternative sources or write scripts to emulate the load.  There are multiple ways to run a load test however may require additional effort (which translates to additional time).  Most tools are useful in coordinating and reporting which your scripts may not be able to achieve.
• What is the environment that you will be conducting the load test? – At times, your load test may be conducted in a scale-down production environment which you may know.  The load test results may not be a good representation of the true performance of the load test.  However, most of the times, running in production is a no-no and it is good to align everyone’s (stakeholders, customers, etc.) expectation of the load test.
• Where do you want the load to be generated to? – For a web application, there are multiple layers that you can access the system.  You can access the normal URL that can be the virtual IP address from a central web server.  You can enter at the load balancer or the web server layer.  Lastly, you can enter at the application server layer.  Where you want the load to enter can help you determine the layer that can be causing the performance problem.  By stepping up from the application layer, you can determine stage-by-stage the layer as the source of bottleneck.
• What do you want to monitor? – There are so many things that you can monitor in an infrastructure.  Which one do you want to focus on your analysis effort on first?  Do you want to monitor all sub-systems or target your effort to the most possible source such as the database?  Setting up monitors takes effort.  What’s more tedious is the effort required to analyze the monitoring results that you have collected!  Therefore, do take note and factor the duration and effort!
• Do you know the backbone network infrastructure? – This information is usually not available to a load tester.  But if you know this, you can least prepare your team that the load test will bound to meet (or not bound to meet) performance issues during the load test. They can be ineffective load balancing, small bandwidth available, invisible proxy that limits bandwidth, etc.  If you do not know this information, it is best to work on those you know such as the method previously mentioned by running the load test stage-by-stage.
• Is your load test conducted in a wireless environment? -  A wireless environment may be limited by the number of access point and the bandwidth available per wireless network. You need to know if the wireless will become a potential bottleneck in the amount of clients that can be generated into the system before your load test.
• Do you have a base set of data to use in a load test? – Most often after a load test, the data in a database will likely be messed out.  Therefore, it is advisable to create a backup of a working set of data in the database for you to reload after each load test.

1. Do you have sufficient hardware? – How may load generators are required for the entire load test?  Refer to this document from HP Support, “Is resource (cpu and memory) footprint information available to help in estimating the load generators required for load testing with LoadRunner?” for estimating the load generators required by multiplying the observations with the total number of virtual users generated in the test.   In addition to the processor utilization recommended in the document, do watch the memory and network consumption of the load generators.   The network interface card (NIC) can become a bottleneck for load generation.  A 1GB NIC will definitely generate and receive more load than a 100MB NIC.  Therefore, do a check on the machines assuming the load generator roles and make appropriate measures to prevent the NIC in becoming the bottleneck.   With the above, you will get a rough gauge of the required resource for your test.
2. Do you need spare hardware? – Do cater some spare machines if additional load generators are required, such as an additional of 5 load generators.  This is to stand by in the event of increasd virtual users and machine malfunction.
3. Remember that it’s a team effort! – Can you secure human resources for the load test?  You will need the web, application and database server teams to be around to observe the system behavior during the test or to assist in any reboot or refresh activities for the system.  Network and firewall/security team may be required to ensure that connectivity is allowed for the load test.    Therefore, in the planning, you have to get the relevant parties to commit the human resources!
4. Installation of LoadRunner – This require administrative or power users privileges in order for installation to be successful.  That raise another set of questions: Will you be able to install LoadRunner in the premises?  Who can you look for to install the LoadRunner program?  Do you need clearance for installation?  Is the system being hardened to prevent installation?
5. Offshore Installation – Is offshore installation required for the load test setup?  Can you perform the installation via remote connection or do you require someone from the site to assist in the installation?
6. Running of JVM/.NET Probes for Diagnostics – This will require installation and instrumentation on the application server, meaning it will be intrusive to the system as a whole.  Do you require additional permissions for such installation?
7. Post-installation – Do remember to test if the installation is complete and the programs are working before moving on to the next phase.  Remember to do a connectivity check between the load generators to the controller to ensure they communicate prior the load test.
8. Working in different environments – Are you required to work in different stages of a load test project?  Example, you are required to run the test in (A) the  data center followed by (B) the local client segment and lastly (C) the remote client segment?  In this context, how do you ensure the scripts worked from A to C?  It will be easy if it is a change of hostnames and URL addresses that is being used in these environments as it can be achieved through parameterization.  But what if there are additional authentication mechanism introduced in environment B and C?  Authentication mechanisms such as additional cookie in the clients are added into the URL or 2-factor authentication can render the scripts developed in environment A useless!  How do you go about it?  Should you cater more time to re-script for the worst scenario or is it better to disable the authentication mechansim during the load test?

These are some basic considerations we usually asked our ourselves and clients.  Hope it will be useful in your planning.  What are other considerations make for a load test for you?

In this post, we will not be covering on the diagnostics of the application server, but we will cover the basic signs of database bottlenecks at the SQL statement level.  The 1^st symptom is a collective symptom that the application server (and all other servers) with the exception of the database server in the architecture are performing reasonably well.  E.g. memory available is well maintained without any exception decrease of this value, the processor % time is always low for the servers.

The 2^nd symptom is a constant high utilization of the processor % time and low memory usage (page faults, page reads and available memory) in the database server.  This provides a tell-tale sign that the database server is processing some instructions, likely to be SQL statement that is taking too long.  It would not be the database buffer or SQL buffer that utilizes the memory as the consumption is low meaning reuse of the buffer is high.

Now with these two symptoms, you are ready to go down deeper.  Access MS SQL and run the following query:

This query will sys.dm_os_schedulers will give you information of how many runnable tasks that exists.  Values other than zero indicate that tasks are waiting to run, and high values are an indication that the CPU is bottlenecking your performance.  Another query that directly targets at SQL statement is the following:

This query shows which batches or procedures are consuming the most CPU and will also include the actual SQL statement. The query aggregates by a specific plan handle. If the plan handle contains more than one SQL statement you must drill into each statement to determine where the greatest CPU contribution comes from.

Codes illustrated in this post are taken from DotNet Bob on SQL.

YSlow

• Download Component Size – YSlow provides the size of the downloaded component and provides suggestion in keeping the optimum amount of data size that the component should be. This include images, Javascripts, CSS and cookie size are just few of the examples.
• 

  YSlow Screenshot

  Bad Requests – It detects missing components that churns 404 errors.  Addressing 404 errors can help improve performance at the client and server end by removing requests made on missing objects as well as removing time needed to handle this requests on missing objects on the server.

• Detect duplicate requests/components – It detects duplicate HTTP requests made which degrades both the client and the server performance.  It also detects duplicate Javascripts which can result in duplicate request to the server or further slowing down the client browser.
• Placement of files – Another good feature made by YSlow is the detection of the placement of the Javascripts and CSS files.  The placement of the Javascripts and CSS files will help improve the performance at the client side.
• Detect expiry of components – YSlow can help determine if a component should be retrieved on every web page loading.  If a component such as an image (company banner) does not changed frequently, it is suggested to it expire at a far later date.  Else, for every time the page loads, a request will be made to the image.

What is great about YSlow is it provides you with the best practises for creating high performance web sites.  It helps you detect which components are too big and provide suggestions for you to improve the performance.  The best practises for high performance web sites can be found at Yahoo! YSlow Developer Network.

The disadvantage of YSlow is it is not installable on Internet Explorer, unlike HTTPWatch.  If you are a fan of the real-time analysing the response (wait) time on HTTPWatch, then you will be disappointed with YSlow on this.  YSlow does not provide any visual indicator on the time taken during loading (or requesting to the server) in real time.  It will only display the time only after the entire web page is downloaded.

The above are just some of the features that are available in YSlow.  You can download YSlow and start profiling your website. We hope to hear your review soon!

HTTPWatch is like a traffic sniffer (they termed it traffic viewer).  Once you start a recording a session, you can view almost anything that is sent via the browser, such as the HTTP headers, the response code, cookies, any caching used, methods, redirections, compression, encoding, and authentication.

What I like most out of the available information is the header, status codes, errors, cookies, redirections and compression.   HTTPWatch can view the following:

• Header – Is a valid header being sent from the client and if the server is returning a valid header as well?
• Status Codes – What is the status code returning?  Is it giving me a correct HTTP 200 or something else (e.g. 302, 404) that may cause degradation of the client side performance?  Note: HTTP codes other than 200 may degrade the web server as well.
• Cookies – Are cookies under-performing?  Are the cookies being retrieved correctly?  Having cookies expired or should they be made to expired?
• Redirections – Are unnecessary redirections made due to invalid requests to invalid objects or missing objects that were removed (e.g. 404 errors)?
• Compression – Have all HTTP requests made from the client returned in compression thus decreasing the throughput?

The plugin can be installed for both Internet Explorer and Firefox users.  There is a free version readily available to download which is sufficient for you to conduct most performance troubleshooting and fix problems on the web server.   The problem with HTTP Watch is that it needs to install in to the system, and this can be considered intrusive if your organisation or your client has a policy against installation of non-compliance software.  It has a Firefox version which means you can use a portable version of Firefox with HTTPWatch installed with it.  This is a workaround against the policy.  Personally, I have a preference over YSlow (another HTTP sniffer/viewer in Firefox portable) than HTTPWatch as it has a cleaner interface (in my opinion).

You can use HTTPWatch if you need to profile one single user for your web application and considering not using a stop watch (at least it’s more professional than a stopwatch technique in your client’s opinion).

Click here to download HTTPWatch!

The overall speed of a particular network connection is limited by (a) the slowest link in the connection chain and (b) the length of the chain. Identifying the slowest link is difficult and may not even be consistent: it can vary at different times of the day or for different communication paths. A network communication path lead from an application through a TCP/IP stack (which adds various layers of headers, possibly encrypting and compressing data as well), then through the hardware interface, through a modem, over a phone line, through another modem, over to a service provider’s router, through many heavily congested data lines of various carrying capacities and multiple routers with different maximum throughputs and configurations, to a machine at the other end with its own hard interface, TCP/IP stack and application. A typical web download route is just like this. In addition, there are dropped packets, acknowledgments, retries, bus contention, and so on.

Because so many possible causes of bad network performance are external to an application, one option you can consider including in an application is a network speed testing facility that reports to the user. This should test the speed of data transfer from the machine to various destinations: to itself, to another machine on the local network, to the Internet Service Provider, to the target server across the network, and to any other destinations appropriate. This type of diagnostics report can tell users that they are obtaining bad performance from something other than your application. If you feel that the performance of your application is limited by the actual network communication speed, and not by other (application) factors, this facility will report the maximum possible speeds to your user.

Latency

Latency is different from the load-carrying capacity (bandwidth) of a network. Bandwidth refers to how much data can be sent down the communication channel for a given period of time and is limited by the link with the lowest bandwidth in the communication chain. The latency is the amount of time a particular data packet takes to get from one end of the communication channel to the other. Bandwidth tells you the limits within which your application can operate before the performance become affected by the volume of data being transmitted. Latency often affects the user’s view of the performance even when bandwidth isn’t a problem.

In most cases, especially Internet traffic, latency is an important concern. You can determine the basic round-trip time for a data packets from any two machines using the ping utility. (Refer to “Understanding Network – How Ping works?”). However, the time measure is for a basic underlying protocol (ICMP packet) to travel between the machines. If the communication channel is congested and the overlying protocol requires re-transmissions (often the case for Internet traffic), one transmission at the application level can actually be equivalent to many round trips.

It is important to be aware of these limitations however it is also often possible to tune the application to minimize the number of transfers by (a) packing data together, (b) caching and (c) redesigning the distributed application protocol to aim for a less conversational mode of operation. At the network level, you need to monitor the transmission statistics (using the ping and netstat utilities and packet sniffers) and consider tuning any network parameters that you have access to in order to reduce re-transmissions.

TCP/IP Stacks

The TCP/IP stack is the section of code that is responsible for translating each application-level network request (send, receive, connect, etc.) through the transport layers down to the wire and back up to the application at the other end of the connection. Because the stacks are usually delivered with the operation system and performance-tested before delivery (since a slow network connection on an otherwise fast machine and fast network is pretty obvious), it is unlikely that the TCP/IP stack itself is a performance problem.

In addition to the stack itself, stacks include several tunable parameters. One parameter worth mentioning is the maximum packet size. When your application sends data, the underlying protocol breaks the data into packets that are transmitted. There is an optimal size for packets transmitted over a particular communication channel, and the packet size actually used by the stack is compromise. Smaller packets are less likely to be dropped, but they introduced more overhead, as data probably has to be broken up into more packets with more header overhead.

If your communication takes place over a particular set of endpoints, you may want to alter the packet sizes. For a LAN segment with no router involved, the packets can be big (e.g. 8KB). For a LAN with routers, you probably want to set the maximum packet size to the size the routers allow to pass unbroken. (Routers can break up the packets into smaller ones; 1500 bytes is the typical maximum packet size and the standard for the Ethernet. The maximum packet size is configurable by the router’s network administrator.) If your application is likely to be sending data over the Internet and you cannot guarantee the route and quality of routers it will pass through, 500 bytes per packet is likely to be optimal.

Network Bottlenecks

Other causes of slow network I/O can be attributed directly to the load or configuration of the network. For example, a LAN may become congested when many machines are simultaneously trying to communicate over the network. The potential throughput of the network could handle the load, but the algorithms to provide communication channels slow the network, resulting in a lower maximum throughput. A congested Ethernet network has an average throughput approximately one third the potential maximum throughputs. Congested networks have other problems, such as dropped network packets. If you are using TCP, the communication rate on a congested network is much slower as the protocol automatically resends the dropped packets. If you are using UDP, your application must resend multiple copies for each transfer. Dropping packets in this way is common for the Internet. For LANs, you need to coordinate closely with network administrators to alert them to the problem. For single machines connected by a service provider, suggesting improvements. The phone line to the service provider may be noisier than expected: if so, you also need to speak to the phone line provider. It is also worth checking with the service provider, who should have optimal configurations they can demonstrate.

Dropped packets and re-transmissions are a good indication of network congestion problems, and you should be on constant lookup for them. Dropped packets often occur when routers are overloaded and find it necessary to drop some of the packets being transmitted as the router’s buffer overflow. This means that the overlying protocol will request the packets to be resent. The netstat utility lists re-transmission and other statistics that can identify these sorts of problems. Re-transmissions may indicate that the maximum packet size is too large.

DNS Lookup

Looking up network address is an often-overlooked cause of bad network performance. When your application tries to connect to a network address such as foo.bar.somthing.org, your application first translates foo.bar.somthing.org into a four-byte network IP address such as 10.33.6.45. This is the actual address that the network understands and uses for routing network packets. DNS translation works as follows:

1. The machine running the application sends the text string of the hostname (e.g. foo.bar.something.org) to the DNS server.
2. The DNS server checks its cache to find an IP address corresponding to that hostname. If the server does not find an entry in the cache, it asks its own DNS server (usually further up the Internet domain-name hierarchy) until ultimately the name is resolved. (This may be by components of the name being resolved, e.g. first .org, then something.org, etc, each time asking another machine as the search request is successively resolved.) This resolved IP address is added to the DBS server’s cache.
3. The IP address s returned to the original machine running the application.
4. The application uses the IP address to connect to the desired destination.

The address lookup does not need to be repeated once a connection is established, but any other connections (within the same session of the application or in other session s at the same time and later) need to repeat the lookup procedure to start another connection.

You can improve this situation by running a DNS server locally on the machine, or on a local server if the application uses a LAN. A DNS server can be run as a “caching-only” server that resets its cache each time the machine is rebooted. There would be little point in doing this if the machine used only one or two connections per hostname between successive reboots. For more frequent connections, a local DNS server can provide a noticeable speedup to connections. Nslookup is useful for investigating how a particular system does translations.

(Source: “Java Performance Tuning”)

Paging affects system performance in many ways. One, is that if a process moved some some pages to disk and the process becomes runnable, the OS has to retrieve the pages from the disk before that process can be run which results to delays in performance. In addition, both CPU and the disk I/O spend time doing the paging, reducing available processing power and increasing the load on the disks. This cascading effect involving both the CPU and I/O can degrade the performance of the whole system in such a way that it maybe difficult to even recognize that paging is the problem. The extreme version of too much paging is thrashing, in which the system is spending too much time moving pages around that it fails to perform any other significant work. (The next step is likely to be a system crash).

A little paging of the system does not affect the performance enough to cause concern. In fact, some paging can be considered good which indicates that the system’s memory resources are fully utilized. But at the point where paging becomes a significant overhead, the system is overloaded. To monitor paging activities is easy. On UNIX, the utilities vmstat and iostat provide details as to the level of paging, disk activity and memory levels. On Windows, the performance monitor has categories to show these details, such as Pages/sec, Page faults/sec and % Page files usage.

If there is more paging than optimal, the system’s RAM is insufficient or processes are too big. To improve this situation, you need to reduce the memory being used by (a) reducing the number of processes or (b) the memory utilization of some processes. Alternatively, you (c) can add RAM.  If the problem is caused by a combination of your application and others, you can partially address the situation by using process priorities. By issuing priorities to the processes, you need to be aware that using this option reduces the amount of RAM available to all other processes, which can make overall system performance worse. Therefore measure the trade-offs before deciding the resolution.

(Source: “Java Performance Tuning”)

Identifying whether the system has a problem with disk utilization first. Each system provides its own tools to identify disk usage (Windows: Perfmon, and UNIX: sar, vmstat, iostat utilities). For a start, identify whether the paging is an issue (look at disk-scan rates) and assess the overall utilization of your disks (e.g. Disk Transfers/sec on Windows, output from iostat –D on UNIX). It may be that the system has a problem independent of your application (e.g. unbalanced disks), and correcting this problem may resolve it.

If the disk analysis does not identify an obvious system problem that is causing the I/O overhead, you could try making a disk upgrade or a reconfiguration. This type of tuning can consist of any of the following:

• Upgrading to faster disks
• Adding more swap space to handle larger buffers
• Changing the disk to be striped (where files are striped across several disks, thus providing parallel I/O. e.g. with a RAID system)
• Running the data on raw partitions when this is shown to be faster.
• Distributing simultaneously accessed files across multiple disks to gain parallel I/O
• Using memory-mapped disks or files

You can never be sure if any particular disk is local to the user if you have applications running on many systems and unsure of the specification of the target system.  There is significant possibility that the disk used by the application is a network-mounted disk. This doubles the variability in response times and throughput. A network disk is a shared resource, as is the network itself, so performance is hugely and unpredictably affected by other users and network load.

We will be touching on the common areas to of performance issue, namely (a) Disk I/O, (b) Clustering Files, (c) Cache File Systems, (d) Disk Fragmentations and (e) Disk Sweet Spots.

Disk I/O

Disk writes on the system can impact performance adversely as a whole. System swap files should be placed on a separate disk from their databases (recommended by database vendors). The impact of not doing so can decrease database throughput (and system activity). This performance decreases come from not splitting I/O of two disk-intensive applications (in this case, OS paging and database I/O).

Identifying that there is an I/O problem is usually fairly easy. The most basic symptom is that things take longer than expected, while at the same time the CPU is not at all heavily worked. The disk-monitoring utilities will also show there is a lot of work being done to the disks. At the system level, you should determine the average peak requirements on the disks. Your disks will have some statistics that are supplied by the vendor, including:

Average and peak transfer rates; normally in megabytes (MB) per seconds, e.g. 5MB/sec. Form this, you can calculate how long an 8K page takes to be transferred from disk, and for example, 5MB/sec is about 5K/ms, so an 8K page takes just under 2ms to transfer.

Average seek time; normally in milliseconds (ms). This is the time required for the disk head to move radially to the correct location on the disk.

Rotational speed; normally in revolutions per minutes (rpm), e.g. 7200rpm. From this, you can calculate the average rotational delay in moving the disk under the disk-head reader, i.e., the time taken for half a revolution. For example, for 7200rpm, one revolution takes 60,000ms (60 seconds) divided by 7200rpm, which is about 8.3 ms. So half a revolution takes just over 4ms, which is consequently the average rotational delay.

With the above list, you are able to calculate the total delay for a disk operation to load a random 8K page from disk with the following formula:

Using the examples given in the list, you have 10 + 4 + 2 = 16 ms to load a random 8K page (almost an order of magnitude slower than the raw disk throughput). This calculation gives you a worst–case scenario for the disk-transfer rates for your application, allowing you to determine if the system is up to the required performance. Note that if you are reading data stored sequentially in disk (as when reading a large file), the seek time and rotational delay are incurred less than once per 8K page loaded. Basically, these two times are incurred only at the beginning of opening the file and whenever the file is fragmented. But this calculation is confounded by other processes also executing I/O to the disk at the same time. This overhead is part of the reason why swap and other intensive I/O files should not be put on the same disk.

One mechanism for speeding up disk I/O is to stripe disks. Disk striping allows data from a particular file to be spread over several disks. Striping allows reads and writes to be performed in parallel across the disks without requiring any application changes. This can speed up disk I/O quite effectively. However, be aware that the seek and rotational overhead previously listed still applies, and there maybe no performance gain if you are making many small random reads.

Finally, note again that using remote disks adversely affects I/O performance. You should not be using remote disks mounted from the network with any I/O-intensive operations if you need good performance.

Clustering Files

Reading many files sequentially is faster if the files are clustered together on the disk, allowing the disk-head reader to flow from one file to the next. This clustering is best done in conjunction with defragmenting the disks. The overhead in finding the location of a file on the disk (detailed in the previous section) is also minimized for sequential reads if the files are clustered.

If you cannot specify clustering files at the disk level, you can still provide similar functionality by putting all the files together into one large file (as is done with the ZIP file systems). This is fine if all the files are read-only files or if there is just one file that is writable (you place that at the end). However, when there is more than one writable file, you need to manage the location of the internal files in your system as one or more grow. This becomes a problem and is not usually worth the effort. (If the files have a known bounded size, you can pad the files internally, thus regaining the single file efficiency.)

Cached File Systems (RAM Disks, tmpfs, cachefs)

Most OS provide the ability to map a file system into the system memory . This ability can speed up reads and writes to certain files to control the target environment. Typically, this technique has been used to speed up the reading and writing of temporary files. For example, some compilers (of languages in general, not specifically Java) generate many temporary files during compilation. If these files are created and written directly to the system memory, the speed of compilation is greatly increased. Similarly, if you have a set of external files that are needed by your application, it is possible to map these directly into the system memory, thus allowing their reads and writes to be sped up greatly.

But note that these types of file systems are not persistent and will be cleared (as with system memory) when it’s rebooted. If the system crashes, anything in a memory-mapped file system is lost. For this reason, these types of file systems are usually suitable only for temporary files or read-only versions of disk-based files (such as mapping a CD-ROM into a memory-resident file system).

Remember that you do not have the same degree of fine control over these file systems that you have over your application. A memory-mapped file system does not use memory resources as efficiently as working directly from your application. If you have direct control over the files you are reading and writing, it is usually better to optimize this within your application rather than outside it. A memory-mapped file system takes space directly from system memory. You should consider whether it would be better to let your application grow in memory instead of letting the file system take up that system memory. For multi-user applications, it is usually more efficient for the system to map shared files directly into memory, as a particular file then takes up just one memory location rather than duplicate in each process.

The creation of memory-mapped file systems is completely system-dependent, and there is no guarantee that it is available on any particular system (though most modern OS do support this feature). For Unix, look up for cachefs and tmpfs while for Windows, look out on how to setup a RAM disk (a portion of memory mapped to a logical disk drive). In a similar way, there are products available that pre-cache shared libraries (DLL) and even executables in memory which allows application to start quicker or loads the quicker, and so may not be much help in speeding up a running system.

But you can apply the technique of memory-mapping file systems directly and quite usefully for applications in which processes are frequently started. Copy the Java distribution and all class files (all JDK, application, and third-party class files) onto a memory-mapped file system and ensure that all executions and classloads take place from the file system. Since everything (executables, DLLs, class files, resources, etc.) is already in memory, the startup time is much faster. Because only the startup (and class loading) time is affected, this technique gives only a small boost to applications that are not frequently starting processes, but can be usefully applied if startup time is a problem.

Disk Fragmentation

When files are stored on disk, the bytes in the files are note necessarily stored contiguously: their storage depends on file size and contiguous space available on the disk. This non-contiguous storage is called fragmentation. Any particular file may have some chunks in one place, and a pointer to the next chunk that may be quite a distance away on the disk.  Hard disks tend to get fragmented over time. This fragmentation delays both reads from files (including loading applications into computer memory on startup) and writes to files. This delay occurs because the disk header must wind on to the next chunk with each fragmentation.

For optimum performance on any system, it is ideal to periodically defragment the disk. This reunites files that have been split up, reducing disk heads time spent searching for data as the file-header locations have been identified, thus speeding up data access. However, defragmenting may not be effective on all systems.

Disk Sweet Spots

Most disks have a location from which data is transferred faster than from other locations. Usually, the closer the data is to the outside edge of the disk, the faster it can be read from the disk. Most hard disks rotate at constant angular speed. This means that the linear speed of the disk under a point is faster the farther away the point is from the center of the disk. Thus, data at the edge of the disk can be read from (and written to) at the faster possible rate commensurate with the maximum density of data storable on disk.

This location with faster transfer rates usually termed the disk sweet spot. Some (Commercial) utilities provide mapped access to the underlying disk and allow you to reorganize files to optimize access. On most server systems, the administrator has control over how logical partitions of the disk apply to the physical layout, and how to position files to the disk sweet spots. Experts for high-performance database system sometimes try to position the index tables of the database as close as possible to the disk sweet spot. These tables consist of relatively small amounts of data that affect the performance of the system in a disproportionately large way, so that any speed improvement in manipulating these tables is significant.

Note that some of the latest OS are beginning to include “awareness” of disk sweet spots, and attempt to move executables to sweet spots when defragmenting the disk. You may need to ensure that the defragmentation procedure does not disrupt your own use of the disk sweet spot.

(Source: Java Performance Tuning)

Processor Load

Two areas of Processor Load are worth watching as primary performance points. They are the Processor Utilization (expressed in percentage) and the Runnable Queue of processes and threads (often called the load or the task queue).

Processor Utilization; The first indicator is simply the percentage of the CPU (Or CPUs) being used by all the various threads. If this is up to 100% for significant periods of time, you may have a problem. On the other hand, if it isn’t, the CPU is under-utilized, but that is usually preferred. However, the amount of processes and threads existing in the system can be huge which it will be overwhelming to look at all of them. Therefore, start with known processes and tasks, such as application ones (user tasks), followed by OS tasks.

Some common symptoms can be resulted from the following:

• Low CPU usage can indicate that your application may be blocked for significant periods on disk or network I/O (High I/O or poor I/O)
• Low CPU usage can indicate that contention is on another server in the architecture and it’s waiting for that server to complete its task and send data back to it.
• High CPU usage can indicate thrashing (lack of RAM or too many threads)
• High CPU contention can indicate inefficient code which (indicating that you need to tune the code and reduce the number of instructions being processed to reduce the impact on the CPU).

A reasonable target is 75% CPU utilization (which from different authors varies from 75% till 85%). This means that the system is being worked toward its optimum, but also allowing some slacks for spikes due to other system or application requirements. However, note that if more than 50% of the CPU is used by system processes (i.e. administrative and IS process), your CPU is probably under-powered. This can be identified by looking at the load of the system over some period when you are not running any applications (always allow the system to run a normal/no load scenario to log its initial benchmark).

Runnable Queue; The second performance indicator indicates the average number of processes or threads waiting to be scheduled for the processor by the OS. They are run-able processes, but the processor has no time to run them and is keeping them waiting for some significant amount of time. As soon as the run queue goes above zero, the system may display contention for resources. However, there are still exceptions where runnable queue is above zero and the system is still performing at an acceptable level. A good way to identify the acceptable runnable queue of the system is to graph the Avg. Transaction Response Time with the runnable queue statistics (in Windows is Processor Queue Length). Observe any degradation of response time when the runnable queue increases. For capacity planning, a guideline proposed by Adrian Cockcroft is that performance starts to degrade if the run queue grows bigger than four times the number of CPUs.

If you can upgrade the CPU of the target environment, doubling the CPU speed is usually better than doubling the number of CPUs. And remember that parallelism in an application doesn’t necessarily need multiple CPUs. If I/O is significant, the CPU will have plenty of time for many threads.

Process Priorities

The OS also has the ability to prioritize the processes in terms of providing processor time by allocating process priority levels. Processor priorities provide a way to throttle high-demand processes, thus giving other processes a greater share of the processes. If there are other processes that need to run on the same machine but it doesn’t matter if they were run slowly, you can give your application processes a (much) higher priority than those other processes, thus allowing your application the lion’s share of CPU time on a congested system. This is worth keeping in mind if your application consists of multiple processes, you should also consider the possibility of giving your various processes different levels of priority. Being tempted to adjust the priority levels of processes, however, is often a sign that the CPU is underpowered for the tasks you have given it.

Performance metrics are often hidden within a seemingly “Black Box” application or component. After many successful projects I hope to open the black box and show that it is possible to validate performance testing requirements. The drive behind this white paper is improving the results gathering and reporting of Performance Testing exercise, the ideas and methods discussed are applicable to Test Engineers and Software Engineers a like. I hope to highlight methods to extend the coverage of Performance Testing to include more in depth analysis of the system under test.

1. Introduction

Can you count? Let’s distinguish between Performance Counters and Performance Monitors:

• Monitoring is referring to the current value of a known measure, such as CPU and Memory utilisation. Monitors are a superset of Counters.
• Counting is the number of times a particular event occurs with in a time interval, such as errors per second.

Performance counters are often hidden away in application log files, in this whitepaper I hope to explain how to spot a potential counter and how to utilize it in performance test results and analysis. Traditionally these types of counters were either ignored or spot checked after a performance test run. Consider when batch processing and job queues are in the equation – do you need to know (down to second intervals) how many X’s you had throughout the 3 hour test?

2. How do you decided what to counters to monitor?

There are many sources of documentation and experiences that may lead you to decide to count something, here are a few examples:

• Performance Requirements – always a good place to start.
• Quality Assurance – are you sure that the performance test is applying the load as you expect? This can, and should, lead into investigations into test scripts and existing system usage.
• Tester Instincts – often after initial runs, you just have the “feeling” that an area requires further investigation.
• Developer Instincts – speak to the people who developed (or have used) the system.

In some cases these points may be addressed by monitors that are already available to you, but quite often custom counters will be required.

3. What can a counter be created from?

This is not an exhaustive list but here goes:

• Logs – if the log (or audit trail) is date/time stamped then patterns within it can be counted.
• Files – how many files exist in the output directory throughout the duration of the performance test?
• Database – specific queries on the database can return counters. How many records or criteria Y were there in table X throughout the test?

Remember that one information source could contain multiple counters, e.g. you have a directory containing output files these files may contain a status or type. In this case can produce counters for each type of file, the total number of files.

How do I produce graphs from counter files?

Counter files should be generic – the simpler the format the better, we tend to work with CSV files which can easily be opened in spreadsheet application and graphed.

If we are working with proprietary performance test tools then there is usually a method to import the CSV files into that tool. This produces the richest results analysis where custom counter graphs can be overlaid onto standard graphs such as the number of concurrent users and CPU Utilisation of a server.

What are the pitfalls?

5.1. Logging gets de-prioritised and stops?

This affects monitors in general, UNIX based systems display this behaviour more readily. Under load a system will simply de-prioritise logging, this quite correct since the system has better things to do than write logs when under load. If your counters are based on log files then watch out for graphs stopping under peak load. Most people performance test to a level which is projected as a peak, at the same time they would expect the system to be capable of writing logs.

There are several ways forward here:

• Accept that the system is overloaded and report on the counters which you have.
• Arrange to have the logging made into a high priority process.
• Refer to the functional requirements to determine of logging is mandatory.
• Audit trails may be a high priority requirement from a security point of view, if the audit trail ceases to function under load – raise a security defect.

5.2. Buggy scripts

Are you sure you got the design correct for the script to produce counter files? Is it parsing the log file or querying the database correctly?

Like any piece of software – the scripts to produce the counter files should follow your organization’s/clients’ quality processes. Test the test!

5.3. Doesn’t counting impact performance?

If scripts are created to produce counter files and these scripts need to be executed during a performance test – they will impact the performance of the system! Monitoring of any kind, yes even the Linux RSTATD and Microsoft PerfMon, has an impact on the system it is monitoring.

This impact can be reduced by:

• Making the scripts as light weight as possible.
• Increase the sample rate or time interval – consider monitoring every 5 minutes and only recording the “per minute” average?

Effort should be made to do the counting separately to the performance test runs, log files can always be post processed. Measure the impact of the counter scripts by running two short tests, one with the counting scripts running and one without. Once results have been analysed and requirements validated, consider monitoring less and using the counter scripts in a small set of future test runs.

6. Example: ATM Service Management System

A major banking client was implementing an upgrade to their ATM Service Management System. This system was hooked into back office and infrastructure services such as a fax to email gateway. The business wanted to know facts like: “How many faxes per hour will the new system generate under the projected future peak load”. The system wrote faxes in the form of a file in a specific directory, a script was written that simply counted the number of files in that directory every minute.

Knowing the number of faxes that the system would generate meant that they were able to determine if the fax gateway could withstand the load. It was costly to have faxes queued up – a fax not being sent meant an ATM downtime was elongated.

7. Conclusions

Once you have produced graphs that no one thought possible and overlaid them onto standard performance testing output – you will wonder why you ever thought that this was difficult.

By using these methods we have proved time and again how performance requirements can contradict and invalidate each other, here are two example requirements:

• n Requirement A: The system will support 300 concurrent users.
• n Requirement B: The system will capable of outputting 20 PDF admin reports per minute.

It is easy to look at Requirement B by counting the PDF files per minute while testing Requirement A. If it is estimated that the scenario to test Requirement B requires 400,000 concurrent users – then you have to question the validity of one of these requirements!

How were you going to test Requirement B anyway? If you are involved in writing performance requirements, like any other requirement, consider if the requirement is testable.

(Source: Whitepaper by Stephen A Readman)