Detecting processor bottlenecks

In this article of “Detecting processor bottlenecks”, we are providing a general idea of determining the bottleneck with two broad categories, namely (a) Processor Load and (b) Process Priorities. Taken from the book, “Java Performance Tuning” written by Jack Shirazi, its recommended to go through it to get a better understanding in determining resource bottlenecks and the of tuning Java technologies. The term CPU and Processor refers to the same thing are used interchangeably in this article.

Processor Load

Two areas of Processor Load are worth watching as primary performance points. They are the Processor Utilization (expressed in percentage) and the Runnable Queue of processes and threads (often called the load or the task queue).

Processor Utilization; The first indicator is simply the percentage of the CPU (Or CPUs) being used by all the various threads. If this is up to 100% for significant periods of time, you may have a problem. On the other hand, if it isn’t, the CPU is under-utilized, but that is usually preferred. However, the amount of processes and threads existing in the system can be huge which it will be overwhelming to look at all of them. Therefore, start with known processes and tasks, such as application ones (user tasks), followed by OS tasks.

Some common symptoms can be resulted from the following:

• Low CPU usage can indicate that your application may be blocked for significant periods on disk or network I/O (High I/O or poor I/O)
• Low CPU usage can indicate that contention is on another server in the architecture and it’s waiting for that server to complete its task and send data back to it.
• High CPU usage can indicate thrashing (lack of RAM or too many threads)
• High CPU contention can indicate inefficient code which (indicating that you need to tune the code and reduce the number of instructions being processed to reduce the impact on the CPU).

A reasonable target is 75% CPU utilization (which from different authors varies from 75% till 85%). This means that the system is being worked toward its optimum, but also allowing some slacks for spikes due to other system or application requirements. However, note that if more than 50% of the CPU is used by system processes (i.e. administrative and IS process), your CPU is probably under-powered. This can be identified by looking at the load of the system over some period when you are not running any applications (always allow the system to run a normal/no load scenario to log its initial benchmark).

Runnable Queue; The second performance indicator indicates the average number of processes or threads waiting to be scheduled for the processor by the OS. They are run-able processes, but the processor has no time to run them and is keeping them waiting for some significant amount of time. As soon as the run queue goes above zero, the system may display contention for resources. However, there are still exceptions where runnable queue is above zero and the system is still performing at an acceptable level. A good way to identify the acceptable runnable queue of the system is to graph the Avg. Transaction Response Time with the runnable queue statistics (in Windows is Processor Queue Length). Observe any degradation of response time when the runnable queue increases. For capacity planning, a guideline proposed by Adrian Cockcroft is that performance starts to degrade if the run queue grows bigger than four times the number of CPUs.

If you can upgrade the CPU of the target environment, doubling the CPU speed is usually better than doubling the number of CPUs. And remember that parallelism in an application doesn’t necessarily need multiple CPUs. If I/O is significant, the CPU will have plenty of time for many threads.

Process Priorities

The OS also has the ability to prioritize the processes in terms of providing processor time by allocating process priority levels. Processor priorities provide a way to throttle high-demand processes, thus giving other processes a greater share of the processes. If there are other processes that need to run on the same machine but it doesn’t matter if they were run slowly, you can give your application processes a (much) higher priority than those other processes, thus allowing your application the lion’s share of CPU time on a congested system. This is worth keeping in mind if your application consists of multiple processes, you should also consider the possibility of giving your various processes different levels of priority. Being tempted to adjust the priority levels of processes, however, is often a sign that the CPU is underpowered for the tasks you have given it.

Related Posts

• How do we determine processor contention?
• Planning a Load Test – Part 2
• Planning for a Load Test
• Find Offending SQL Bottlenecks!
• Analyzing Client-side Performance Issue with YSlow
• Performance Profiling Using HTTP Watch
• Understanding Processor: Thread State
• Understanding Processor: Ready Queue
• Understanding Processor: Ready Queue Management
• Understanding Processor: Interrupt Processing