Thread state Counter is an instantaneous counter that you will need to observe at very fine granularity to catch this behavior. The following tables described the thread states and reasons.

Values for Thread State Counter

0 – Initializing
1 – Ready
2 – Running
3 – Standby
4 – Terminated
5 – Waiting
6 – Transition
7 – Unknown

Values for the Thread Wait Reason counter

1 – Waiting for a page to be freed
0 – Waiting for a component of the Windows NT Executive
1 – Waiting for a page to be freed
2 – Waiting for a page to be mapped or copied
3 – Waiting for space to be allocated in the page or nonpaged pool
4 – Waiting for an Execution Delay to be resolved
5 – Suspended
6 – Waiting for a user request
7 – Waiting for a component of the Windows NT Executive
8- Waiting for a page to be freed
9 – Waiting for a page to be mapped or copied
10 – Waiting for space to be allocated in the page or nonpaged pool
11 -Waiting for Execution Delay to be resolved
12 – Suspended
13 – Waiting for a user request
14 – Waiting for an event pair high
15 – Waiting for an event pair low
16 -Waiting for an LPC Receive notice
17 – Waiting for an LPC Reply notice
18 – Waiting for virtual memory to be allocated
19 – Waiting for a page to be written to disk
20+ – Reserved

(Source: Windows 2000 Performance Guide by Mark Friedman & Odysseas Pentakalos)

Since there is one Scheduler Dispatch Queue that services all processors, the Queue Length counter is only measured at System level. The Thread State counter in the Thread object indicates precisely which threads are waiting for service at the processor(s). In other words, the Processor Queue Length counter indicates how many threads are waiting in the Scheduler dispatch Ready Queue, while the Thread State counter tells which thread are in the queue. A good working assumption is that when the processor is very busy queuing delays impact the performance of executing threads. The longer the queue, the longer the delays that threads encounter.

By monitoring the Thread State counter for all instances of the Thread object at the same time, it is possible to determine which threads are waiting in the Processor Ready Queue at the time measurements are taken. Note that the instantaneous measure of the size of the Ready Queue is influenced by the dispatchability of the performance monitor application itself. In other words, the Ready Queue at the time the Scheduler was finally able to dispatch the System Monitor Measurement thread. The System Monitor runs at high priority, but by no means at the highest priority in the system.

Priority Scheduling is the general solution designed to cope with situations where processor is very busy. Priority scheduling orders the Ready Queue, ensuring that under conditions of scarcity, the highest-priority work gains favored access to the resource.

(Source: Windows 2000 Performance Guide by Mark Friedman & Odysseas Pentakalos)

Once a thread is running, it executes continuously on the processor until one of the following events occurs:

1. An external interrupt occurs
2. The thread voluntarily relinquishes the processor, usually because it needs to perform I/O
3. The thread involuntarily relinquishes the processor because it incurred page fault, which requires the system to perform I/O on its behalf
4. A maximum uninterrupted execution time limit is reached

Interrupts

An interrupt is a signal from an external device to the processor. Hardware devices raise interrupts to request immediate servicing. An I/O request to a disk device, for example, once initiated, is processed at the device independently of the CPU. When the device completes the request, it raises an interrupt to a signal the processor that the operation has completed. This signal is treated as a high-priority event: the device is relatively slow compared to the processor, the device needs attention, and some other user may be waiting for the physical device to be free. When the processor recognizes the interrupt request, it:

1. Stops whatever it is doing immediately (unless it is already servicing a higher-priority interrupt request)
2. Saves the status of the current running thread (including the current values of processor registers, e.g. Program Counter showing the next instruction to be executed and the Stack Pointer pointing to the program’s working storage) in an internal data structure called the Thread Context.
3. Begins processing the interrupt.

The thread that was running when the interrupt occurred return to the Ready Queue, and it might not be the thread the Scheduler selects to run following interrupt processing.

Interrupt processing likely adds another thread to the Ready Queue, namely the thread that was waiting for the event to occur. In Windows, one probable consequence of an interrupt is a reordering of the Scheduler Ready Queue following interrupt processing. The device driver that completes the interrupt processing supplies a boost to the priority of the application thread that transitions from waiting to ready when the interrupt processing completes. Interrupt processing juggles priorities so that the thread made ready to run following interrupt processing is likely to be the highest-priority thread waiting to run in the Ready Queue. Thus, the application thread waiting for an I/O request to complete is likely to receive service at the processor next.

Voluntary wait

A thread voluntarily relinquishes the processor when it issues an I/O request and then waits for the request to complete. Other voluntary waits include a timer wait or waiting for a serialization signal from another thread. A thread issuing a voluntary wait enters the Wait State, causing the Windows Scheduler to select the highest-priority task in the Ready Queue to execute next. The Thread Wait Reason for a thread in a voluntary wait is 7.

Involuntary wait

Involuntary waits are most frequently associated with virtual memory management. For example, a thread enters an involuntary wait if the processor attempts to execute an instruction referencing data in buffer that is currently not resident in memory. Since the instruction cannot be executed, the processor generates a page fault interrupt, which Windows must resolve by allocating free page in memory and reading the page containing the instruction or data into memory from disk. The currently running process is suspended and the Program Counter reset to re-execute the failed instruction. The suspended task is placed in an involuntary wait state until the page requested is brought into memory and the instruction that originally failed is executed. At that point, the VM manager component of Windows is responsible for transitioning the thread from wait state back to ready.

Time allotment exceeded

A thread does not need to perform I/O or wait for an event is not allowed to monopolize the processor completely. Without intervention from the Scheduler, some very CPU-intensive execution threads will attempt to do this. A program bug may also cause the thread to go into an infinite loop, in which it attempts to execute continuously. Either way, the Windows Scheduler eventually interrupts the running thread if no other type of interrupt occurs. If the thread does not relinquish the processor voluntarily, the Scheduler eventually forces it to return to the Ready Queue. This form of processor sharing is called time-slicing, and it is designed to prevent a CPU-bound task from dominating the use of the processor for an extended period of time. Without time-slicing, a high-priority CPU-intensive thread could delay other threads waiting in the Ready Queue indefinitely. The Windows Scheduler implements time-slicing by setting a clock timer interrupt to occur at regular intervals to check on the threads that are running

When a thread’s allotted time-slice is exhausted, Windows Scheduler interrupts it and looks for another ready thread to dispatch. Of course, if the interrupted thread happens to be the highest-priority thread (or only ready thread), the Scheduler selects it to run again immediately. However, Windows also lowers the likelihood that a CPU-intensive thread will monopolize the processor. This technique of boosting the relative priority of threads waiting on device interrupts and reducing the priority of CPU-intensive threads approximates a mean time to wait algorithm, a technique for maximizing throughput in a multiprogramming environment.

(Source: Windows 2000 Performance Guide by Mark Friedman & Odysseas Pentakalos)

In Windows, hardware device interrupts are serviced by an interrupt service routine, or ISR, which is a standard device driver function. Device drivers are extensions of the OS tailored to respond to the specific characteristics of the devices they understand and control. The ISR code executes at the interrupt level priority , with interrupts at the same or lower level disabled. An ISR is high priority by definition since it interrupts the regularly scheduled thread and executes until it voluntarily relinquishes the processor (or itself interrupted by a higher-priority interrupt).

The ISR normally signals the device to acknowledge the event, stops the interrupt from occurring, and saves the device status for later processing. It then schedules a deferred procedure call (DPC) to a designated routine that performs the bulk of the device-specified work associated with interrupt processing. DPCs are a special feature of Windows designed to allow the machine to operate enabled for interrupts as much as possible. DPC code executes at a higher priority than other OS privileged modes, but one that does not disable further interrupts from occurring and being serviced.

The DPC mechanism in Windows keeps the machine running in a state enabled for interrupts as much as possible. This architecture is especially useful with Intel PC hardware where many devices connected to a single PCI bus can share the same Interrupt Request Queue (IRQ) level. DPC routines execute from a separate DPC dispatcher queue, which Windows empties before it calls the Scheduler to re-dispatch an ordinary kernel or application thread. A typical function carried out by a DPC routine is to reset the device for the next operation and launch the next request if one is queued. When the DPC completes, a high-priority kernel or device driver thread then marks the I/O function complete. At the end of this chain events, a waiting thread transitions to the ready state, poised to continue processing now that the I/O it requested has completed.

When an interrupt occurs, the thread executing loses control of the processor immediately. When the DPC performing the bulk of the interrupt processing completes, the Windows Scheduler checks the Ready Queue again and dispatches the highest-priority thread. The interrupted thread does not necessarily regain control following interrupt processing because a higher-priority task may be ready to run. This is variously known as preemptive scheduling, preemptive multithreading or preemptive multitasking.

(Source: Windows 2000 Performance Guide by Mark Friedman & Odysseas Pentakalos)

Processor time usage in Windows is broken out into two additional subcategories. % Interrupt Time represents processor cycles consumed in device driver interrupt service routines (ISRs), which process interrupts from attached peripherals such as the keyboard, mouse, disks, network interface card, etc. This is worked performed at very high priority, typically while other interrupts are disabled. It is captured and reported separately not only because of its high priority, but also because it is not easily associated with any particular user process. Windows also tracks the amount of time device drivers spend in deferred procedure calls (DPCs), which also service peripheral devices but run with interrupts enabled. DPCs represent higher-priority work than other system calls and kernel thread activity. Note that % DPC Time is already included in the % Privileged Time measure.

The Scheduler’s thread timing function is notified whenever any context switch occurs, and dutifully records the processing time for the completed function in the appropriate bucket. The context switch might involve going from one user thread to another, a user thread to a kernel function, or a kernel thread to an ISR, followed by a DPC.

(Source: Windows 2000 Performance Guide by Mark Friedman & Odysseas Pentakalos)

The basic rationale for multiprogramming is that most computing tasks do not execute instructions continuously. After a program thread executes for some period of time, it usually needs to perform an I/O operation like reading information from the disk, printing characters on a printer, or drawing data on the display. While the program is waiting for this I/O function to complete, it does not need to hand on to the processor. An OS that supports multiprogramming saves the status of a program that is waiting, restores its status when it is ready to resume execution, and finds something else that can run in the interim.

Because I/O devices are much slower than the processor, I/O operations typically take a long time compared to CPU processing, A single I/O operation to a disk may take 10 milliseconds, which means that the disk is only capable of executing perhaps 100 such operation per seconds. Printers, which are even slower, are usually rated in pages printed per minute. In contrast, processors might execute an instruction every one or two clock cycles.

In a multiprogramming OS, programs execute until they block, normally because they are waiting for an external event to occur. When this awaited event finally does occur, interrupt processing makes the program ready to run again.

Multiprogramming introduces the possibility that a program will encounter delays waiting for the processor while some other program is running. In Windows following an interrupt, the thread that was notified that an event it was waiting on occurred usually receives a priority boost from Windows Scheduler. The result is that the thread that was executing when the interrupt occurred is often pre-empted by a higher-priority thread following interrupt processing. This can delay thread execution, but does tend to balance processor utilization across CPU- and I/O bound threads.

Due to the great disparity between the speed of the devices and the speed of the processor, individual processing threads are usually not a very efficient use of the processor if allowed to execute continuously. It is important to understand that multiprogramming actually slows down individual execution threads because they are not allowed to run uninterrupted from start to finish. In other words, when the waiting thread becomes ready to execute again, it may well be delayed because some other thread is in line ahead of it. Multiprogramming represents an explicit trade off to improve overall CPU throughput, quite possibly at the expense of the execution time of any individual thread.

(Source: Windows 2000 Performance Guide by Mark Friedman & Odysseas Pentakalos)

Processor Load

Two areas of Processor Load are worth watching as primary performance points. They are the Processor Utilization (expressed in percentage) and the Runnable Queue of processes and threads (often called the load or the task queue).

Processor Utilization; The first indicator is simply the percentage of the CPU (Or CPUs) being used by all the various threads. If this is up to 100% for significant periods of time, you may have a problem. On the other hand, if it isn’t, the CPU is under-utilized, but that is usually preferred. However, the amount of processes and threads existing in the system can be huge which it will be overwhelming to look at all of them. Therefore, start with known processes and tasks, such as application ones (user tasks), followed by OS tasks.

Some common symptoms can be resulted from the following:

• Low CPU usage can indicate that your application may be blocked for significant periods on disk or network I/O (High I/O or poor I/O)
• Low CPU usage can indicate that contention is on another server in the architecture and it’s waiting for that server to complete its task and send data back to it.
• High CPU usage can indicate thrashing (lack of RAM or too many threads)
• High CPU contention can indicate inefficient code which (indicating that you need to tune the code and reduce the number of instructions being processed to reduce the impact on the CPU).

A reasonable target is 75% CPU utilization (which from different authors varies from 75% till 85%). This means that the system is being worked toward its optimum, but also allowing some slacks for spikes due to other system or application requirements. However, note that if more than 50% of the CPU is used by system processes (i.e. administrative and IS process), your CPU is probably under-powered. This can be identified by looking at the load of the system over some period when you are not running any applications (always allow the system to run a normal/no load scenario to log its initial benchmark).

Runnable Queue; The second performance indicator indicates the average number of processes or threads waiting to be scheduled for the processor by the OS. They are run-able processes, but the processor has no time to run them and is keeping them waiting for some significant amount of time. As soon as the run queue goes above zero, the system may display contention for resources. However, there are still exceptions where runnable queue is above zero and the system is still performing at an acceptable level. A good way to identify the acceptable runnable queue of the system is to graph the Avg. Transaction Response Time with the runnable queue statistics (in Windows is Processor Queue Length). Observe any degradation of response time when the runnable queue increases. For capacity planning, a guideline proposed by Adrian Cockcroft is that performance starts to degrade if the run queue grows bigger than four times the number of CPUs.

If you can upgrade the CPU of the target environment, doubling the CPU speed is usually better than doubling the number of CPUs. And remember that parallelism in an application doesn’t necessarily need multiple CPUs. If I/O is significant, the CPU will have plenty of time for many threads.

Process Priorities

The OS also has the ability to prioritize the processes in terms of providing processor time by allocating process priority levels. Processor priorities provide a way to throttle high-demand processes, thus giving other processes a greater share of the processes. If there are other processes that need to run on the same machine but it doesn’t matter if they were run slowly, you can give your application processes a (much) higher priority than those other processes, thus allowing your application the lion’s share of CPU time on a congested system. This is worth keeping in mind if your application consists of multiple processes, you should also consider the possibility of giving your various processes different levels of priority. Being tempted to adjust the priority levels of processes, however, is often a sign that the CPU is underpowered for the tasks you have given it.

For example, instances/processes such as IIS and MS Exchange can be examined with Process(inetinfo)\% Processor Time and Process(store)\% Processor Time respectively. While WebLogic Server can be examined with Process(java.exe)\% Processor Time. (Note: WLS may generate more than one java.exe which you will have to be sure of the correct instance). A rule of thumb, keep the Processor(_Total)\% Processor Time under 85%. However, there maybe cases of spikes due to backup jobs, which you can determined with patterns of constant periodic spikes or known backup tasks schedules.

If the processor is running at around 70% to 80%, its normally a good sign and means your machine is handling its load effectively and not under utilized. Average processor utilization of around 20% or 30% may suggests your machine is under utilized and maybe a wise to do server consolidation using VMWare, or allowing more applications to be house in it. (Source: Mitch Tulloch, Windows Networking)

Going down a step deeper, we can determine if the processor is busy working on kernel-related task or user-related task with the use of Processor(_Total)\% Privileged Time and Processor(_Total)\% User Time respectively. Kernel-related task will comprises of OS tasks to ensure the functionality of the OS while user-related task comprises of application tasks not native to the OS (e.g. WebLogic Servers as java.exe). I will discuss it in details in future posts.

If kernel mode utilization is high, your machine is likely underpowered as it’s too busy handling basic OS housekeeping functions to be able to effectively run other applications. It may be wiser to (1) reduce the amount of OS tasks and (2) services or distribute the schedules in non-peak timing. If user mode utilization is high, it may be your server running too many specific roles which you may consider (1) scaling the hardware (adding more or better processor), (2) reduce the number of user applications or (3) horizontal scaling of the server architecture. (Source: Mitch Tulloch, Windows Networking)

In addition, you can use System\Processor Queue Length to determine processor contention. Use this counter when the processor utilization peaks at a period of 100%. If this counter is consistently higher than around 5 when processor utilization approaches 100%, then this is a good indication that there is more work (active threads) available (ready for execution) than the machine’s processors are able to handle.

Mitch highlighted that it is not a definitive way of determining the bottleneck for some services like IIS 6 pool (similarly for JVM which spawns threads from the java.exe process) which manage their own worker threads. For example, on a busy IIS, you would want to look at other counters like ASP\Requests Queued or ASP.NET\Requests Queued as well . Also, the larger the number of active services and applications running on your server will have a busier the processor queue then norm. On such server running near 100% utilization, contention may only be a significant factor when System\Processor Queue Length exceeds more than the recommended queue length of 5 . (Source: Mitch Tulloch, Windows Networking)

Similarly, Microsoft TechNet suggests that different types of multiprocessor to have different guidelines. The suggestion was 3 threads in the queue. That is, if there are 4 processors, the threshold should be 12 for all processors. (Source: Microsoft TechNet)