• QTP scripts are developed in QTP not in LoadRunner Vugen – You won’t be able to record and replay a QTP script in Vugen.  All the recording and modification has to be done on QTP.  Once the modification of the QTP script is completed, it will be ported into LoadRunner Controller as GUI Vuser for load testing.
• QTP is required to install in the Load Generators (LG) – The load generators will be required to install QTP as they are used to run the scripts.
• Load Generators will be required to set the same resolution to the machine that developed the QTP script – As QTP is object sensitive, the resolution of the LG is required to be the same with the machine that was initially used to develop the QTP script to avoid any problems arising from missing objects (due to a different resolution)
• [1] GUI Vuser license is required for LoadRunner and QTP concurrent license is required for QTP – This is the 1^st setup approach and budgeting for your licenses.  The total amount of Vuser that you wanted to generate will be the same for both LoadRunner and QTP.  Meaning, if you want to run 50 concurrent users using GUI Vuser protocol, you will need 50 GUI Vuser license and 50 QTP Concurrent license.  In this setup, the LG will create 50 instances of QTP to run the GUI Vuser.  This is described in “What licenses are required to run a scenario with GUI Vusers” from HP Software Support.
• [2] GUI Vuser license is required for LoadRunner, QTP seat/concurrent license is required for QTP and Windows Terminal License is required for Windows – This is the 2^nd setup approach and budgeting for your licenses.   For a 50 concurrent user load test, you will need 50 GUI Vuser protocol, 50 Windows Terminal License and 1 QTP license.  In this setup, the QTP license is merely used to develop the script.  The LG will create 50 remote desktop connections and each of these connections will run 1 GUI Vuser.  This setup utilizes remote desktop functionality to emulate the virtual users.  I believed that HP Support does not really state about the licenses required from Windows Terminal and this is additional cost for you to factor if you are going though this mode.  Personally, I experimented using Windows Server 2003 (that is capable of 3 remote connections by default) with this approach and it works for me.  On the server, the remote connections are established and the QTP script launches the browser for testing.  (I wonder if this is permissible in the first place and like to know if anyone else is doing it.  If you got some input on this, please feel free to shout it here!)

With the above information, you should be slightly more familiar with Quick Test Professional (QTP) integrating with LoadRunner.  However, there may be more things that need to be considered and I may have left out.  Do feel free to feedback and experiences and anything that I missed out when using QTP in load testing!

Thread state Counter is an instantaneous counter that you will need to observe at very fine granularity to catch this behavior. The following tables described the thread states and reasons.

Values for Thread State Counter

0 – Initializing
1 – Ready
2 – Running
3 – Standby
4 – Terminated
5 – Waiting
6 – Transition
7 – Unknown

Values for the Thread Wait Reason counter

1 – Waiting for a page to be freed
0 – Waiting for a component of the Windows NT Executive
1 – Waiting for a page to be freed
2 – Waiting for a page to be mapped or copied
3 – Waiting for space to be allocated in the page or nonpaged pool
4 – Waiting for an Execution Delay to be resolved
5 – Suspended
6 – Waiting for a user request
7 – Waiting for a component of the Windows NT Executive
8- Waiting for a page to be freed
9 – Waiting for a page to be mapped or copied
10 – Waiting for space to be allocated in the page or nonpaged pool
11 -Waiting for Execution Delay to be resolved
12 – Suspended
13 – Waiting for a user request
14 – Waiting for an event pair high
15 – Waiting for an event pair low
16 -Waiting for an LPC Receive notice
17 – Waiting for an LPC Reply notice
18 – Waiting for virtual memory to be allocated
19 – Waiting for a page to be written to disk
20+ – Reserved

(Source: Windows 2000 Performance Guide by Mark Friedman & Odysseas Pentakalos)

Since there is one Scheduler Dispatch Queue that services all processors, the Queue Length counter is only measured at System level. The Thread State counter in the Thread object indicates precisely which threads are waiting for service at the processor(s). In other words, the Processor Queue Length counter indicates how many threads are waiting in the Scheduler dispatch Ready Queue, while the Thread State counter tells which thread are in the queue. A good working assumption is that when the processor is very busy queuing delays impact the performance of executing threads. The longer the queue, the longer the delays that threads encounter.

By monitoring the Thread State counter for all instances of the Thread object at the same time, it is possible to determine which threads are waiting in the Processor Ready Queue at the time measurements are taken. Note that the instantaneous measure of the size of the Ready Queue is influenced by the dispatchability of the performance monitor application itself. In other words, the Ready Queue at the time the Scheduler was finally able to dispatch the System Monitor Measurement thread. The System Monitor runs at high priority, but by no means at the highest priority in the system.

Priority Scheduling is the general solution designed to cope with situations where processor is very busy. Priority scheduling orders the Ready Queue, ensuring that under conditions of scarcity, the highest-priority work gains favored access to the resource.

(Source: Windows 2000 Performance Guide by Mark Friedman & Odysseas Pentakalos)

Once a thread is running, it executes continuously on the processor until one of the following events occurs:

1. An external interrupt occurs
2. The thread voluntarily relinquishes the processor, usually because it needs to perform I/O
3. The thread involuntarily relinquishes the processor because it incurred page fault, which requires the system to perform I/O on its behalf
4. A maximum uninterrupted execution time limit is reached

Interrupts

An interrupt is a signal from an external device to the processor. Hardware devices raise interrupts to request immediate servicing. An I/O request to a disk device, for example, once initiated, is processed at the device independently of the CPU. When the device completes the request, it raises an interrupt to a signal the processor that the operation has completed. This signal is treated as a high-priority event: the device is relatively slow compared to the processor, the device needs attention, and some other user may be waiting for the physical device to be free. When the processor recognizes the interrupt request, it:

1. Stops whatever it is doing immediately (unless it is already servicing a higher-priority interrupt request)
2. Saves the status of the current running thread (including the current values of processor registers, e.g. Program Counter showing the next instruction to be executed and the Stack Pointer pointing to the program’s working storage) in an internal data structure called the Thread Context.
3. Begins processing the interrupt.

The thread that was running when the interrupt occurred return to the Ready Queue, and it might not be the thread the Scheduler selects to run following interrupt processing.

Interrupt processing likely adds another thread to the Ready Queue, namely the thread that was waiting for the event to occur. In Windows, one probable consequence of an interrupt is a reordering of the Scheduler Ready Queue following interrupt processing. The device driver that completes the interrupt processing supplies a boost to the priority of the application thread that transitions from waiting to ready when the interrupt processing completes. Interrupt processing juggles priorities so that the thread made ready to run following interrupt processing is likely to be the highest-priority thread waiting to run in the Ready Queue. Thus, the application thread waiting for an I/O request to complete is likely to receive service at the processor next.

Voluntary wait

A thread voluntarily relinquishes the processor when it issues an I/O request and then waits for the request to complete. Other voluntary waits include a timer wait or waiting for a serialization signal from another thread. A thread issuing a voluntary wait enters the Wait State, causing the Windows Scheduler to select the highest-priority task in the Ready Queue to execute next. The Thread Wait Reason for a thread in a voluntary wait is 7.

Involuntary wait

Involuntary waits are most frequently associated with virtual memory management. For example, a thread enters an involuntary wait if the processor attempts to execute an instruction referencing data in buffer that is currently not resident in memory. Since the instruction cannot be executed, the processor generates a page fault interrupt, which Windows must resolve by allocating free page in memory and reading the page containing the instruction or data into memory from disk. The currently running process is suspended and the Program Counter reset to re-execute the failed instruction. The suspended task is placed in an involuntary wait state until the page requested is brought into memory and the instruction that originally failed is executed. At that point, the VM manager component of Windows is responsible for transitioning the thread from wait state back to ready.

Time allotment exceeded

A thread does not need to perform I/O or wait for an event is not allowed to monopolize the processor completely. Without intervention from the Scheduler, some very CPU-intensive execution threads will attempt to do this. A program bug may also cause the thread to go into an infinite loop, in which it attempts to execute continuously. Either way, the Windows Scheduler eventually interrupts the running thread if no other type of interrupt occurs. If the thread does not relinquish the processor voluntarily, the Scheduler eventually forces it to return to the Ready Queue. This form of processor sharing is called time-slicing, and it is designed to prevent a CPU-bound task from dominating the use of the processor for an extended period of time. Without time-slicing, a high-priority CPU-intensive thread could delay other threads waiting in the Ready Queue indefinitely. The Windows Scheduler implements time-slicing by setting a clock timer interrupt to occur at regular intervals to check on the threads that are running

When a thread’s allotted time-slice is exhausted, Windows Scheduler interrupts it and looks for another ready thread to dispatch. Of course, if the interrupted thread happens to be the highest-priority thread (or only ready thread), the Scheduler selects it to run again immediately. However, Windows also lowers the likelihood that a CPU-intensive thread will monopolize the processor. This technique of boosting the relative priority of threads waiting on device interrupts and reducing the priority of CPU-intensive threads approximates a mean time to wait algorithm, a technique for maximizing throughput in a multiprogramming environment.

(Source: Windows 2000 Performance Guide by Mark Friedman & Odysseas Pentakalos)

In Windows, hardware device interrupts are serviced by an interrupt service routine, or ISR, which is a standard device driver function. Device drivers are extensions of the OS tailored to respond to the specific characteristics of the devices they understand and control. The ISR code executes at the interrupt level priority , with interrupts at the same or lower level disabled. An ISR is high priority by definition since it interrupts the regularly scheduled thread and executes until it voluntarily relinquishes the processor (or itself interrupted by a higher-priority interrupt).

The ISR normally signals the device to acknowledge the event, stops the interrupt from occurring, and saves the device status for later processing. It then schedules a deferred procedure call (DPC) to a designated routine that performs the bulk of the device-specified work associated with interrupt processing. DPCs are a special feature of Windows designed to allow the machine to operate enabled for interrupts as much as possible. DPC code executes at a higher priority than other OS privileged modes, but one that does not disable further interrupts from occurring and being serviced.

The DPC mechanism in Windows keeps the machine running in a state enabled for interrupts as much as possible. This architecture is especially useful with Intel PC hardware where many devices connected to a single PCI bus can share the same Interrupt Request Queue (IRQ) level. DPC routines execute from a separate DPC dispatcher queue, which Windows empties before it calls the Scheduler to re-dispatch an ordinary kernel or application thread. A typical function carried out by a DPC routine is to reset the device for the next operation and launch the next request if one is queued. When the DPC completes, a high-priority kernel or device driver thread then marks the I/O function complete. At the end of this chain events, a waiting thread transitions to the ready state, poised to continue processing now that the I/O it requested has completed.

When an interrupt occurs, the thread executing loses control of the processor immediately. When the DPC performing the bulk of the interrupt processing completes, the Windows Scheduler checks the Ready Queue again and dispatches the highest-priority thread. The interrupted thread does not necessarily regain control following interrupt processing because a higher-priority task may be ready to run. This is variously known as preemptive scheduling, preemptive multithreading or preemptive multitasking.

(Source: Windows 2000 Performance Guide by Mark Friedman & Odysseas Pentakalos)

Processor time usage in Windows is broken out into two additional subcategories. % Interrupt Time represents processor cycles consumed in device driver interrupt service routines (ISRs), which process interrupts from attached peripherals such as the keyboard, mouse, disks, network interface card, etc. This is worked performed at very high priority, typically while other interrupts are disabled. It is captured and reported separately not only because of its high priority, but also because it is not easily associated with any particular user process. Windows also tracks the amount of time device drivers spend in deferred procedure calls (DPCs), which also service peripheral devices but run with interrupts enabled. DPCs represent higher-priority work than other system calls and kernel thread activity. Note that % DPC Time is already included in the % Privileged Time measure.

The Scheduler’s thread timing function is notified whenever any context switch occurs, and dutifully records the processing time for the completed function in the appropriate bucket. The context switch might involve going from one user thread to another, a user thread to a kernel function, or a kernel thread to an ISR, followed by a DPC.

(Source: Windows 2000 Performance Guide by Mark Friedman & Odysseas Pentakalos)

The basic rationale for multiprogramming is that most computing tasks do not execute instructions continuously. After a program thread executes for some period of time, it usually needs to perform an I/O operation like reading information from the disk, printing characters on a printer, or drawing data on the display. While the program is waiting for this I/O function to complete, it does not need to hand on to the processor. An OS that supports multiprogramming saves the status of a program that is waiting, restores its status when it is ready to resume execution, and finds something else that can run in the interim.

Because I/O devices are much slower than the processor, I/O operations typically take a long time compared to CPU processing, A single I/O operation to a disk may take 10 milliseconds, which means that the disk is only capable of executing perhaps 100 such operation per seconds. Printers, which are even slower, are usually rated in pages printed per minute. In contrast, processors might execute an instruction every one or two clock cycles.

In a multiprogramming OS, programs execute until they block, normally because they are waiting for an external event to occur. When this awaited event finally does occur, interrupt processing makes the program ready to run again.

Multiprogramming introduces the possibility that a program will encounter delays waiting for the processor while some other program is running. In Windows following an interrupt, the thread that was notified that an event it was waiting on occurred usually receives a priority boost from Windows Scheduler. The result is that the thread that was executing when the interrupt occurred is often pre-empted by a higher-priority thread following interrupt processing. This can delay thread execution, but does tend to balance processor utilization across CPU- and I/O bound threads.

Due to the great disparity between the speed of the devices and the speed of the processor, individual processing threads are usually not a very efficient use of the processor if allowed to execute continuously. It is important to understand that multiprogramming actually slows down individual execution threads because they are not allowed to run uninterrupted from start to finish. In other words, when the waiting thread becomes ready to execute again, it may well be delayed because some other thread is in line ahead of it. Multiprogramming represents an explicit trade off to improve overall CPU throughput, quite possibly at the expense of the execution time of any individual thread.

(Source: Windows 2000 Performance Guide by Mark Friedman & Odysseas Pentakalos)

Both system code and device driver code occupies memory. In addition, the OS allocates data structures in two areas of memory: a pool for non-pageable storage and a pool for pageable storage. Data structures accessed by OS and driver functions when interrupts are disabled must be resident in RAM at the time they are referenced. These data structure are usually allocated from the non-pageable pool so that they reside permanently in RAM. The Pool Nonpages Bytes counter in the Memory object shows the amount of RAM currently allocated that is permanently resident in RAM.

Mainly, though, most system data structure are pageable: they are created in a pageable pool of storage and subject to page replacement like the virtual memory pages of any other process. The Windows OS maintains a working set of active pages in RAM that are subject to the same page replacement policy as ordinary process address spaces. The Cache Bytes counter provides the total number of resident pages in the current system working set. Cache Bytes is the sum of the System Cache Resident Bytes, System Driver Resident Bytes, System Code Resident Bytes, and Pool Paged Resident Bytes counters. The OS’s working set is known as the cache because it also includes resident pages of the Windows file cache, the OS function that typically consumes more RAM than any other.

The system working set is subject to the same local page replacement policy as ordinary process working sets. By default, the system cache has a minimum working set of about 4.8MB and a maximum working set of a little more than 2000 pages, or about 8MB. Just like ordinary processes, when the OS is running at its maximum working set value and it references a page that is not currently resident (causing a page fault), the new page displaces an older page from the system cache and the older page is moved to the Standby List. Just like ordinary processes, when the Balance Set manager thread responsible for page trimming detects that the OS is running at its current working set maximum and at least 4MB of RAM available, it adjusts the system working set upwards 20 pages at a time.

A registry parameter called LargeSystemCache can be set to change the system working set maximum value. When the LargeSystemCache is turned on, the system working set maximum is set to approximately 80% of the total amount of RAM installed.

Turning on the LargeSystemCache setting, which boosts the value of the system’s maximum working set size, favors the system working set over the working sets of other processes. It allows the working set of the system, which includes the Windows file cache, to grow relatively unconstrained in relation to other process working sets and to absorb the bulk of the system’s uninstalled RAM.

(Source: Windows 2000 Performance Guide by Mark Friedman & Odysseas Pentakalos)

In practice, many LRU virtual memory implementations, including the one found in Windows, attempt to maintain some pool of free pages in RAM. This pool of Available Bytes allows the OS to schedule an I/O to the paging file to resolve the page fault immediately. Then, when this pool becomes depleted, a round of page stealing is triggered, which replenishes the pool of free pages before it is completely exhausted. (Windows refer it as page trimming). Windows uses page trimming on a regular basis to maintain its pool of free pages (reported as Available Bytes).

The challenge in Windows is to identify older pages without relying on specific processor hardware support. This is accomplished by trimming pages from process working sets provisionally. After trimming pages aggressively from active processes, Windows adds them to the pool of Available Bytes, but tags them initially as being in a provisional state. They are not immediately removed from RAM. The next time threads from active processes are scheduled to run, they reference the pages are active and these stolen pages are allowed to transition back into the process working set. So-called transition faults or soft page faults are handled quickly, with a minimum overhead. Eventually, what are left behind in the pool of Available Pages are older, unused pages that are good candidates to be replaced.

Trimmed process working set pages are placed initially in the Standby List, where they are allowed to transition fault back into their process working set the next time they are referenced. Transition faults are distinguished from hard page faults, which must be satisfied by reading the paging file disk. Pages in the Standby List that are un-referenced long enough are eventually moved to the Free List. When the System Zero Thread executes, it zeros out the contents of pages on the Free List, which allows them to move to the Zero List. Pages from the Zero List are allocated whenever either a hard page fault occurs or a process references a brand new page (also known as Demand Zero page faults). Together, the size of the Standby List, the Free List, and the Zero List are added together and reported as Available Bytes.

The page trimming procedure is threshold-driven. Page trimming is invoked when one or more of the List structures that make up the pool of Available Bytes is depleted. In addition, Windows schedules writes to the paging file when the number of changed pages exceeds a threshold value. Modified pages are written in bulk to disk by Modified Page Writer threads, again based on the threshold values being exceeded.

Measurement Support

The Transition Faults/sec counter in the Memory object reports the rate at which soft page faults occur. Similarly, Demand Zero Faults/sec reports the rate at which new pages are being created. Pages Output/sec shows the rate at which changed pages had been copied to disk. By implication, since real memory is a closed system:

Pages trimmed/sec = Transition Faults + Demand Zero Faults + Hard page faults

Plus any change in the size of the Available Bytes buffer from one interval to the next. The individual sizes of the Standby, Free, and Zero lists are not reported. Nor does Windows report the rate at which the Balance Set Manager’s page trimming routine runs, nominally at least once per second.

The Pages Read/sec counter corresponds to the rate of hard page faults requiring the OS to retrieve a page from disk. The fact that Pages Input/sec is usually larger than Pages Read/sec reflects the use of anticipatory paging. Pages Input/sec is the actual number of pages retrieved from disk. Calculating:

Pages per page fault = Pages Input/sec / Pages Read/sec

Reports the average number of pages retrieved per page fault. Similarly, the Pages written/sec counter shows the number of page write operations that were initiated, and Pages output/sec counts the number of physical 4K pages actually written.

When the average number of pages read or written to disk is consistently above two pages per page fault or page write operation, and more than 10 – 20% of all I/Os to the disk are the result of paging operations, it is probably time to add another paging file , if possible. Adding a second paging file spreads the paging I/O load across another physical disk and usually improves page fault resolution time substantially.

Windows supplies one other important measurement of overall paging activity, namely Page Faults/sec. Unfortunately, interpreting how the Page Faults/sec counter relates to the other paging activity counters is problematic. Page Faults/sec includes both Transition (soft) Faults/sec and hard page faults (Pages Read/sec). It evidently also includes the number of Cache Faults/sec due to the normal diversion of application I/O to the paging subsystem. There is strong evidence that it also includes the number of Demand Zero Faults/sec.

Page Faults/sec = Cache Faults/sec + Demand Zero Faults/sec + Pages Read/sec + Transition Faults/sec

(Source: Windows 2000 Performance Guide by Mark Friedman & Odysseas Pentakalos)

When you launch Task Manager, on the Performance Tab, the MEM Usage and Memory Usage History Graphs refer to virtual memory Committed Bytes. The value corresponds to the Commit Charge (K) Total field in the lower-left quadrant. The Task Manager Commit Charge field reports on virtual memory allocations in 1024 kilobytes (KB) segments; the corresponding Committed Bytes Limit is also shown. The Memory Usage History graph charts Committed Bytes against the Commit Limit, which servers as the maximum value for the Y-axis.
In the upper-right text quadrant (immediately below the Memory Usage History timeline), the Task Manager shows Total system memory (the amount of installed RAM), Available, and System Cache, all denominated in 1000 bytes. The Task Manager Available field corresponds exactly to the System Monitor Available Kbytes counter, by the way.

As you open new applications on your desktop, watch as virtual memory Committed Bytes increases while Available Bytes decreases in tandem. As a general rule of thumb, a 5 – 10% cushion of Available Bytes is normally ample.

If you are able to start enough application programs, you will eventually observe that the number of Committed Bytes begins to exceed the amount of physical RAM installed. This is the first indication that real memory might be filling up. As long as Committed Bytes is less than the amount of RAM installed, every virtual memory page requested by application process can fit in RAM. That means the amount of RAM installed is sufficient for this workload. However, just because Committed Bytes is greater than the amount of RAM installed dos not mean RAM is completely allocated. Some applications committed pages that have not been referenced in a long time may not be currently resident in real memory. Idle pages are subject to being paged out of the system, so RAM may still not be completely allocated.

Windows continues to permit applications to allocate more virtual memory up to the Committed Bytes Limit. If your applications continue to add more Committed Bytes to the system, at some point all those active virtual memory pages will not fit easily into physical RAM and you will start to notice an impact on system performance. At this point of time, when you click on an application that you have not accessed in a while, you are likely to hear the hard disk grinding away. Windows is performing paging I/O to the disk, taking older pages in memory and writing them to the paging file to make room for current pages that t must read back into memory from the paging file disk. As you continue to open new applications and drive Committed Bytes higher, the slowdown in system performance will eventually become acute, which eventually might leads to trashing.

When the number of virtual memory Committed Bytes starts to exceed the amount of physical RAM installed, you can probably also observe that the number of Available Bytes begins to stabilize at approximately 4MB. Windows attempts to maintain a 4MB cushion of Available Bytes in order to service application requests for a new page quickly. If the working sets of active processes exceed the size of physical memory, there is contention for real memory. In Windows, you can often observe signs of this contention when the system’s pool of Available Bytes reaches about 4MB. At approximately 4MB of available RAM, Windows crosses a threshold where behavior of its virtual memory management policy changes. If Windows has difficulty maintaining a 4MB pool of available RAM and the demand for virtual memory continues to increase, you will likely observe telltale signs of trashing.

(Source: Windows 2000 Performance Guide by Mark Friedman & Odysseas Pentakalos)