• What is the total user base?  What is the concurrent user base? – Now after working the maths out, do you required a load test.  Example, is a 5 concurrent user base an effective load test?  Is it worth the effort to generate a script that emulates 5 concurrent users?  Your customer may want you to run a load test for 5 users even though it is not effective to run that kind of load test.  However, if it is in the contract, you may have obliged it.  If they are willing to pay for that service, then it’s a plus point!
• Do the available load testing tools support your system in a load test? – A very strange question to ask.  Most tools are unable to support asynchronous application and map applications.  Either you may have to purchase more licenses to run the load test like in LoadRunner, find alternative sources or write scripts to emulate the load.  There are multiple ways to run a load test however may require additional effort (which translates to additional time).  Most tools are useful in coordinating and reporting which your scripts may not be able to achieve.
• What is the environment that you will be conducting the load test? – At times, your load test may be conducted in a scale-down production environment which you may know.  The load test results may not be a good representation of the true performance of the load test.  However, most of the times, running in production is a no-no and it is good to align everyone’s (stakeholders, customers, etc.) expectation of the load test.
• Where do you want the load to be generated to? – For a web application, there are multiple layers that you can access the system.  You can access the normal URL that can be the virtual IP address from a central web server.  You can enter at the load balancer or the web server layer.  Lastly, you can enter at the application server layer.  Where you want the load to enter can help you determine the layer that can be causing the performance problem.  By stepping up from the application layer, you can determine stage-by-stage the layer as the source of bottleneck.
• What do you want to monitor? – There are so many things that you can monitor in an infrastructure.  Which one do you want to focus on your analysis effort on first?  Do you want to monitor all sub-systems or target your effort to the most possible source such as the database?  Setting up monitors takes effort.  What’s more tedious is the effort required to analyze the monitoring results that you have collected!  Therefore, do take note and factor the duration and effort!
• Do you know the backbone network infrastructure? – This information is usually not available to a load tester.  But if you know this, you can least prepare your team that the load test will bound to meet (or not bound to meet) performance issues during the load test. They can be ineffective load balancing, small bandwidth available, invisible proxy that limits bandwidth, etc.  If you do not know this information, it is best to work on those you know such as the method previously mentioned by running the load test stage-by-stage.
• Is your load test conducted in a wireless environment? -  A wireless environment may be limited by the number of access point and the bandwidth available per wireless network. You need to know if the wireless will become a potential bottleneck in the amount of clients that can be generated into the system before your load test.
• Do you have a base set of data to use in a load test? – Most often after a load test, the data in a database will likely be messed out.  Therefore, it is advisable to create a backup of a working set of data in the database for you to reload after each load test.

Before we are able to analyze and determine tuning areas, we need to understand Oracle Server Architecture and it’s underlying mechanism. Oracle Server Architecture consist of two main categories: logical (memory) structures and physical (or file) structures. The Oracle memory structure is collectively known as Oracle Instance. While the physical file structures are collectively known as Oracle database. Together, they are call an Oracle Server.

Oracle Instance

The Oracle Instance consist of the Oracle’s main memory structure, System Global Area (SGA) and the Oracle background processes. The SGA is made up of the following components: (a) Shared Pool, (b) Database Buffer Cache, (c) Java Pool and (d) Redo Log Buffer. In Oracle 9.2 and later, (e) Streams Pool was introduced into the SGA. Other components such as (f) Large Pool may also exist as it is dependent on the optional features from Oracle. The background processes, included (a) System Monitor (SMON), (b) Process Monitor (PMON), (c) Database Writer (DBW0), (d) Log Writer (LGWR) and the (e) Checkpoint Process (CKPT). Other optional processes maybe running as well such as Recoverer (RECO).

The following illustrates the entire System Global Area Components:

The components of the instance – the SGA and the required background processes – acquired space in the server’s memory immediately upon startup of the database. While the required background processes will always acquire the same amount of memory for their purposes, there are four init.ora parameters are the primary determinants of SGA’s memory requirements. They are:

• DB_CACHE_SIZE
• SHARED_POOL_SIZE
• LOG_BUFFER
• LARGE_POOL_SIZE

However, there is still one parameter in init.ora that determines the overall size of the System Global Area, which is SGA_MAX_SIZE.  Within this parameter contains two component, Shared Pool and Database Buffer Cache that can be dynamically resized using ALTER SYSTEM command. If the value for SGA_MAX_SIZE is less that the sum of the component at instance startup, Oracle will ignore the setting for SGA_MAX_SIZE and use the sum of the components as the value instead.

Oracle Database

An Oracle database is made up of physical files called (a) control files, (b) datafiles and (c) redo logs. Additional files associated with the database are (d) init.ora, trace and alert log files, password file, and any archived redo log files. The usage of the files are as followed:

Of all the files mentioned, the init.ora file have great importance in Oracle Server performance tuning with many parameters associated to improve the performance. The default location of init.ora on Windows is %ORACLE_HOME%\database while Unix is $ORACLE_HOME/dbs.

(Source: OCP Oracle 9i Performance Tuning Study Guide)

Following this article, we will be covering the following sections on the individual components that can be tuned in the Oracle Instance and the Database itself.

There are four implementations for monitoring without SiteScope and Diagnostics.

1. Direct
2. Open port on firewall
3. Use one Mercury Agent inside the firewall
4. Use two Mercury Agents: one Listener outside the firewall and one Monitoring Agent inside the firewall.

First, I presume you have a fundamental concept of how the monitoring works in LoadRunner. Bear this in mind so the discussion will be easily understood. If you are new to it, please refer to “How does monitoring work in LoadRunner?” to get a better understanding.

In [1], direct connection is required between the SUT (Server Under Test) usually inside a firewall; meaning the port is cleared and not blocked on the server. Also, prior to this the native monitor should already be working.

For example, Windows System Resources required port 139 to be opened. Therefore, ensure direct connectivity between the Controller and SUT on port 139.

In [2], where most environment is protected by a firewall, you have to lift the port that the Controller communicates to the SUT. Back to the same example, port 139 is to be lifted on the firewall to allow the monitoring data to be sent back.

Moving on to the 3rd and 4th implementation, the need for installing LoadRunner agents arises and this increases the complexity (but a little effort in understanding will definitely reduce it).

In [3], the full installation of LoadRunner (Controller) will install everything on the machine, resulting in the machine to have the LoadRunner agent. This agent is called the MI Listener, talks to the MOFW agent (Monitoring Over Firewall) for monitoring data on port 54345 (strange number huh?).

The MOFW agent which is actually installed (recommended standalone) inside the firewall. It also utilizes port 54345 to communicate back to the MI Listener. Put it simply, Controller talks to the MOFW on port 54345 in a two-way traffic.

From the MOFW agent onwards, it is the typical native monitoring ports that is required to be opened. For example, with the Window System Resources, the communication is as followed: (1) Controller to MOFW is 54345 and (2) MOFW to SUT is 139. Simple right?

In [4], the concept is similar to [3], however we extend the MI Listener as a separate machine by itself. This way, you will need to have port 50500 between the Controller and MI Listener. The communication between the MI Listener to MOFW now, is 443, the default SSL communication.

Example with Windows System Resources, you will need the following communication enabled: (1) Controller to MI Listener is 50500, (2) MI Listener to MOFW is 443 and (3) MOFW to SUT is 139.

Why do we have these monitoring implementations? There can be various reasons to it but will the following are some suggested or experienced so far:

1. For [1] and [2], it’s a simple setup through direct connection. The cavaet is, it’s security implication on opening the native ports which is usually disallowed. Example, port 139 datagram used by Windows System Resource Perfmon, is typically not allowed due to the security implication. Unless, the Controller and the SUT belongs to the same side of the firewall.

2. For [3], similar to [2], it attempts to open the port for communicating to the MOFW agent inside the firewall. In this implementation, it could be a situation that the agents had already been installed inside the firewall and all the Controller have to do is to connect to them. 54345 is not a popular port to be hacked I guessed.

3. For [4], is an attempt to use HTTP SSL (443) to communicate to the MOFW agents inside the firewall. This implementation proposed a secure way for communication as compared to lifting native monitoring ports (eg. 139).

4. For [3] and [4], you may have a dedicated machine to be a MOFW agent to collect monitoring data for every load test. This way, it’s more effective to just clear one port (either 443 or 54345).

5. For Unix System Resource monitoring which uses a dynamic port, the methods [3] and [4] is clearly more efficient for any port clearance procedures where the Controller connects to the MOFW on 54345 and MOFW connects to the Unix box on the dynamic port (no port lifting is required except 54345 is needed on the firewall).

So well, in summary, we covered the implemtation models in a general overview with the ports required and the benefits and cavaets each of them may have. This should be useful to you in planning the monitoring setup as well as

What LoadRunner does is to record whatever activities on the web application. Replay them as network traffic to the servers in the 3-tier architecture. Prior to the loading, monitoring setup has to be done on the servers in the architecture. Say, there are 2 Web Servers, 3 Application Servers and 2 Database Servers. It doesn’t matter how many of them (except the extra effort), all of them (depending on your monitoring requirements) have to be configured properly so that the monitoring data can be successfully sent back when the load test is been executed.

Put it simply.

1. Record the web application.
2. Setup the scenario for execution.
3. Setup the monitors in the 3-tier architecture.
4. Load test it!
5. Collect the monitoring after the run.
6. Analyze the results.

I would suggest reading my previous post on “What’s LoadRunner?” to get a better understanding and not dwell on this question too much. I will be covering some case studies in future which you will have a better idea of the requirements and assist you in your planning and engagement.

1. First, ensure the environment for load testing is consistent for all participating machines (ie. Controller and Load Generators). As the scripts are sent to the Load Generator to be executed, it must have the same environmental settings of the recorded scripts (in VuGen). For example, if the script was recorded in RMI-Java, JDK 1.4 exists on the machine that performed the recording. In the same context, the Load Generator will also require the JDK 1.4 to be installed on it for the script to execute properly. Therefore, ensure JDK versions are installed and similar to the recording machine.

Another example is an activity of uploading of a file by the script. If the script points to a directory that only exists in the recording machine, the Load Generator will require to create the same directory for it to be executed correctly with a copy of the designated file too.

2. Prior the scenario execution, conduct a trial run of about 10% of the actual load. This will weed out and ensure your scripts and parameters are working.

3. Once you are satisfied with the 10% load, turn off all the logs (especially Full Extended Log). These options will create an overhead on the entire load test which may skew the final results.

4. Do not panic when errors occurred during a scenario execution and start posting on Yahoo Groups for quick-and-dirty answers. What the tool does is to find out problems, right? When errors start to occur during a scenario execution, what you should do is to scope down the possibility of the errors. Use the following questions to help you with it.

1. Does it happen at the start of the scenario execution?
2. Does it happen at the middle of the scenario execution?
3. If (2), is yes, then how many Vusers were running at that point of time when the errors appeared?

With the above, you can determine if the scenario is actually failing due to the following reasons to list a few.

1. The script is not working properly in the first place or configuration of the script failed.
2. The application is experiencing load during the scenario execution.
3. Network/server issues not pertaining to load have occurred during the scenario execution.