• What is the total user base?  What is the concurrent user base? – Now after working the maths out, do you required a load test.  Example, is a 5 concurrent user base an effective load test?  Is it worth the effort to generate a script that emulates 5 concurrent users?  Your customer may want you to run a load test for 5 users even though it is not effective to run that kind of load test.  However, if it is in the contract, you may have obliged it.  If they are willing to pay for that service, then it’s a plus point!
• Do the available load testing tools support your system in a load test? – A very strange question to ask.  Most tools are unable to support asynchronous application and map applications.  Either you may have to purchase more licenses to run the load test like in LoadRunner, find alternative sources or write scripts to emulate the load.  There are multiple ways to run a load test however may require additional effort (which translates to additional time).  Most tools are useful in coordinating and reporting which your scripts may not be able to achieve.
• What is the environment that you will be conducting the load test? – At times, your load test may be conducted in a scale-down production environment which you may know.  The load test results may not be a good representation of the true performance of the load test.  However, most of the times, running in production is a no-no and it is good to align everyone’s (stakeholders, customers, etc.) expectation of the load test.
• Where do you want the load to be generated to? – For a web application, there are multiple layers that you can access the system.  You can access the normal URL that can be the virtual IP address from a central web server.  You can enter at the load balancer or the web server layer.  Lastly, you can enter at the application server layer.  Where you want the load to enter can help you determine the layer that can be causing the performance problem.  By stepping up from the application layer, you can determine stage-by-stage the layer as the source of bottleneck.
• What do you want to monitor? – There are so many things that you can monitor in an infrastructure.  Which one do you want to focus on your analysis effort on first?  Do you want to monitor all sub-systems or target your effort to the most possible source such as the database?  Setting up monitors takes effort.  What’s more tedious is the effort required to analyze the monitoring results that you have collected!  Therefore, do take note and factor the duration and effort!
• Do you know the backbone network infrastructure? – This information is usually not available to a load tester.  But if you know this, you can least prepare your team that the load test will bound to meet (or not bound to meet) performance issues during the load test. They can be ineffective load balancing, small bandwidth available, invisible proxy that limits bandwidth, etc.  If you do not know this information, it is best to work on those you know such as the method previously mentioned by running the load test stage-by-stage.
• Is your load test conducted in a wireless environment? -  A wireless environment may be limited by the number of access point and the bandwidth available per wireless network. You need to know if the wireless will become a potential bottleneck in the amount of clients that can be generated into the system before your load test.
• Do you have a base set of data to use in a load test? – Most often after a load test, the data in a database will likely be messed out.  Therefore, it is advisable to create a backup of a working set of data in the database for you to reload after each load test.

Let’s go through an overview for implementing the Oracle DB. Basically, we are doing in this sequence:

1. Get an account in Oracle and access to the V$ tables that contains the monitoring data.
2. Install 32-bit Oracle client on the Controller machine.
3. Ensuring that proper configuration is done to connect to the DB using TNSNAMES.ora and defining the Oracle path.
4. Login to the DB using SQL*PLUS and run a query to see if you can collect the statistics from the V$ tables.
5. Launch LoadRunner Controller, configure the monitor and start your load test!

Ok, let’s dive into the details!

1. Request for an account and password to be created in the database instance (unless you are the DBA). Example, loadtester, and grant them access to read the following table. These tables consist of the statistic of the database instance.

• V$SESSTAT
• V$SYSSTAT
• V$STATNAME
• V$INSTANCE
• V$SESSION

2. Ensure that Oracle client libraries are installed on the Controller. Remembered that in order to monitor, you need local installation of the client in order for LoadRunner to query the monitoring data. If you do not have the client libraries, download it from Oracle Downloads. Download only the 32-bit Oracle client.

3. If you have the client installed, ensure the 32-bit Oracle client is installed on the Controller, not the 16-bit client.

4. Verify that %OracleHome%\bin is included in the path environment. This can be done by going to START > My Computer > Properties > “Advanced Tab” > Enviromental Settings.

5. Configure the TNSNAMES.ora file on the Controller. TNSNAMES.ora file is a SQL*Net configuration file that defines databases addresses for establishing connections to them. This file normally resides in the ORACLE_HOME\NETWORK\ADMIN directory. (Source: OraFaq)

Attached is an example of the TNSNAMES.ora file. Click here to download.

6. Ensure login successful with the created account (e.g. loadtester) with SQL*Plus from the Controller machine.

7. Ensure the privileges are given properly by typing the following command. If they don’t return results or return access rights issues, most probably the steps for prior to this one has not been configured properly. Therefore, approach the Database Administrator for assistance or go back to the previous steps to ensure they are properly configured.

• SELECT * FROM V$SESSTAT
• SELECT * FROM V$SYSSTAT
• SELECT * FROM V$STATNAME
• SELECT * FROM V$INSTANCE
• SELECT * FROM V$SESSION

8. Launch LoadRunner Controller as per norm and add the Oracle Database Monitor. When prompt, enter the account name (e.g. loadtester), it’s corresponding password and the destination server name. Statistics should be drawn from the V$ table and displayed in Controller.

With the above, you should be able to successfully configure the monitoring environment and the Controller with minimal difficulty.

Note:
You can change the sampling interval for the monitor using the vmon.cfg file located in {loadrunner-installed-dir}\monitors\vmon.cfg

To configure Sybase database monitoring, we will require ASE Monitoring Server (not the ASE) to be installed on the Controller (refer to “How does the monitoring work in LoadRunner?” for monitoring concepts). Furthermore, we will require an additional DLL in Controller for it to work. From the Monitor Reference, the DLL is named, libct.dll. That is for version 12. While in version 15, it’s named libsybct.dll. ASE in short for Adaptive Server Enterprise is a database product from Sybase. To get the latest copy of the ASE server can be obtained from the Sybase website.

• Downloading the installation requires the registration with Sybase. Registration process is hassle-free without any approval needed and you will be able to proceed with download once the registration completes.
• Upon full installation, all components are installed which includes the ASE database and the ASE Monitoring Server. Sample databases are provided in this installation if you required them.
• Both servers are not started by default. Therefore, you will need to configure them and start them up manually.
• To start the server go to START > Programs > Sybase > Adaptive Server Enterprise > Server Config.
• To start the ASE Monitoring Server, go to Service of Windows and find Sybase MONServer_<ASE Monitoring Server Name>.
• Alternatively, the ASE Monitoring Server can be booted from configuring the ASE Monitoring Server. Details for booting the ASEMS can be found in the additional resources.

NOTES

• Controller uses ASE Monitoring Server to monitor and not ASE and therefore, there is a need to ensure the ASE Monitoring Server is started, properly configured and accessible by the Controller.
• Login information to configure the ASE and ASE Monitoring Server are; Login: sa, Password: <blank>
• There maybe a difference of the filenames for the DLLs being used by ASE Open Client Connectivity. The filenames in v15 is libsybct.dll as compared to libct.dll in v12.
• Replacing the file libct.dll with a renamed libsybct.dll (libsybct.dll renamed to libct.dll) in the Controller does not have any impact to the monitoring.
• When configuring the ASE Monitoring Server in the Controller, in the Monitored Server Machines, put the machine name on the network (not the Sybase ASE server itself). In the Resource Measurements on the server machine, put the ASE Monitoring Server name (not the Sybase ASE server itself)

TROUBLESHOOTING

• After installing the Sybase ASE. Ensure that monitoring can be performed by the ASE Monitoring Server on the local machine. Use the Sybase Central console to perform the previous mentioned, by adding the “Monitor” folder. If the monitoring failed, it maybe the case that the ASE Monitoring Server is not started or configured properly. From here, troubleshoot accordingly.
• If monitoring was successful from ASE Monitoring Server locally, use dsedit tool from Sybase client to ensure connectivity.
• If dsedit tool locates the ASE and ASE Monitoring Server successfully, but connectivity fails when connecting via Controller, there might be a configuration problem on the Controller-side.

ADDITIONAL RESOURCES

• http://www.sypron.nl/main.html -Webpage that contains Sybase ASE information.
• http://www.sypron.nl/get_ase_soft.html – Download information for Sybase ASE
• http://www.sybase.com/ase_1500devel – Downloading Adaptive Server Enterprise version 15.0 Developer’s Edition
• http://www.isug.com/Sybase_FAQ/ASE/section1.6.html#scview – Information on ASE Monitoring Server
• http://manuals.sybase.com/onlinebooks/group-as/asg1250e/monbook – Monitoring Server; Information on methods of booting the ASE Monitoring Server.

Looking for Linux performance monitoring and tuning resources to conduct analyzing work? ufsdump.org has a wide array of resource for Unix systems. In these two part PDF provided by ufsdump.org, titled, “Extreme Linux Performance Monitoring and Tuning”, ufsdump.org highlights the various components from CPU, memory and I/O that should be monitored and tuned accordingly. Concepts such as context switches, run queue, CPU utilization, time slicing and priorities are discussed for CPU. For memory, the PDF steps through performance concerns in Virtual Memory, Virtual Size (VSZ) and Resident Set Size (RSS), Paging/ Swapping, Kernel Paging with pdflush and kswapd. Lastly in I/O, the articles walks through concepts on reading and writing data for memory pages, major and minor page faults, file buffer cache, types of memory pages and writing data pages back to disk. All these concepts are great tips for performance tester in determine the performance of the Linux servers. To download the PDF, click on link below:

• Extreme Linux Performance Monitoring and Tuning – I
• Extreme Linux Performance Monitoring and Tuning – II

(Special thanks to Aravind Kumar for contribution, CSS Corp)

Check out this script from Bad Penguin.co.uk. This script at it’s latest version when placed in a Unix machine, collects performance statistics at desired intervals and convert them into CSV report which can be imported to a spreadsheet, database and LoadRunner Analysis to perform analyzing work. It uses standard unix utilities (iostat, vmstat, netstat and uptime) and has been tested on Linux and Solaris systems. The usage of vmstat, netstat, iostat and uptime is far more comprehensive than those used by rstatd.

Click on the link below to get a copy of the script!

performanceMonitor.sh

The script is used in two ways. One to generate statistics and two to convert to a CSV format. The below describes the command to execute.

What’s JMX in the first place!

JMX (Java Management Extensions) technology provides a simple, standard way of managing resources such as applications, devices, and services. Because JMX technology is dynamic, you can use it to monitor and manage resources as they are created, installed and implemented. (Source: Sun Microsystems, “Ch1 : Introduction to JMX Technology”)

A simple illustration of how the JMX architecture can be found in Wiki, “Java Management Extensions”. This will give you an overview of the different levels and how the communication work between them.

Additional references can be found in Sun Microsystems which we recommend browsing through to get an general understanding of this technology.

• “Overview of Monitoring and Management”
• “Java Management Extensions (JMX) Technology”
• “JDK 6 Java Management Extensions (JMX)”

Ok, now we’ve got an understanding on JMX. In the context of WebLogic, JMX is in place to monitor the resources usage of the server.

How does WebLogic (JMX) monitoring work in LoadRunner?

JMX specification is designed to monitor Java applications from another Java application. Therefore, in order for LoadRunner to monitor WebLogic resources via JMX, it will be required that LoadRunner to be a Java-application to perform the monitoring and collect the data from the MBeans. This is achieved by installing JDK of the same version as the WebLogic server and running a mini version of application server in the LoadRunner machine to collect the monitoring data which is reason the installation of JDK as well as the copying of the weblogic.jar file is required.

What can you get from WebLogic (JMX) monitoring?

You will be able to monitor the runtime state of the WebLogic via JMX from the Runtime MBeans. The list of JMX counters can be found in “WebLogic Server MBean Reference”.

If you need to see how much time is spent or breakdown on individual methods, SQL calls or even the chain of calls, you are looking at the wrong tool. You may like to explore Diagnostics for a solution. Refer to “What’s Diagnostics?” for more information

We’ve put together a list of references that will you understand the JMX technology and also the fundamentals that WebLogic (JMX) working with LoadRunner. We hope this information is useful to you. If you have additional information with regards to the monitoring, please feel free to write in to us.

MySQL database monitoring will require SiteScope to be deployed acting as a bridging collector before it can be displayed in LoadRunner. The connection is illustrated in the following.

The specified statistics retrieval mechanism utilizes Database Query monitor in SiteScope. This is done by logging into MySQL database via SiteScope Database Query monitor and issuing a SHOW STATUS command to retrieve the statistics. Example of the command is as followed:

show status where variable_name like ’innodb_buffer_pool_pages_total’

Therefore, prior this monitor setup, SiteScope should already be deployed or considered in the deployment. Refer to SiteScope setup for more information. The ports between LoadRunner and SiteScope, SiteScope and MySQL should be opened on TCP 8888 and TCP 3306 respectively by default. On top of this basic requirement, monitoring a MySQL database requires the use of JDBC Driver, which SiteScope uses to interact with the MySQL database.

In summary, what you need before the setup is as followed.

1. SiteScope to be deployed.
2. Sufficient license points for Database Query monitor.
3. TCP 8888 (default) opened two-way for LoadRunner to SiteScope.
4. TCP 3306 (default) opened two-way for SiteScope to MySQL.
5. Monitoring privileges for SiteScope to monitor MySQL.
6. JDBC driver to be installed on SiteScope.
7. Have knowledge of the counter you want to monitor.

Let’s go into the details!

1. Download the JDBC driver from MySQL Download Connectors page.
2. Uncompress the distribution file.
3. Among all the other files, find and extract a file with .jar extension to Sitescope/java/lib/ext directory. (Source: SiteScope Monitor Guide)
4. Gather information about the hostname of the server and the database instance prior this step. In SiteScope console, add Database Query monitor to the existing monitoring group for the server (refer to SiteScope User Guide for more information). You should have the Name the monitor as mysql-{status_variable} to standardize the monitors (e.g. mysql-innodb_buffer_pool_pages_total).
5. In the settings enter the following:

Fig. 2 SiteScope Settings

Frequencies

What SiteScope documentation proposes is:

- For most critical and common query 2-5min.
- Database statistics that change less frequently can be monitored every 30 or 60min.

(Source: SiteScope Monitor Guide)

As for me, what I feel is to run the query at about 15-30secs interval as typical LoadRunner monitoring, it is defaulted at 15 secs. However by doing so, it may create additional overhead for making the query.

Database Connection

• Enter in the following format: Jdbc:mysql://{database hostname}[:{tcp port}]/{database}
• E.g. jdbc:mysql://DB-MySQL01:3306/mysql

Database Driver

• Enter following: org.gjt.mm.mysql.Driver (This Database Driver only applies for MySQL JDBC driver)

Query

• Enter the Command Line for Query to Database
• E.g. show status where variable_name like ’innodb_buffer_pool_pages_total’
• You can also run queries that directly execute on table based on your needs!

Advanced Settings

• Enter Database User Name and Database Password for logging in of MySQL Database. I’m using root account to execute the query. I’m not too sure if there is any possibility of knowing the correct privilege needed to monitor. If you do, please let us know.

Upon successful connection, you will prompt with ‘Successful Connection’ and the time Sitescope took to Query MySQL Database.

If you need to monitor another MySQL counter, a new monitor needs to be created. Example, if you want to monitor the Qcache_hits, you will have to perform step 4 to step 5 again for this new counter. Do ensure you have sufficient SiteScope license points for another monitor!

This concludes the monitoring for MySQL database server. Point to note that the queries executed may cause overhead to the database server which may required you to balance the pros and cons to implement this monitor.

Parsing error.
Details: host localhost,port 1112, line: 1.
Reason: Invalid at the top level of the document.
.
Contents: java.lang.Exception:java.lang.SecurityException: [Security:090398]Invalid Subject: principals=[weblogoc.admin.mbean, Administrators] [MsgId:MMSG-47587]

Background information: I’m monitoring 4 WebLogic servers using WebLogic (JMX) Monitor in LoadRunner and adding their JMX stats into Controller. Upon adding the subsequent server, the error appeared after entering the username and password to the Controller. I’m using LoadRunner 9.0 and Weblogic 8.1 in this context.

The principle behind it is that the mini Web server launched by the Controller for monitoring WebLogic JMX listens on port 1112 by default (defined in {LoadRunner}\dat\monitors\weblogicmon.ini). This port is occupied and the mini Web server cannot be started according to Document ID 42701, Error: “Failed to connect to XML source on 1112″. This hinted me on the change of port. As suggested in the document, for every Weblogic server (JMX) that I’ve added, I change the port number on the file as below (Source: HP Openview Document ID 42701):

The solution is to configure the default port to a one not in use

1. Navigate to {LoadRunner}\dat\monitors.
2. Open weblogicmon.ini in Notepad.
3. Change

Weblogic=weblogic.jar
Port=1112

to:

Weblogic=weblogic.jar
Port=1114 {Or any other port that is free on the Controller machine}

4. Save the changes.
5. Close the “Add Measurement” dialogue and re-open it. You should be able to add the JMX measurements. Repeat from [1] to [5] in the event that when the error occured for subsequent WebLogic servers.

There are four implementations for monitoring without SiteScope and Diagnostics.

1. Direct
2. Open port on firewall
3. Use one Mercury Agent inside the firewall
4. Use two Mercury Agents: one Listener outside the firewall and one Monitoring Agent inside the firewall.

First, I presume you have a fundamental concept of how the monitoring works in LoadRunner. Bear this in mind so the discussion will be easily understood. If you are new to it, please refer to “How does monitoring work in LoadRunner?” to get a better understanding.

In [1], direct connection is required between the SUT (Server Under Test) usually inside a firewall; meaning the port is cleared and not blocked on the server. Also, prior to this the native monitor should already be working.

For example, Windows System Resources required port 139 to be opened. Therefore, ensure direct connectivity between the Controller and SUT on port 139.

In [2], where most environment is protected by a firewall, you have to lift the port that the Controller communicates to the SUT. Back to the same example, port 139 is to be lifted on the firewall to allow the monitoring data to be sent back.

Moving on to the 3rd and 4th implementation, the need for installing LoadRunner agents arises and this increases the complexity (but a little effort in understanding will definitely reduce it).

In [3], the full installation of LoadRunner (Controller) will install everything on the machine, resulting in the machine to have the LoadRunner agent. This agent is called the MI Listener, talks to the MOFW agent (Monitoring Over Firewall) for monitoring data on port 54345 (strange number huh?).

The MOFW agent which is actually installed (recommended standalone) inside the firewall. It also utilizes port 54345 to communicate back to the MI Listener. Put it simply, Controller talks to the MOFW on port 54345 in a two-way traffic.

From the MOFW agent onwards, it is the typical native monitoring ports that is required to be opened. For example, with the Window System Resources, the communication is as followed: (1) Controller to MOFW is 54345 and (2) MOFW to SUT is 139. Simple right?

In [4], the concept is similar to [3], however we extend the MI Listener as a separate machine by itself. This way, you will need to have port 50500 between the Controller and MI Listener. The communication between the MI Listener to MOFW now, is 443, the default SSL communication.

Example with Windows System Resources, you will need the following communication enabled: (1) Controller to MI Listener is 50500, (2) MI Listener to MOFW is 443 and (3) MOFW to SUT is 139.

Why do we have these monitoring implementations? There can be various reasons to it but will the following are some suggested or experienced so far:

1. For [1] and [2], it’s a simple setup through direct connection. The cavaet is, it’s security implication on opening the native ports which is usually disallowed. Example, port 139 datagram used by Windows System Resource Perfmon, is typically not allowed due to the security implication. Unless, the Controller and the SUT belongs to the same side of the firewall.

2. For [3], similar to [2], it attempts to open the port for communicating to the MOFW agent inside the firewall. In this implementation, it could be a situation that the agents had already been installed inside the firewall and all the Controller have to do is to connect to them. 54345 is not a popular port to be hacked I guessed.

3. For [4], is an attempt to use HTTP SSL (443) to communicate to the MOFW agents inside the firewall. This implementation proposed a secure way for communication as compared to lifting native monitoring ports (eg. 139).

4. For [3] and [4], you may have a dedicated machine to be a MOFW agent to collect monitoring data for every load test. This way, it’s more effective to just clear one port (either 443 or 54345).

5. For Unix System Resource monitoring which uses a dynamic port, the methods [3] and [4] is clearly more efficient for any port clearance procedures where the Controller connects to the MOFW on 54345 and MOFW connects to the Unix box on the dynamic port (no port lifting is required except 54345 is needed on the firewall).

So well, in summary, we covered the implemtation models in a general overview with the ports required and the benefits and cavaets each of them may have. This should be useful to you in planning the monitoring setup as well as

The objective of LoadRunner is simple; to load an application (set upon a certain environment and hardware), monitor the servers that is in the environment housing the applications, and lastly collect the results for analysis. LoadRunner sells itself as a load testing tool that is not intrusive in any environment with the capability of not requiring installation of agent programs on the servers in the environment (anyway, most monitoring tools work in this way).  How is this achievable?

What LoadRunner does is to collect information of monitoring counters of the native monitors. Native monitors in this context mean the programs/modules that is in-built with the system that you are monitoring. Example of the commons, for monitoring Windows System Resources, LoadRunner draws information from the in-built tool, Perfmon which is available for almost all Windows OS. For Unix System Resources, LoadRunner uses rstatd daemon of Unix OS for monitoring.

For non-System Resources, such as Web Server Resources, Apache Web Server, the server must be configured to allow monitoring before LoadRunner can draw monitoring data out. This applies for Web Application Server such as BEA WebLogic and IBM WebSphere.  For Database Servers, there is an additional step that requires a client to be installed for the monitoring to be successful. Example, for Sybase Database, the Sybase AES client (if I do not remember wrongly) needs to be installed on the Controller; Oracle Database requires the V$SESSION table to be working. With the exception of MS SQL, where the counters are in-built into Perfmon.  Client programs that are required to installed also applies to WebLogic JMX Monitoring.  The list can go on however, the concept of monitoring is the same and applies for other monitors that is not mentioned in the above examples too. (More information of the monitoring requirements can be consulted in the Monitor Reference provided with each installation of LoadRunner.)

Therefore, before any monitoring, the native monitors must be working prior to the test. If the native monitors are not working, the monitoring by LoadRunner will certainly fail. Usually, the monitors failed on the following reasons which I guessed for all load tester should be aware of. Usually, I give myself 1 to 2 days for monitoring setup before the load test.

• No support for the native monitor; Check out the support level on the documentations, knowledge base articles and the support (if the information is not readily available online).
• Configuration problems on the native monitors; Check out and verify the configuration settings on the native monitors against the documentations and knowledge base articles.
• Environmental Issues such as security, firewall and policies; Ensure connectivity between the Controller and monitoring machines (dependent on the port number and how the load testing was implemented) and the access rights given to the load testing user.

A sound understanding of the native monitors (Perfmon, rstatd, etc) such as its configuration, type of monitoring data it can provide, and how it generally works will serve a good foundation before the load test. (Therefore quite a fair bit of reading and googling will do you good!) This will be useful for the load tester to relate the required configuration to the System Admin/Engineer, Network Engineers, etc and get the job done with minimal effort. (You need to talk their “language”!)

In summary, LoadRunner collects monitoring data from the native monitors. This requires the native monitors to be configured properly prior the load test. However, they do fail at times and it’s possible to scope down to either (1) no support, (2) configuration problem and (3) environmental issues that can resolved with adequate planning. Furthermore, a sound understanding of the native monitors will be useful when conveying instructions to the system/network team in configuring them.