LoadRunner – The parameter file is stored in the script directory as .dat file extension.  It can be opened with a simple text editor like Notepad as it is stored in pure text format.  Therefore, you can manipulate the files via the Notepad.  You can also use office applications such as Microsoft Excel or OpenOffice.org Calc to work on the file.  This greatly the amount of work required for maintaining a long list of data.

Quick Test Professional – The parameter file is also stored in the script directory as an excel file, which will be an .xls file extension.  Being an excel file, it can be opened by office applications like Microsoft Excel.  Again, if you do not have money to purchase a licensed Microsoft product, you can always use open source products like OpenOffice.org Calc as an alternative tool as it is still able to work with .xls files.

We hope that by providing you such tips, you will be able to work efficiently with large list of data (that is especially prevalent in LoadRunner)!

• What is the total user base?  What is the concurrent user base? – Now after working the maths out, do you required a load test.  Example, is a 5 concurrent user base an effective load test?  Is it worth the effort to generate a script that emulates 5 concurrent users?  Your customer may want you to run a load test for 5 users even though it is not effective to run that kind of load test.  However, if it is in the contract, you may have obliged it.  If they are willing to pay for that service, then it’s a plus point!
• Do the available load testing tools support your system in a load test? – A very strange question to ask.  Most tools are unable to support asynchronous application and map applications.  Either you may have to purchase more licenses to run the load test like in LoadRunner, find alternative sources or write scripts to emulate the load.  There are multiple ways to run a load test however may require additional effort (which translates to additional time).  Most tools are useful in coordinating and reporting which your scripts may not be able to achieve.
• What is the environment that you will be conducting the load test? – At times, your load test may be conducted in a scale-down production environment which you may know.  The load test results may not be a good representation of the true performance of the load test.  However, most of the times, running in production is a no-no and it is good to align everyone’s (stakeholders, customers, etc.) expectation of the load test.
• Where do you want the load to be generated to? – For a web application, there are multiple layers that you can access the system.  You can access the normal URL that can be the virtual IP address from a central web server.  You can enter at the load balancer or the web server layer.  Lastly, you can enter at the application server layer.  Where you want the load to enter can help you determine the layer that can be causing the performance problem.  By stepping up from the application layer, you can determine stage-by-stage the layer as the source of bottleneck.
• What do you want to monitor? – There are so many things that you can monitor in an infrastructure.  Which one do you want to focus on your analysis effort on first?  Do you want to monitor all sub-systems or target your effort to the most possible source such as the database?  Setting up monitors takes effort.  What’s more tedious is the effort required to analyze the monitoring results that you have collected!  Therefore, do take note and factor the duration and effort!
• Do you know the backbone network infrastructure? – This information is usually not available to a load tester.  But if you know this, you can least prepare your team that the load test will bound to meet (or not bound to meet) performance issues during the load test. They can be ineffective load balancing, small bandwidth available, invisible proxy that limits bandwidth, etc.  If you do not know this information, it is best to work on those you know such as the method previously mentioned by running the load test stage-by-stage.
• Is your load test conducted in a wireless environment? -  A wireless environment may be limited by the number of access point and the bandwidth available per wireless network. You need to know if the wireless will become a potential bottleneck in the amount of clients that can be generated into the system before your load test.
• Do you have a base set of data to use in a load test? – Most often after a load test, the data in a database will likely be messed out.  Therefore, it is advisable to create a backup of a working set of data in the database for you to reload after each load test.

Now, if you are proficient in LoadRunner, you may miss out on the details of how to go creating the start-stop transaction for the GUI Vuser script as the details are actually found in the QTP User Manual under the section, “Measuring Transaction” of HP Quick Test Professional User Guide – Volume 2 instead of LoadRunner manual.  In anyway, this is possible and only achievable via amending the scripts in QTP because Vugen will not be able open (read) QTP scripts.

Two ways you can insert a transaction.

• Via Keyword View – You can use Start Transaction icon > Insert Transaction at the step that you want the transaction to start measuring.   Subsequently, you will use Stop Transaction icon to stop the measurement.
• Via Expert View – You can use Services.StartTransaction “Transaction Name” and Services.EndTransaction “Transaction Name” to encapsulate where the start and end measurement.  “Transaction Name” is the name you give to defined for the transaction and this will appear when you run the GUI Vuser script in LoadRunner.

With the above information, you should be slightly more familiar with Quick Test Professional (QTP) integrating with LoadRunner.  However, again there may be more things that need to be considered and I may have left out.  Do feel free to feedback and experiences and anything that I missed out when using QTP in load testing!

Quick Test Professional (QTP)

During replay, the script will then be replayed like a Windows VB Script, launching the windows and activating the buttons as what is being recorded.  A sound knowledge of VB script is actually helpful in maintaining the scripts however not necessarily needed as QTP is relatively easy to use.  A tip here is you can actually use VB script in Windows to run automation without even using QTP.  It works the same way but a little low-level work is required and you need to understand the various ways the windows and objects are called in the Windows environment.

In QTP, you can add check points to determine if your application response as intended.  You can also use output values similar to the concept of correlation in QTP to handle dynamic data.  The output values are also used to be input values for the next step depending on how the test case is designed.

As for the licenses, they worked in two ways.  Either you use a Seat license or a Concurrent license.  The seat license is tied to the machine that installed the license while the concurrent license requires a license server to be available for it to retrieve the license information.  The license will decide the type of application you can record; generally called an Add-In.  QTP has various Add-In to suit the needs of the customer.  The common one is Web, Java script and ActiveX.  Other supported technologies that require an Add-In are Java, Oracle, SAP, etc.

Of all the virtues of using QTP, you will still need to be careful on the following and use it correctly:

• Resolution of the recording machine and replay machine MUST be the same – QTP rely on the resolution of the machines to recognize the coordinates and size of the objects.  Problem will arise when the resolution of the recording machine and replay machine is different.  Example, the script was recorded in 1024 x 768 resolution while it was replayed in a 1280 x 800 resolution may result in the script generating error as the objects have changed their coordinates.
• Iteration of the script requires the start and end point to be the same – For a script to iterate successfully, the script should be recorded in a way that the start and end point are the same.  Example, the script starts off with the login page and proceeds with login and some activities then logout to return to the login page.  In this manner, the script will be able to iterate with the next set of value such as a new login account.
• Context-sensitive – Context sensitive menus may require the use of Object Spy feature in QTP.  Context sensitive menu such as mouse-over menu that will show the sub-menu.  This will require pressing “Crtl-Key” when using the Object Spy and inserting the action (and object) into the script.  However, we recommend recording in a lower granularity (in analog) to capture as much objects as you can before proceeding to use the Object Spy.
• Preparation of initial test data – Most often people ignored the need for a baseline set of data for testing.  After each functional (regression) test, the data needs to be refreshed to a clean set before the script can work again.
• Still dependent on a well-designed test case – To make the tool really useful, the tool still highly depends on a well-designed test case.  Without a well designed test case, the downstream activities such as recording and modifying may not serve its purpose well.

The information should give you an overview of how QTP works and what you should take note after purchase (or even before purchase).  We hope that it will be useful to you and do share with us on anything related to QTP.

• Beware of “Thread.run()” versus the real intend of “Thread.start()” – We may unknowingly code the thread as “run()”.  This will cause the thread to run sequentially instead of concurrently. Beware of this!  Use “Thread.start()” to allow the thread to run concurrently.
• Use “String.length ()  == 0” instead of “String.equals(“”)” – Both constructs do the same thing of comparing if any characters exists for the string. However, the first construct, “String.length() == 0” is more efficient than the latter one.
• Avoid calling “String.toCharArray()” – “toCharArray()” method is inefficient as the method reallocates the entire array stored in the string.  However, sometimes this is necessary.  Therefore, as a rule, avoid using “toCharArray()” method if possible.  If not possible, do use it with care.
• Place “try/catch/finally” block outside loop – Placing “try/catch/finally” blocks inside loops can slow down the execution of code. Do take note that moving the “try/catch/finally” block outside the loop can change program behaviour. An exception that has the “try/catch/finally” block outside the loop will terminate early.  On the other hand, the “try/catch/finally” block inside the loop may cause the program to continue iterating over the loop even after an exception has occurred.
• Use “StringBuffer.Append ()” instead of “+=” operators for concatenating strings – “Append()” has better improvement over “+=” operators and its recommended to use it instead of “+=” operators for concatenating strings.
• Do not use “StringBuffer” as constants – Dynamically resizable strings are unnecessary for constant strings (as they do not change).  Instead, use “String” for non-modifiable string constants.
• Do not open or close JDBC connections in loops – It is inefficient to open and close a connection in loop.  Instead, the connection should be opened before the loop and closed after the loop to avoid redundant calls make to the database.

• QTP scripts are developed in QTP not in LoadRunner Vugen – You won’t be able to record and replay a QTP script in Vugen.  All the recording and modification has to be done on QTP.  Once the modification of the QTP script is completed, it will be ported into LoadRunner Controller as GUI Vuser for load testing.
• QTP is required to install in the Load Generators (LG) – The load generators will be required to install QTP as they are used to run the scripts.
• Load Generators will be required to set the same resolution to the machine that developed the QTP script – As QTP is object sensitive, the resolution of the LG is required to be the same with the machine that was initially used to develop the QTP script to avoid any problems arising from missing objects (due to a different resolution)
• [1] GUI Vuser license is required for LoadRunner and QTP concurrent license is required for QTP – This is the 1^st setup approach and budgeting for your licenses.  The total amount of Vuser that you wanted to generate will be the same for both LoadRunner and QTP.  Meaning, if you want to run 50 concurrent users using GUI Vuser protocol, you will need 50 GUI Vuser license and 50 QTP Concurrent license.  In this setup, the LG will create 50 instances of QTP to run the GUI Vuser.  This is described in “What licenses are required to run a scenario with GUI Vusers” from HP Software Support.
• [2] GUI Vuser license is required for LoadRunner, QTP seat/concurrent license is required for QTP and Windows Terminal License is required for Windows – This is the 2^nd setup approach and budgeting for your licenses.   For a 50 concurrent user load test, you will need 50 GUI Vuser protocol, 50 Windows Terminal License and 1 QTP license.  In this setup, the QTP license is merely used to develop the script.  The LG will create 50 remote desktop connections and each of these connections will run 1 GUI Vuser.  This setup utilizes remote desktop functionality to emulate the virtual users.  I believed that HP Support does not really state about the licenses required from Windows Terminal and this is additional cost for you to factor if you are going though this mode.  Personally, I experimented using Windows Server 2003 (that is capable of 3 remote connections by default) with this approach and it works for me.  On the server, the remote connections are established and the QTP script launches the browser for testing.  (I wonder if this is permissible in the first place and like to know if anyone else is doing it.  If you got some input on this, please feel free to shout it here!)

With the above information, you should be slightly more familiar with Quick Test Professional (QTP) integrating with LoadRunner.  However, there may be more things that need to be considered and I may have left out.  Do feel free to feedback and experiences and anything that I missed out when using QTP in load testing!

• Handle the appropriate exceptions (checked); do not leave it to the unchecked exceptions – When an exception is thrown, the runtime system will exhaustively search all the methods on the call stack for the appropriate exception handler.  This is expensive and thus we want to minimize the search in the call stack in finding the appropriate exception handler to improve performance which can be achieved by throwing and catching the appropriate exceptions (checked exceptions).  Do not leave exception handling as an after-thought or code-as-you-go during development.
• Do not incorporate exception handling as control flow – As mentioned earlier, exceptions are expensive and should only be used for abnormality (planned and unplanned).  An if-then-else construct will perform better without the exception handling as there is no need to transfer control to locate the appropriate exception handling.   As such, review the codes and remove any throw statements from the coding that are used in an if-then-else construct.
• Avoid and remove the use of Reflection API in the codes – Reflection involves types that are dynamically resolved, which restricts optimization in certain JVMs.  Consequently, reflective operations have slower performance than non-reflective counterparts.  Therefore, unless there is a dire need for reflection to be used in the java program, it is best to leave it out if you need a high performance program.
• Let the system decide the garbage collection – If possible, remove any attempt to call system.gc(). system.gc() triggers a full collection , which includes tracing all live objects in the heap and sweeping and compacting the old generation. This can be a lot of work. In general, it is better to let the system decide when it needs to collect the heap, and whether or not to do a full collection.  Note: system.gc() merely signals (marks) the java program to do garbage collection, but it will not be an immediate garbage collection taking place.
• Reduce memory by avoiding the use of finalizers – An object is in the finalized state if it is still unreachable after running the finalize method, if any, has been run.  The finalized object is then awaiting deallocation and deallocation will not take place until the finalizer is run which is dependent on the VM.  The use of finalizer will extend the lifetime of the finalized object and it can be counter-intuitive when no deallocation takes place and when you intended to have the object being short-lived.   In order to avoid these short-lived objects being extended during the lifetime of the heap memory, it is best to consider the use of finalizers or free the resources explicitly.   (Source: Appendix A, The Truth About Garbage Collection and Java theory and practice: Garbage Collection and Performance)

1. Do you have sufficient hardware? – How may load generators are required for the entire load test?  Refer to this document from HP Support, “Is resource (cpu and memory) footprint information available to help in estimating the load generators required for load testing with LoadRunner?” for estimating the load generators required by multiplying the observations with the total number of virtual users generated in the test.   In addition to the processor utilization recommended in the document, do watch the memory and network consumption of the load generators.   The network interface card (NIC) can become a bottleneck for load generation.  A 1GB NIC will definitely generate and receive more load than a 100MB NIC.  Therefore, do a check on the machines assuming the load generator roles and make appropriate measures to prevent the NIC in becoming the bottleneck.   With the above, you will get a rough gauge of the required resource for your test.
2. Do you need spare hardware? – Do cater some spare machines if additional load generators are required, such as an additional of 5 load generators.  This is to stand by in the event of increasd virtual users and machine malfunction.
3. Remember that it’s a team effort! – Can you secure human resources for the load test?  You will need the web, application and database server teams to be around to observe the system behavior during the test or to assist in any reboot or refresh activities for the system.  Network and firewall/security team may be required to ensure that connectivity is allowed for the load test.    Therefore, in the planning, you have to get the relevant parties to commit the human resources!
4. Installation of LoadRunner – This require administrative or power users privileges in order for installation to be successful.  That raise another set of questions: Will you be able to install LoadRunner in the premises?  Who can you look for to install the LoadRunner program?  Do you need clearance for installation?  Is the system being hardened to prevent installation?
5. Offshore Installation – Is offshore installation required for the load test setup?  Can you perform the installation via remote connection or do you require someone from the site to assist in the installation?
6. Running of JVM/.NET Probes for Diagnostics – This will require installation and instrumentation on the application server, meaning it will be intrusive to the system as a whole.  Do you require additional permissions for such installation?
7. Post-installation – Do remember to test if the installation is complete and the programs are working before moving on to the next phase.  Remember to do a connectivity check between the load generators to the controller to ensure they communicate prior the load test.
8. Working in different environments – Are you required to work in different stages of a load test project?  Example, you are required to run the test in (A) the  data center followed by (B) the local client segment and lastly (C) the remote client segment?  In this context, how do you ensure the scripts worked from A to C?  It will be easy if it is a change of hostnames and URL addresses that is being used in these environments as it can be achieved through parameterization.  But what if there are additional authentication mechanism introduced in environment B and C?  Authentication mechanisms such as additional cookie in the clients are added into the URL or 2-factor authentication can render the scripts developed in environment A useless!  How do you go about it?  Should you cater more time to re-script for the worst scenario or is it better to disable the authentication mechansim during the load test?

These are some basic considerations we usually asked our ourselves and clients.  Hope it will be useful in your planning.  What are other considerations make for a load test for you?

In this post, we will not be covering on the diagnostics of the application server, but we will cover the basic signs of database bottlenecks at the SQL statement level.  The 1^st symptom is a collective symptom that the application server (and all other servers) with the exception of the database server in the architecture are performing reasonably well.  E.g. memory available is well maintained without any exception decrease of this value, the processor % time is always low for the servers.

The 2^nd symptom is a constant high utilization of the processor % time and low memory usage (page faults, page reads and available memory) in the database server.  This provides a tell-tale sign that the database server is processing some instructions, likely to be SQL statement that is taking too long.  It would not be the database buffer or SQL buffer that utilizes the memory as the consumption is low meaning reuse of the buffer is high.

Now with these two symptoms, you are ready to go down deeper.  Access MS SQL and run the following query:

This query will sys.dm_os_schedulers will give you information of how many runnable tasks that exists.  Values other than zero indicate that tasks are waiting to run, and high values are an indication that the CPU is bottlenecking your performance.  Another query that directly targets at SQL statement is the following:

This query shows which batches or procedures are consuming the most CPU and will also include the actual SQL statement. The query aggregates by a specific plan handle. If the plan handle contains more than one SQL statement you must drill into each statement to determine where the greatest CPU contribution comes from.

Codes illustrated in this post are taken from DotNet Bob on SQL.

How can we build a high performance web site through coding then?  Here are some basic tips you should note during development which should not be a big hassle for you to make changes anytime!

1. Remove any unused variables – The first thing and most simple thing you can do is to avoid creating unused variables.  Use static code analysers and look for variables that are initialized but not used.  Sometimes, in the midst of development, we leave out variables that aren’t used anymore and this can be removed to converse memory.

2. Free all resources – All created resources and variables should be released once they are not used.  This helps converse memory.  This will help optimize the memory usage and thus reduce the need for garbage collection cycle to take place.  Resources such as database connections and files are simple examples that you should close and release the resources.

3. Free all resources in the ‘finally’ block – Similar to [2], we want to free all resources.  However, this is catered to “unhappy” flow of the program code.  Free all resources that are used when exception arises if applicable.

4. Avoid calling methods in loops – This can be circumvented when the compiler optimizes the code.  However, if it doesn’t the loop condition will be calculated for each iteration during the loop

5. Avoid methods that may cause memory leaks – Methods such as split(java.lang.String), split(java.lang.String, int), substring(int), substring(int,int), nextElement(), nextToken(), nextToken(java.lang.String), group() and group(int) are candidates of memory leak.  There are quite a fair bit of search results on the methods that causes memory leaks.  You can refer to “Strings and Memory Leaks” from The Java Explorer for some information.  Avoid such methods whenever possible.  If not, do use it with care.

More to come on coding for high performance applications!  Share with us on your performance coding tips here too!