SilkPerformer

From the TCodes, it does a reasonable job of monitoring the SAP system on the overall performance.  The TCodes should cover (a) SAP Buffer performance, (b) Dialog Step performance (response time), (c) database performance at the database buffer cache and shared pool level and (d) operating system performance such as processor and memory.

Through this setup, SP Controller merely make a normal connection like a typical SAPGUI Front-end.  Therefore, typical ports used by the SAPGUI Front-end such as TCP32## and TCP33## are required.  Furthermore, the user account used for the monitoring should have sufficient privileges to access the mentioned TCodes.

We will go through the approaches in LoadRunner and briefly touch on the similar options available in SilkPerformer.

[1] Using Web (HTTP/HTML) protocol

The actual approach that was utilized in the load test. Back to the basics, the communication channel used by ArcGIS Desktop (ArcMap) is Web Services over HTTP, that’s SOAP/HTTP. The Web Services include finding the service provided by a ArcGIS Server, calculating and loading of the map. Therefore, normal means of using Web (HTTP/HTML) should suffix. For licensing concerns, Web (HTTP/HTML) protocol is grouped under the Web 2.0 Bundle license.

Further to this, (a) options for capturing the Content-Headers should be enabled and (b) the old recording engine in VUGen should be used. An article will be posted on the sample recording of ArcGIS Desktop (ArcMap) using Web (HTTP/HTML).

[2] Using Web Services protocol

As communcation is using Web Services, the most clear-cut protocol will be Web Services protocol which you can parameterize it with the Wizard. Refer to “Is it possible to load test Web Services without front-end?” . Understanding of the location of the WSDL file and parameters to passed in will be required for this approach. Of course, this requires the SOA Bundle license in order for load execution in Controller.

[3] Using .NET protocol

Using .NET protocol, you will need to know the assemblies and methods used by ArcMap. Therefore, go through the entire list of assemblies and methods that maybe involved in the communication between the client, ArcGIS Desktop (ArcMap) to the server, ArcGIS Server.

[4] Using Visual Studio AddIn

Using Visual Studio AddIn (after the installation of the plugin), you can create the user by calling methods from the ArcGIS SDK. Our evaluation of using ArcGIS SDK was using the C# language and making calls of zooming and panning activities. Unlike the conventional approach of knowing the communication between the client and server, and ignoring the client-activities, in this approach, we focus on the client-activities and ignore the communication used.

Similarly in SilkPerformer (refer to “What’s SilkPerformer” for a review of the product), there are four approaches to record and replay of an activity in ArcGIS Desktop (ArcMap) that are almost the same to LoadRunner with of course a different branding and licensing mechanism.

1] Using Web License

The basic function provided by SilkPerformer with same capabilitis as the Web (HTTP/HTML) protocol in LoadRunner.

[2] Using SOA Edition

Similar to SOA Bundle in LoadRunner, the SilkPerformer SOA Edition allows the reading of WSDL file and generation of the APIs based on the WSDL file.

[3] Using .NET Explorer

Similar to .NET protocol in LoadRunner, SilkPerformer will require the knowledge of the assemblies and methods being used by the user in a normal context. Using the .NET Explorer, SilkPerformer will generate the API based on the selected assemblies and methods.

[4] Using Visual Studio Plugin

Similar to Visual Studio Addin in LoadRunner, an additional plugin provided by SilkPerformer is installed into Visual Studio. And by using the desired langauges (Java, Visual Basic or C#) and the knowledge of ArcGIS SDK, you can create powerful scripts that emulates the activities of the real user.

If you have other approaches in load testing the application, please let us know and we can share it out with everyone here. Our understanding that there are other alternatives such as EggPlant Load Tester and using of RDP (Remote Desktop Protocol in LoadRunner 9.0 onwards) to load test ArcGIS.

SilkPerformer licensing mechanism is based on three mainly groups. Namely Web, Standard and Premium. The differences in each group is the type of technologies being used in the group. For example, Web encompasses the commonly used protocols while Standard focuses on additional components. This is somewhat similar to LoadRunner in terms of grouping the protocols in bundles. The following table is the full list of what the group covers in SilkPerformer 2008 (which is also available in the Release Notes).

As briefly mentioned in “What’s SilkPerformer?”, SilkPerfomer allows the capabilities of checking out the license with the license server, SilkMeter. This is great in a way that there are no limitation on the “Controller” and you can install it anytime. Furthermore, you can fully maximise the “controller” resource when it is not needed for an existing project.

Take note that there are still some limitations on the use of SilkPerformer on the group licenses. The groups does not cover SOA applications. If you like to test the Web Services via WSDL files, you will require the SilkPerformer SOA Edition. Similarly with LoadRunner that requires the SOA Bundle license to work for Web Service protocol.

Also, for unit-level testing tools available from SilkPerformer, namely .NET Explorer and Java Explorer, will require additional development licenses for them (the explorer tools) on top of the vuser type license. This can be thought as the same as using Quick Test Professional (QTP) scripts for LoadRunner in a load test where it requires both licenses on the QTP and LoadRunner to be functional.

As both LoadRunner and SilkPerformer offers the same functionality, what SilkPerformer is appealing is it handles load test in a project approach. With this approach, each load test is saved as a project with various settings of profiles and workloads. This is a great feature when you want to track the load test from a customer/project perspective. Furthermore, within a project, for every load test that was executed, a result directory for each run is created. This is useful in tracking the number of runs that were conducted. In this way, there is a structural approach in managing the tests that were conducted (especially useful when you are running multiple load test). As for LoadRunner 9.0, an additional scenario profile setting was introduced in Controller. This feature allows the different configuration of scenario profiles similar to SilkPerformer. Whereas the generation of a results directory in LoadRunner remained the same in the Controller results setting.  SilkPerformer has another great feature, which is resource management of the load generator. By defining a pool of load generators, the load tester is able to have an overview of the resources that is participating in the load test such as if any load generators are being over-worked. This is almost similar to the Resource Pool in Performance Center and is not available in LoadRunner.

During the execution, SilkPerformer allows monitoring of the resources in terms of CPU usage in the Load Generator. This is beneficial when differing problem in resources on the Load Generator or the SUT itself. Furthermore, feature allows capacity planning for load generators. SilkPerformer also allows monitoring on the progress of the vusers in terms of percentage completed by each vuser. This is a great feature that allows the load tester to estimate the duration of the load test and forecast the end time of the load test.

For verifying replay of script, the Silk TrueLog used in SilkPerfomer has better capabilities than Vugen Runtime Viewer. If offers replay of the screens and providing views of the data in and out from the client and server with one single component instead of toggling with runtime viewer and client-server request views.

SilkPerformer allows monitoring similar to LoadRunner. However, as compared with LoadRunner, it has better control and ease-of-use in setting up monitors. On top of this, SilkPerformer requires additional logging for non-default monitors unlike LoadRunner, which automatically logs the data when the counter is been added into the monitor list. Also, SilkPerformer is restricted to a defined set of custom monitors namely JMX, SNMP, Perfmon and Rexec unlike LoadRunner, which provides more monitors.

SilkPerformer utilizes Silk Performance Explorer for analysing. However, it’s capability to merge, manipulate, handle external data is not as flexible and robust as LoadRunner Analysis. The analysing feature is an important component used by the Performance Team to determine problem causes.

SilkPerformer have the similar license mechanism as LoadRunner, which is bounded by a single host. However it features an additional mechanism that allows the option to “check-out” license to the installed application. This is useful in an organization context in managing our performance test in different environments and will be discussed in a later article.