How Many Load Generators Do We Need For Performance Testing

As a general rule, the more complex a protocol is to script, the more the protocol relies on monitoring the network stream and the less that it needs to understand the actual application under test’s user interface. It also tends to be true that with this type of protocol, the more virtual users of that protocol type will run on any specific load generator. 

This has led to the development of the Protocol Complexity Matrix and can be visualized by the graphic below.

This is a representation of common results and you may find that your individual use of a protocol may provide more or less virtual users than indicated.  This diagram shows you that if you want to make the job of scripting easier, it comes at the cost of memory.  Using more memory makes the protocol less scalable up to the point that you reach the GUI protocol, which out of the box has a 1:1 ratio between virtual user and load generator.

The goal of any organization is to find the right balance.  Nearly any application can be scripted with the Winsock protocol, regardless of technology used by the application under test. However, working with raw socket commutations can be difficult to learn and requires a great deal more time to develop and test; thus making the solution impractical for many projects. This is why higher-level protocols have been developed, to save time.

What is not easily represented in the matrix is another fact; the more reliant you are to the network stream, the more susceptible your scripts are to changes in that stream.  So the application that is scripted with QTP can undergo massive changes in the background and the scripts for that application can still function without modification.  But these same changes to an application recorded with the Winsock protocol could render the entire script useless and a throw away. This fact brings me to my next question:

How many users can my Load Generator run?

In previous versions of LoadRunner and Performance Center, a protocol footprint spreadsheet was distributed by HP to provide a guideline for how much memory a script of a particular protocol would consume.  The problem was recipients of the footprint spreadsheet thought of it as definitive rather than prescriptive. It is much better to use a repeatable process and a formula to mathematically estimate the total number of users a load generator can run on a per-script basis. The process and formula outlined below should be used for every script that you intend to use in load testing, and every load generator that will participate in the test as well.

Here are the steps you should take these scripts that you intend to use in a load test:

1)      Develop your script and ensure that it will run with multiple users, multiple iterations.  This is the normal development of the script of any protocol.

2)      Create a load test using the script that you want to examine with one virtual user and execute the test. Use a delay of a few minutes in starting the script and monitor the load generators available MB of RAM and note any decrease when the virtual user starts.  This is the “First VUser Memory”.

3)      Modify the load test to use five – ten virtual users and execute the test again.  Keep the same delay for the start and have each user start one minute apart.  Monitor the load generators available MB of RAM and note the drop in RAM as each user starts.  Average the amount of RAM that the second virtual user on up uses and this is the “Each Additional VUser Memory”

4)      Determine the total RAM available on the load generator This is the “TOTAL LG RAM”

5)      Subtract 700 – 750 MB of RAM for the OS

6)      Determine what is 75 percent of the remaining available RAM

7)      Subtract the “First Vuser Memory”

8)      Divide what is left by the “Each Additional Vuser Memory”. 

9)      Add one and you have the “Total Virtual Users that will run on that load generator”

10)  Repeat this process and formula for each script and every load generator that has a different level of memory.

The equation will look something like this:

This equation will work for all but a handful of protocols.  These are the protocols that this equation may not work: Citrix, RDP, any protocol with the term GUI in it, and QTP\UFT scripts.  That is because these scripts can further be limited due to GDI resources.  These are graphical resources that are an attribute of the video card installed on the computer.

GDI resources are not able to be monitored and they cannot be adjusted without replacing the video card.  When GDI resources are consumed any virtual user that attempts to run on that LG will fail.

To learn more about LoadRunner and Performance Center and how to estimate the total number of users a load generator can run on a per-script basis visit the product homepages here.

As an example.  If you have a 4GB LG machine the TOTAL LG RAM is 4,000

If your First Virtual User Memory is 10MB and Each Additional Virutual User Memory is 3MB then the result would be:

4,000 - 750 = 3,250

3,250 * .75 = 2,437

2,437 - 10 = 2,427 ( 1 virtual user running)

2,427 / 3= 809

809 + 1 = 810 (Total Virtual Users)

Hope this makes the process a little clearer.