Scalability testing

Increment loads

This is why when creating a scalability test, it is important to scale up in increments. These steps can be split into small, medium and high loads.

We must scale up in increments as each stage tests a different aspect. Small loads ensure the system functions as it should on a basic level. Medium loads test the system can function at its expected level. High loads test the system can cope with a high load.^[5]

Test environment

The environment should be constant throughout testing in order to provide accurate and reliable results. If the testing is a success, we should see a proportional change in performance. For example, if we double the users on the system, we should see a drop in performance of 50%.^[6]

Alternatively, if measuring system statistics such as memory or CPU usage over time, this may have a different graph that is not proportional as users are not being plotted on either axis.

Unproportional outcome

In Figure 1, we can see a graph showing a resources usage (in this case, memory) over time. The graph is not proportionate but can still be considered a passed test as initially there is a ramp up phase as the system begins to run, however, as more users are added, there is little change in memory usage.^[7] This means that the current memory capacity can cope with all 3 stages of the test.

Proportional outcome

In figure 2, we can see a more proportional increase, comparing the number of users to the time taken to execute a report. With a low load of 20 users, the average time is 5.5 seconds, as we increase the load to medium (40 users) and a high load (60 users), the average time increases to 9.5 and 18 seconds respectively.^[6]

In some cases, there may be changes that have to be made to the server software or hardware. Once the necessary upgrades have been made, we must re-run the tests to ensure the upgrades have been effective in addressing the issues previously raised.^[8]

When we have a proportional outcome, there are no bottlenecks as we scale up and increase the load the system is placed under.^[9]

Advantages and disadvantages

There are advantages and disadvantages to both methods of scaling. Although scaling up may be simpler, the addition of hardware resources can result in diminishing returns.^[11] This means that every time we upgrade the processor for example, we do not always get the same level of benefits as the previous change.

However, horizontal scaling can be extremely expensive, not only the cost of entire systems such as servers, but we must also take into account their regular maintenance costs .^[11]