Measuring
How?
If you cannot measure you can’t optimise it, so we need to measure first.
How can we do it? We can do it in several ways. We need to measure page performance accurately.
You need to get the right loading time.
We need to try to be unobtrusive because otherwise, we might create some disruptions to our customers.
So let’s think about you went to launch a JMeter test, or an AB test the performance of the website we will dramatically
decrease, so let’s try to be unobtrusive. We also need to have a good sampling, we can’t just benchmark everything; we
need to sample what we need to test.
And also we need to focus on useful metrics, we cannot just optimise everything, let’s focus on matters most.
Random Sampling
Sampling is about selecting units from a large population of interest, to make generalisations of the whole population.
A small group of individuals or subjects taken from a larger population is called a sample.
Random sampling methods are directly used, so the sample is actually representing the population in terms of style,
quality and nature. If this is not done, then the sample will be biassed, and wrong conclusions will be drawn from the
results.
When applying a random sampling, every individual in the population has the same chance of being selected. The whole
method of sampling is carried out in a one-step procedure where every unit is selected uniquely from every other member
of the population. How can we actually randomise? We can randomise it in several ways. Two easy ways, just you can pick
a percentage, just stick to that 10%, 15%. Otherwise, you can actually try to select the user at runtime.
You can check if a user can be eligible for a testing phase. So, this if he is in the bucket of the testable user, then
they should be tested. This means that users will always be the same.
Central Tendency
We need to try to get the central tendency of our distribution of samplings.
We can use three values, the mean, the median and the mode. Usually, we can focus on the median.
Mean
The mean is generally known as average, that is the sum of all the numbers divided by the total number of occurrences, but it is different in statistical terms. There are different ways of averaging and the mean is just one of them. Every measure of the central tendencies is just a simple average. The basic formula is:
So we add up all the unordered data and divide it by their amount. The mean, unfortunately, is affected by outliers. So if you have very different values, then is not the right one.
Mode
The mode is the number with the highest frequency, it is usually seen as the most straightforward measure of central tendency to calculate. In order to get our mode out of raw data, we have to get the frequencies of all the numbers. After the sorting is done, we can now declare our mode which to be the number with the highest frequency. In our example of numbers, the mode is 9 because it occurs most (three times). A set of numbers or data having too many similar frequencies can be said to have no mode, but two or three numbers with the same frequencies are generally accepted.
Median
Median is also easy to calculate, the element corresponding to the average or central position of the data. In
determining the mode, start with arranging the numbers in numerical order (ascending or descending) so as make the
middle position correspond to the right element. Then the total number of data has to be known. This is, again, easier
when having frequency data because the sum of frequencies is actually the total number of data. The total number of
occurrences gives us the total number of data.
Another metric which is useful to focus on is the percentile.
Percentiles
Are perfect for automatic baselining. This is because the averages can be misleading. So, not all the user will load
your website at the same time. Even if your website loads in half a second, not all the user will, unfortunately. There
are many factors, and I’m gonna cover this later on. But I’m gonna tell you it cannot be the same time every time.
A percentile is a statistical measure that tells us the value below which a given percentage in a group of observations
would fall. A simple example will be the 25th percentile which is the value below which 25% of the population is found
in the observation.
So percentiles are actually useful in this case because they allow us to understand more about the real distribution of
the page loading times.
The percentiles to focus on are the 50th (i.e. the median), 95th, and 98th, these percentiles can be used for
comparison against the median to see the average and worst scenarios.
Conclusions
So this is because you cannot improve what you cannot measure, how can you do it? So what’s the reference to say, this
is faster, this is slower?
If it’s precisely the same amount of time, if you cannot measure it, it cannot be appropriately optimised.