02 Key Concepts Determination of Performance Indicators and Scenarios

02 Key Concepts Determination of Performance Indicators and Scenarios #

Hello, I’m Gao Lou.

In the previous lesson, we elevated performance from “testing” to an “engineering” level. Now, let’s organize a few important concepts at the engineering level, including:

  • Performance requirements metrics;
  • Performance scenarios;
  • Performance analysis decision tree;
  • Finding evidence of performance bottlenecks.

These concepts run through the entire field of performance engineering. With them, we won’t lose our direction in performance projects. Why do I say this? I will explain it one by one in the upcoming lessons.

To help you better digest this content, we will divide these concepts into three lessons to explain them in detail. In today’s lesson, let’s first look at “performance requirements metrics” and “performance scenarios”.

Performance Requirements #

When it comes to performance requirements, it’s been a bittersweet experience in my career spanning over a decade. I have rarely come across precise and explicit requirements, and in many cases, performance requirements end up being empty statements. If you haven’t had much experience with this, let’s take a look at two negative examples.

Negative Example 1:

Performance requirements like this are basically just the business side’s intuitive feelings, wanting to see the response time for a single user’s operation. So it cannot be considered a professional performance testing requirement.

However, you should note that there might be a pitfall lurking behind such requirements: the person listing this table may want the system to achieve these performance targets under any stress scenario. In that case, you should know that having clear performance requirements is a key point. We need to clarify under what business stress scenarios we expect such targets. In high-stress scenarios, the time requirements listed in this table are likely not achievable. Therefore, the performance requirements in this table are considered inadequate.

Negative Example 2:

Now let’s take a look at the following performance requirements.

These requirements seem very clear, but there is still one problem: they are not detailed enough. Why do I say that?

Let me give you an example. One of the target values mentioned here is “Front-end account service transactions: ≤5 seconds”. We know that account transactions have multiple functionalities, so if the requirement here is that each transaction should not exceed 5 seconds, it would be too broad. Other requirements also have similar issues.

Furthermore, there is a target in the table stating “Average CPU utilization not exceeding 80%”. This technical requirement seems specific, but let me ask you: what kind of CPU utilization is this referring to? If it’s the %us utilization of the CPU, does it mean that as long as the utilization is not higher than 80%, the system is good? Are there any other limiting conditions? Should we also look at the CPU queue?

After hearing all this, do you think any of the above-mentioned requirements can be used? In fact, there are many negative examples in actual work scenarios.

So what kind of performance requirements are reasonable? This requires us to start from four different performance scenarios and treat them differently.

Performance Requirements for Baseline Scenarios #

Let’s start with the business requirement targets. We can create a table like this for a single business performance indicator.

(This is obviously a list of indicators for a baseline testing scenario, and I have only included a few important parameters here. You can assemble other parameters yourself.)

Of course, if you prefer, you can change the parameter “standard deviation” to “response time percentile”, such as 90%, 95%, 99%. If we add these limits to the previous table, it will look like this:

(Note: The data here is just for illustrative purposes and is not meant to imply that these relations must be satisfied.)

Do you know why we set standards for standard deviation or response time percentiles? Because for performance, when the average value is quite good, there may be significant fluctuations. It’s important to note that the roles of “standard deviation” and “response time percentile” are different. The former is used to describe the range of fluctuation of a curve, while the latter is used to determine where the upper limit of the curve is.

Usually, I focus on the standard deviation because when there are some momentary spikes affecting the system’s performance, there may be some high values. Of course, this doesn’t mean that the system is not stable enough. We need to analyze the reasons to understand why there are spikes. Now let’s look at a few charts to understand the importance of standard deviation in influencing a curve. When the standard deviation is small (approximately 2.5), the schematic diagram is as follows:

When the standard deviation is large (approximately 17.6), the schematic diagram is as follows:

If there is a spike, the schematic diagram is as follows:

(Note: I don’t know if you noticed, but the schematic diagram above is actually flawed. It is obvious that the diagram is under a fixed pressure thread, and there is a high spike in the middle, which is problematic. If all transactions are correct, such a high spike should not occur. Therefore, these spikes are usually caused by transaction errors during the pressure process.)

The benchmark scenario is used to define the performance requirements of business metrics in this way. What about capacity scenarios? What parameters do we need to add to define them?

Performance Requirements for Capacity Scenarios #

For capacity scenarios, the most important thing is the business ratio, also known as the business model we often talk about. At the same time, other important performance parameters can also be redefined. The schematic diagram is as follows:

(Note: The data here is just for illustrative purposes and does not mean that it must satisfy such relationships.)

As you can see, in this capacity scenario, we determine the business ratio and the overall TPS (transactions per second) requirement, and we can calculate the TPS target for each business through the percentage. In addition, I also impose restrictions on response time.

From a performance perspective, these parameter restrictions are sufficient to define a scenario. From a technical perspective, such capacity requirements are also measurable. Of course, there can be multiple capacity scenarios, depending on the characteristics of the business.

Performance Requirements for Stability Scenarios #

I will explain the indicators for this scenario in detail in the later courses. Here I mainly emphasize two points:

  1. The duration of stability should be reasonable, meaning we need to determine how long the stability scenario needs to run.
  2. The TPS level used for stability should be reasonable, meaning we need to determine what level of pressure should be applied in the stability scenario.

These are the two important indicators for stability scenarios.

Performance Requirements for Exception Scenarios #

For this scenario, you only need to remember this process: Analyze the requirements of the system architecture for exception scenarios first, and then design corresponding test cases to cover them. Why do we need to analyze the system architecture? Because in an application, after we have completed functional testing, there are usually two types of exceptions: architectural-level exceptions and performance exceptions caused by capacity. And for architectural-level exceptions, we can only analyze them from the perspective of architecture.

After describing these performance requirements, can we move forward? From the perspective of performance technology, we can indeed move forward. But don’t forget, we haven’t described the user-level indicators yet. What should we do? This is where concurrency calculation comes in. Because without concurrency calculation, we cannot answer the relationship between TPS and users. I will provide detailed explanations on this in the practical section later.

Performance scenarios #

In general, after obtaining performance requirements, we need to match these performance requirements to scenarios and see which type they belong to.

Some people may say, “I think we have everything.” Actually, no! In my performance engineering concept, scenarios are not limited to just scripts and business models. In my previous columns, I have already described scenario design and execution, with two key points:

  • There are four types of scenarios (benchmark, capacity, stability, exception);
  • The execution process should maintain continuous progression.

In this course, I will expand the concept of scenarios from an engineering perspective.

You may wonder, how important are scenarios really? You can think of scenarios as the implementation of performance solutions, the core of performance implementation, and the starting point of performance analysis… I can write about 10 sentences to explain the importance of “scenarios”.

The following diagram represents the content included in performance scenarios.

I believe you can sense that in my performance engineering concept, scenarios are complex and vast. Because in my work, once the scenarios are running, all these contents in the diagram are what I need to observe. Why? Let me explain each of them separately, hoping that you can understand their importance.

  • Performance scripts: Performance scripts are only part of the scenarios used for applying pressure. They record what needs to be done in the scenario, whether it’s interface-level scripts, business-level scripts, or user-level scripts.

  • Parameterized data: I have repeatedly emphasized this point during training sessions, but there are still many people who think that running scenarios with a small number of parameters in a loop is reasonable. This idea is absolutely wrong! Because if it is severe, it will directly lead to unusable results.

  • Monitoring strategy: Please note that in the initial execution of scenarios, do not overly rely on monitoring tools. It is best to start with global monitoring tools. After encountering problems, we can repeat the scenario execution and use targeted monitoring tools.

  • Execution control: First, we need to follow the logic of “benchmark-capacity-stability-exception” during execution. Second, during the execution process, we need to view real-time data curves and determine whether to stop or continue, as well as which content to analyze, so that we are clear about what to do next.

  • Scenario adjustment: In this step, we need to clarify many things, such as where the pressure threads should start and what is the maximum; what the increment strategy should be configured as; whether a decrement strategy is needed; how long the duration should be, and so on.

  • Software and hardware environment: During scenario execution, we should have a concept in mind - under this scenario design, how the software and hardware should perform, how much CPU, IO, memory, and network should be used, whether the thread pool is reasonable, etc. These require experience-based judgment and comparison.

  • Base data / Seeding data: Different scenario objectives have different requirements for base data / seeding data. The base data / seeding data we require in performance scenarios should be consistent with production.

  • Service virtualization / Mock / Third party: In scenarios, it is crucial to control uncontrollable third parties, as their speed directly impacts the results. In this step, we need to determine based on the scenario objective. If we want to test the real production logic, then this step should be included. If we want to test whether our own system has performance problems, it can be blocked. However, in the results report, we need to clearly state this risk.

In my performance concept, the above are all essential concepts. Scenarios only truly take shape with these contents. In the following courses, I will describe in detail how to apply them to specific projects.

Having a perfect scenario does not mean that we will obtain perfect answers. Because there are more important things waiting for you, that is performance analysis. The logic of performance analysis is much more complex than scenario design, and generally has two main points:

  • Performance analysis decision tree;
  • Evidence chain of performance bottlenecks.

In the first half of my professional career, I mainly relied on these two approaches to support my work, and I have also used them to surpass many “ma baoguo” (a Chinese internet celebrity known for his erroneous claims). In the upcoming courses, I will explain the essence of these approaches in detail.

In addition, there are two key points: building a performance analysis decision tree and judging the evidence chain of performance bottlenecks. I will also clarify these points in the later courses.

Summary #

Alright, let’s summarize the content of this class.

The transition from “performance testing” to “performance engineering” is not something that can be explained in a single sentence, nor can it be represented by a diagram, an article, or a topic. The true engineering work lies in applying each of the steps mentioned above to specific tasks. This is also the purpose behind my clarification of performance concepts for you.

Let’s review the key points of this class together.

Performance Requirement Indicators: Without business indicators, there are no technical indicators, and our work is to align business indicators (such as concurrent user count and online user count) with technical indicators (such as CPU and IO). Different performance scenarios require well-defined performance requirement indicators, some of which are for our own reference and some for others to see.

Performance Scenarios: In fact, there is no need for more classifications when it comes to performance scenarios. The four categories we discussed (benchmark, capacity, stability, and abnormality) are sufficient to cover all performance execution processes.

Homework #

After completing this lesson, please carefully consider the following two questions:

  1. Why is it sufficient to classify performance scenarios into only four categories?
  2. To what extent have you refined the performance requirements indicators commonly used by you?

Feel free to write your thoughts and answers in the comments section and discuss with me. If you found this lesson useful, please share it with your friends. Their ideas may bring you even greater insights. See you in the next lesson!

About the Course Reader Group

Click the link on the course details page and scan the QR code to join our course reader group. We hope that the discussions and collisions of thoughts here will help you make greater progress. We look forward to your participation~