25 Let the Data Speak, How to Establish an Enterprise-Level Data Metric Platform #

Hello, I’m Shi Xuefeng. Today I’m going to talk to you about data measurement platforms.

First, let me digress. At the 2019 DevOps World | Jenkins World conference, CloudBees announced their new product: SDM - Software Delivery Management. In my opinion, this is a groundbreaking product.

In simple terms, SDM aims to solve the problem of consolidating fragmented software development processes onto one platform. By collecting data from the entire software development process and conducting intelligent analysis, SDM visualizes every aspect of the software delivery process for everyone to see.

Regardless of whether this product ultimately succeeds, the design concept behind it is certainly visionary, as it is the first solution to connect the business and development perspectives.

For business people, they can see real-time delivery progress of features; for developers, they can also see real-time business metrics and user feedback on the delivered features; for managers, they can have an overview of the entire process and identify any bottlenecks and efficiency issues during the delivery process.

Doesn’t this sound amazing? Don’t worry, I will bring you more information about this product in future special broadcasts, so stay tuned.

Now, back to the point. Every company I visited is putting tremendous effort into building data measurement platforms. Although these platforms may look different, their core mission is the same: visualizing the software development process.

Why is visualization so important for software development? It’s because visualization can greatly reduce the collaboration cost of software development, increase the transparency of the development process, and thus significantly reduce waste and rework in the development process.

For a simple example, the time cost of weekly meetings is generally high, but if the boss has a clear understanding of the project’s status, why bother putting so much effort into reporting work?

In Lesson 19 of this column, I introduced the related content of the DevOps measurement system to you. Do you remember the typical characteristics of good measurement metrics? These characteristics include clear audience, problem-focused, quantifiable trends, and tension.

In fact, these characteristics also apply when evaluating a measurement platform. Because the content presented on a data measurement platform is precisely the measurement metrics. This means that by summarizing the data and details of the measurement metrics, then displaying various views based on the dimensions of the metrics, the different needs of different users can be met.

As a result, the delivery situation of the entire team, including delivery efficiency and quality, can be objectively shown through data, rather than relying on individual subjective speculations. With objective data as a yardstick, the team’s improvement opportunities become clear at a glance.

Doesn’t it sound wonderful? However, in reality, building a measurement platform to meet this expectation is not a simple task.

I believe that there are a lot of challenges in the process of building and implementing a data measurement platform in the preparation, execution, and post-execution phases. Next, I will start from these three phases and discuss some ideas for building a measurement platform.

Preliminary: Consensus on Metrics #

There is no doubt that metrics are the foundation of a data measurement platform. Before building the platform, if there is no consensus within the team on the definition, data sources, calculation methods, and statistical scope of the metrics, the data presented by the data measurement platform will also be problematic.

Let me give you an example. The metric of demand cycle time is generally calculated by summing up the durations of demand cards at each stage of the demand, including analysis, design, development, testing, and release.

The testing cycle time, for example, is calculated from the time the demand card enters the “waiting for testing” stage to the time it completes the testing and enters the “waiting for release” stage, let’s say it’s 5 days. However, in a system that supports testing tasks, there is also a testing cycle time. This cycle time calculates the average execution time of each testing task, and the testing cycle calculated in this way may be only 1 day.

Setting aside which calculation method is right or wrong, what I want to express is that even for the same metric, the results obtained from different platforms and based on different calculation methods can be vastly different.

If the definition of the metrics is not aligned, then during the implementation of measurement, everyone will be unclear about which data is correct, which obviously does not facilitate the advancement of measurement work.

In addition, when defining metrics, it is generally necessary to hold a metric review meeting. However, at this point, since specific data cannot be presented, people tend to focus only on the definition of the metrics and usually cannot identify any problems. It is not until the data on the platform is available that it is discovered that some of the data seems to be incorrect. Therefore, it is necessary to reorganize the definition of the metrics, which often means rework in platform development and data recalculation. During the platform construction process, the time spent on data calibration and aligning metrics may be longer than the time spent on platform development itself.

“Data itself does not speak; it is people who give data meaning”, and that “meaning” is the metrics.

When defining metrics, everyone tends to choose interpretations that are favorable to themselves, which leads to misaligned perspectives in how data is viewed.

Therefore, before implementing the construction of a measurement platform, the most important thing is to detail the data sources and calculation methods of metrics, and they must be detailed to the extent that they can be realized and produce data results.

For example, the metric of development delivery cycle time generally refers to the duration from the start of actual development by the R&D team until the final online release. But this description is still not detailed enough, so our team describes this metric as: the time cycle from when the R&D team drags a task to the development stage on the demand management platform until this task enters the “released” state.

The task types here include features, defects, and improvement tasks, but do not include epic tasks and technical exploratory tasks. We will only collect statistics for tasks that have reached the delivery state, and unfinished tasks are not included in the scope of statistics. You see, only by describing at this granularity can the development team know how to operate, and data collection knows how to obtain effective data scope.

I suggest that before starting the construction of the data measurement platform, you should ensure that at least these metric data can be calculated offline, or even manually, and reach a consensus internally.

Avoid blindly starting construction right away! Often, even though we put in a lot of effort to build the platform and eventually succeed in building it, no one pays attention to it, and the core problem still lies with the metrics.

As an internal platform of an enterprise, a data platform’s accuracy is crucial. If there is any deviation in the data, it will not only lead to erroneous judgment and incorrect results but also cause significant harm to the operation and promotion of the platform itself.

In Progress: Platform Construction #

With the increasing complexity of software delivery activities, more and more tools and platforms are being used throughout the delivery process. Although the use of continuous delivery pipelines has helped integrate the delivery chain and drive the use of different tools and platforms, it’s fair to say that internal platform tools within an organization are still fragmented and not fully integrated.

This brings up a problem: each platform, to a greater or lesser extent, has its own data measurement capabilities and even fine-grained data display dimensions, but this data is stored in the respective tool platform’s own databases.

Let me give you an example. Jira is a commonly used requirement management platform in the industry and a mature commercial tool. Therefore, for such commercial systems, comprehensive APIs are provided. In addition, Jira has its own JQL query language, which allows for relatively simple querying and retrieval of metadata information. However, for a self-developed platform, the externally developed APIs may be relatively simple, and some systems may not even expose APIs externally. In such cases, if you want to obtain platform data, you either have to rely on developing new APIs or have to extract data by directly accessing the backend database through JDBC.

Furthermore, some platform data is obtained through message push, making it impossible to actively obtain the data. Instead, it can only be obtained through subscribing to broadcast messages from a message queue.

So you see, the ways in which data is obtained vary greatly for different metadata platforms.

Challenge 1: Integration of Numerous Data Source Platforms #

As a unified data measurement platform, the first challenge we face is how to extract useful data from these various types of platforms and ensure the isolation of data source access for flexible integration.

I suggest using a pipeline design approach, which is to be pluggable, but this time we are implementing data collectors as plugins. Take a look at this simple architecture diagram:

The collector is implemented for each connected data source platform and its role is to model data for each data source. This way, the collector shields the platform from the various ways of data acquisition, and the collected data is formatted uniformly for reporting and storage. An Operation layer can be designed on top of the collector to adjust the execution frequency of the collector and control the range of data collection.

If there is a large amount of data, you can also allow the collectors to connect to message queues similar to Kafka, which can be implemented as needed. This way, if a new platform wants to be connected, it only needs to implement a collector based on the data characteristics of that platform, and the overall architecture of the platform does not need to change.

Take a look at this example code for a collector:

@Override
public void collect(FeatureCollector collector) {
    logBanner(featureSettings.getJiraBaseUrl());
    int count = 0;

    try {
        long projectDataStart = System.currentTimeMillis();
        ProjectDataClientImpl projectData = new ProjectDataClientImpl(this.featureSettings,
            this.projectRepository, this.featureCollectorRepository, jiraClient);
        count = projectData.updateProjectInformation();
        log("Project Data", projectDataStart, count);
    } catch (Exception e) {
        // catch exception here so we don't blow up the collector completely
        LOGGER.error("Failed to collect jira information", e);
    }
}

Challenge 2: Storage and Analysis of Massive Data #

Generally speaking, common data storage methods are either RDMS (Relational Database Management System) or NoSQL (non-relational) databases. So, the choice depends on the data characteristics of the data measurement platform.

• The first typical characteristic is large data volume. For a large company, there are nearly tens of thousands of code submissions per day, which results in hundreds of thousands or even millions of individual data entries.
• The second characteristic is inconsistent data structure. This is understandable since the data fields related to requirements and those related to code have very little in common. Additionally, the number of fields may change depending on adjustments to metrics.
• The third characteristic is frequent data access. The data measurement platform needs to perform operations like random access and data read processing in large-scale datasets, which requires excellent scalability.

In addition, data measurement platforms generally store metadata and processed data. Metadata refers to raw data that has been collected but has not been processed, while processed data is cleaned and processed data.

Let me give you an example using a code repository. Metadata refers to individual code submission records by users, while processed data includes aggregated submission information by minute, such as quantity and line changes. These processed data can be easily provided to the front end for chart display. Storing processed data is done to avoid the need for real-time extensive data processing, thereby improving the performance of the data measurement platform.

Based on these typical characteristics and scenarios, it is evident that non-relational databases are better suited for storing a large number of metadata entries.

I recommend using HBase, which is a database suitable for storing unstructured data and has built-in support for distributed storage systems. For storing processed data, you can use the relational database MySQL.

Of course, there are other options for database selection, and the industry has many open-source and commercial tools. For example, the open-source data measurement platform Hygieia uses MongoDB, while the commercial tool Insight is widely used by many large companies for massive data usage.

Let me also share with you an architecture diagram for data measurement. From this diagram, you can see that the underlying data is stored based on HBase and HDFS.

Challenge 3: Customized Display of Measurement Views #

A data measurement platform needs to meet the requirements of various dimensions, so it generally provides multiple dashboards, such as management dashboard, technical manager dashboard, and individual dashboard. However, these preset dashboards are difficult to meet the individual needs of each person, just like the saying “There are a thousand Hamlets in a thousand people’s eyes.” Therefore, the views on the measurement platform should also be customizable.

If you want to achieve custom views for measurement, such as supporting dragging and editing of chart positions, adding new components, and sending reports based on customized views, you need to put some effort into front-end development. Fortunately, modern front-end frameworks have ready-made solutions, and you only need to import the corresponding components.

I recommend two front-end components for you to reference:

• Component 1
• Component 2

Both of these components support common operations such as dragging, scaling, automatic alignment, adding, and deleting widgets. In this way, everyone can freely customize their workspace views according to their needs. Different roles can also customize and send reports without the need to extract data from the measurement platform and manually organize it into presentation slides.

Taking vue-grid-layout as an example, you can place echarts charts inside custom components and also implement your own methods. You can refer to this article for specific methods.

Now that you have gained a general understanding of the architecture of a measurement platform after learning about the three common challenges in the initial construction phase, let’s move on to the third phase.

Aftermath: Implementation of Rules #

In terms of current development efficiency, building a data measurement platform is not a difficult task. In fact, constructing a measurement platform can only be considered as the first step in data measurement, with the remaining 99 steps relying on the operation and promotion of the platform.

This statement is not an exaggeration; it might even be said that if no one cares about the data on the measurement platform, the significance of the first step itself would be questioned.

During the initial operation, the biggest challenge faced by the measurement platform is data accuracy, which is also the easiest aspect to be challenged by others.

There are many reasons for inaccurate data, such as issues with the calculation of measurement indicators, problems caused by abnormal data injection, and problems with departmental categorization and aggregation. However, in reality, the most frequent cause of issues is the lack of adherence to proper development practices.

For example, consider the data that heavily relies on the standardized flow of demand cards, such as demand delivery cycle. If the demand card is dragged directly to the “online” state after development, the calculated demand delivery cycle might only be a few seconds, which is obviously problematic.

The correct approach is to follow the actual flow of the demand card, such as associating the development test with the demand so that the demand card is automatically transitioned to the “waiting for testing” state. After the test acceptance is passed, the card should again be automatically transitioned to the “testing completed” state, and so on. Efforts should be made to achieve automated operations, rather than relying on personal awareness.

Another example is the code lines associated with the demands. If there is no established association between the code and the demands when development is submitted, the resulting data will also be significantly distorted. These non-standardized data will not change due to subsequent operations, and it is difficult to repair and clean up the data. They will persist in the data pool of the measurement system, leaving an indelible mark.

Therefore, the measurement platform can only play a truly effective role through top-down project driving. Rules should be established for non-standardized operations, data from malicious actions should be audited, problems discovered by the measurement should be included in continuous improvement efforts, and the trends and locations of each indicator should be tracked.

In addition, in order to allow the data to directly point out problems, it is also necessary to indicate whether the current data in the measurement platform is good or bad.

There are various ways and methods to achieve this, such as setting reference values (e.g., setting a minimum reference value of 50% for unit test coverage), so that the discrepancy between the current data and the reference value can be reflected in measurement charts. Alternatively, you can also indicate the current position of each comparative indicator within the department, whether it is in the top 10% or the bottom 10%. Such data helps promote improvement efforts.

In the end, the purpose of measurement is continuous improvement. If data from 100 indicators is collected and displayed in the measurement platform, but it cannot be determined which indicator has brought improvement to the team and how the improvement has been achieved, then what is the value of such a measurement platform?

Summary #

In this lecture, I introduced the core value of building a data measurement platform, which is to visualize the software delivery process. On this point, the ideas of major companies in the industry are consistent. Therefore, a data measurement platform is an indispensable part of the current enterprise DevOps platform construction.

When building the platform, you need to focus on three stages: pre-event, mid-event, and post-event.

• In the pre-event stage, consensus needs to be reached on the definition of metrics. The metrics here need to be detailed to the level of data sources and calculation formulas. Even without a measurement platform, the corresponding results can be calculated.
• In the mid-event stage, in terms of platform construction, for multi-data source platforms, flexible adaptation can be achieved through the use of collector plugins. It is recommended to use non-relational databases such as HBase for data storage, and existing front-end components can be used to implement visual interface display.
• In the post-event stage, it is about the operation of data and the implementation of rules. Only when measurement data can reflect problems and drive team improvement, can measurement make sense.

Reflection Questions #

When building and applying a measurement platform in a company, what problems have you encountered? How did you solve them?

Feel free to share your thoughts and answers in the comments section. Let’s discuss and learn together. If you find this article helpful, you are also welcome to share it with your friends.