User Story - The Man in the Yellow Cloud Travels a Hundred Miles Half Ninety #

Hello, I’m Huang Yun, currently a backend developer based in Suzhou. I have been working in this field for over three years. When the “Kafka Core Technology and Practice” column went online, I subscribed to it immediately. Over the past few months of studying this column, I have gained a lot of insights and experiences that I would like to share with you.

Why should I learn the “Kafka Core Technology and Practical” column? #

As we all know, in today’s era, “big data” has become a well-known concept. Large amounts of data ranging from GB level, TB level, to even PB level require servers to process them in a short period of time. How to make full use of the performance of each server, smooth out the peaks and fill in the valleys, has become the key to data processing. Kafka, as a mainstream message queue tool, has been used by many large companies. Mastering Kafka has become an essential specialized skill in the era of big data.

In our company, we are currently using Kafka to interface real-time traffic data, with a data volume of about 20GB per day. As a developer engineer, I must quickly master the core technology of Kafka and cultivate specialized skills in order to make better use of it. All along, in my spare time, I have been actively collecting Kafka-related materials and reading articles shared by experts. However, those scattered fragments have always been difficult to form a systematic knowledge system, which has been a headache for me.

The Balance between “Dao” and “Art” #

As a programmer, like many others, I always want to get my hands dirty when I come across something great. After just discovering the powerful features of Kafka, I couldn’t wait to follow the Kafka official tutorial and set up a Kafka server. In addition, I even tried my hand at using Kafka as a message queue in a real-time integration environment with a throughput of 10,000 messages per second.

However, during the actual usage, I discovered a serious problem: I knew how to use Kafka, but not how to use it well. Recognizing this problem, I quickly looked through the official website and articles from many mainstream websites. The official website listed hundreds of configuration parameters with explanations, while other materials generally focused more on theory than practice. Overall, these resources prioritized “Dao” (or principles) but completely neglected the implementation of “Art” (or skills).

To be honest, as programmers, we feel frustrated when reading these articles. We find ourselves staring at pages full of principles but have no idea where to start, which means we are destined to be unable to solve our actual problems.

Until one day, I accidentally discovered Professor Hu Xi’s column “Core Technologies and Practices of Kafka” on the platform GeekTime. I instantly felt like I had found a priceless treasure. After reading the course introduction and preface, I realized that this was the course I had been eagerly anticipating, so I subscribed immediately. From then on, it became a habit for me to study the updated articles every Tuesday, Thursday, and Saturday.

During the learning process, the most impressive thing to me was that Professor Hu Xi used the word “most” three times in the titles of the 7th and 8th lectures: “The Most Important Cluster Configuration,” and in the articles, he shared the most important and core cluster configuration. After reading these two lectures, the complaint I had always held about “why there are so many parameters in Kafka configuration optimization” was finally resolved. There is no need to pursue a comprehensive understanding of every single parameter; just mastering the core content is enough. Professor Hu Xi’s column truly achieves a balance between “Dao” and “Art,” gradually unveiling the mysterious veil of Kafka for me.

How do I study the column? #

To be honest, reading the columns on Geek Time is a very enjoyable thing. Every morning, before I start working, I have a fixed 20-minute time slot for studying the columns. When studying the Kafka column, I need to focus my attention because there is a lot of valuable information in it. If I just read it superficially without thinking, I will have many questions left unanswered.

When I encounter confusion, the first thing I do is check the comments below the article. Many people have strong summarization skills and are able to reprocess and refine the information. These comments help me to reorganize the content of the article. Most importantly, I often see the same confusion in the comments as I have, and instantly feel a sense of finding like-minded people. It is a wonderful experience.

Hu Xi’s replies are also timely, and she solves our doubts as soon as possible. “Today’s doubt, today’s answer, don’t accumulate unresolved questions” also allows me to study the subsequent content more efficiently.

While studying the columns, I have taken some detours and constantly adjusted my study methods and habits. In this process, I have learned some experiences and have many insights that I would like to share with you.

1. Think more, summarize more. Every time I look at the comments, I first try to think about the answer to the question myself. After the teacher’s reply, I confirm the answer against it. Often, when I see the summary of others in the comments, I feel that it is concise and to the point. But when I try to describe it myself, I find that I completely lack the ability to abstract and summarize. In other words, I cannot describe the meaning expressed in the article in my own words, which means that I have not truly understood the content of the article. So, I keep reminding myself to summarize more and improve my ability to generalize.

2. Practice more, practice brings true knowledge. There is a lot of practical knowledge in the column. Every time I find content with strong operability, I will practice it myself in the test environment I have set up, and check the results. This way, I have a better understanding and mastery of the course content.

3. Review more, don’t leave doubts. I have found that many key contents in the columns are not deeply impressed if I only read them once. But if I review them after a period of time, I will have new insights and gains. Some chapters are full of valuable information, and once I encounter unfamiliar terms, my brain instinctively skips them. After realizing this habitual action of my brain, when I encounter unfamiliar terms, I will take the initiative to spend time researching related materials until I understand the meaning of the terms. Otherwise, if I continue to read without fully grasping the previous concepts, it will greatly affect my understanding of the subsequent content.

4. Take more notes. For some course content that I am deeply interested in, I will make targeted records. When making notes, I generally do not directly copy and paste the original text, but write it in my own words. If I can’t express it in my own words, I will transcribe the original text, which will also deepen my understanding.

5. Ask more questions. For questions that I really can’t understand, I ask myself to ask more questions and leave more comments. Every time the teacher answers my question, I feel very happy, and my enthusiasm for learning will be higher afterwards. This is probably what the teacher often refers to as improving participation and sense of belonging.

Regarding leaving comments, I would like to share with you a change in my mindset. When I first started learning, I was actually afraid to ask questions when encountering things I didn’t understand. I was worried that the reason I didn’t understand was due to my own lack of understanding, and I was afraid that the questions I asked were too basic, so I kept them to myself. But later I realized that the more I did this, the less I understood, and I gradually lost a lot of enthusiasm for learning. After realizing this, I promptly adjusted my learning strategy and reminded myself: for questions that I can’t understand even after thinking, I must ask more. To not hide your weaknesses is the best path to growth.

What are the gains of studying this column? #

The biggest gain I have achieved from studying this column is a deeper understanding of the principles, mechanisms, and parameters of Kafka. I greatly admire Hu Xi’s ability to abstract and summarize information. Hu Xi spent over a year reading the Kafka source code and related literature, but what he presents to us is not lengthy discourse, but rather content that has been abstracted and summarized multiple times. Starting from the aspects that we users are most concerned about - introduction, configuration, clients, internals, and monitoring - Hu Xi introduces Kafka’s content and condenses the hundreds of parameters in the official documentation into a few important ones, abstracting hundreds of pages of content into the core knowledge covered in dozens of lessons.

This has inspired me greatly. In the learning process, it is not good to pursue breadth over depth. We only need to master the dozen or so parameters that are most commonly used in actual production environments to ensure stable, fast, and lossless message transmission. For many profound topics, if energy and time allow, we can delve deeper, but we should always begin with the end in mind and not get stuck in the details.

Perhaps only programmers understand the pain of reading large chunks of source code, but Hu Xi has done it and even shared his methods for reading source code in the “extra meals” section. What is delightful is that Hu Xi directly displays the source code in an IDE and describes the code while referring to the actual code, making it highly practical. In fact, this aligns perfectly with my initial expectations for studying the Kafka column. Most of the content in the column can be put into practice. Don’t build a high platform on loose sand; Hu Xi steadily guides us forward step by step.

Summary #

The column has ended without me even realizing it. It’s quite embarrassing that in the second half, I couldn’t keep up with the teacher’s pace and often accumulated a lot before starting to read. However, I am really grateful for Hu Xi’s efforts. The content shared by the teacher in the column helped me form my own understanding of Kafka’s parameters and some practical knowledge.

There is a sentence that Hu Xi mentioned in the column that left a deep impression on me, “The last 10% take as much effort as the first 90%”. The journey of learning Kafka will not stop with the end of the column. It is only halfway through the journey after finishing the content of the column. In the future, I will record and practice the implementation details mentioned in the column one by one, so that I can truly understand Kafka, know how to use it, and use it well.

Alright, my sharing ends here. Did you have any unique learning methods and experiences while studying the column? Feel free to write in the comments section. I am looking forward to exchanging and discussing with you here, and learning together.