Closing Remarks - Be a Determined Learner Before Time #

Hello, I am Wu Lei.

Time flies like a white steed, and unconsciously, it is time to bid you farewell. When the editor said to me, “Teacher, we need to finish the concluding remarks this week”, I suddenly paused, feeling a bit dazed and unreal. I didn’t respond as usual, “Alright, no problem!” because I seemed to have gotten used to rushing through articles, to answering questions in the comments section, to interacting with all of you, and even to cracking jokes in the group…

However, no matter how accustomed I am, our column officially comes to an end today. Here, I want to take this final opportunity to sincerely thank you, the person in front of the screen, and thank you for listening to the audio. Thank you for your continuous companionship and encouragement, thank you for participating in the discussions in the comments section, thank you for promptly correcting any mistakes in the column, and more importantly, thank you for your tolerance and understanding of the column.

At the same time, I also hope that in this column, we can not only discuss methods for quickly and efficiently mastering Spark performance, but also transfer the same learning techniques to other fields. After all, efficient learning efficiency can always allow us to be one step ahead.

How Should We Learn? #

When it comes to learning, we often hear things like “learning is against human nature” or “the process of learning is growth, and growth is inherently painful.” It seems that whenever we mention learning, we should be prepared to endure difficulties. But is that really the case?

Honestly, I don’t quite agree with the “painful learning” theory. The reason is simple: we have many methods and techniques that can make learning interesting and efficient. For me, the three most effective methods for improving learning efficiency are a “producer mindset,” the “235” learning principle, and the “life-like association” learning technique.

Producer Mindset #

Let’s start with the producer mindset. If we consider various information, knowledge, principles, and theories as commodities, then it’s undeniable that those of us who love learning are the consumers of these “commodities.” We spend a great deal of time, energy, and attention trying to extract value from these commodities.

However, consuming commodities is a one-way and passive process. It’s like passively listening to or watching pre-recorded programs on the radio or television. No matter how well-designed and well-produced the programs are, as consumers, our ability to digest and absorb them is limited.

At this point, we can adopt a learning approach that is output-oriented, which is often more efficient than a passive learning approach. Let’s take technical blogging as an example. The process of producing content helps you discover flaws and deficiencies. When you find yourself stuck or unable to continue writing, it often indicates that your understanding of a particular knowledge point or principle is not solid enough, and you need to go back and relearn it. Therefore, purposeful and targeted learning often yields better results. To prioritize output in order to acquire input is what we often refer to as a “producer mindset.”

However, you might say, “Even if the producer mindset can indeed help me improve learning efficiency, I really don’t know where to start with the production aspect.” If you haven’t picked up a pen in a long time, you can start with taking notes. Note-taking has at least two benefits: first, it reinforces memory, as the saying goes, “the palest ink is better than the best memory”; second, the notes themselves are a form of product, or more precisely, the primary form of a product. The concept of entropy reduction may sound mysterious, so let’s give an example. Imagine there are two rooms, each with 5 boys and 5 girls, resulting in a 1:1 gender ratio in both rooms. If we calculate the entropy based on gender, the entropy of both rooms would be 1.

Then, we move all the girls from the first room to the second room and move all the boys from the second room to the first room. After this exchange, the entropy of each room is reduced to 0. This is an example of entropy reduction, as it eliminates the uncertainty of the “gender” attribute within the rooms.

The process of learning is similar. For a new technology, there are many unknown variables such as what it is, who created it, what problem it solves, its components, how it works, and its applicable scenarios. Each of these questions represents an unknown variable, and all of them combined contribute to the entropy of the technology, which tends to approach 1. The process of learning works by gradually converting these unknown variables into known variables. During this process, the entropy of the new technology decreases step by step, approaching zero.

Different methods contribute to entropy reduction in different ways during the learning process. Communication and discussion contribute exponentially to entropy reduction, making it the most efficient approach. That’s why I always encourage you to participate in post-class discussions. Through participation, you can internalize new problems and solutions that you haven’t encountered in your work, expanding your cognitive boundaries and deepening your understanding of the technology itself.

On the other hand, theoretical learning and hands-on practice contribute to entropy reduction linearly and gradually. You may argue, “Hands-on practice should have the highest contribution!” While practice is indeed important, think about your past work experiences. How many of the hundreds or thousands of code paths did you actually explore?

No matter how dedicated or diligent we are, our cognitive boundaries are always limited. To expand these boundaries, continuous sharing, communication, and discussion are necessary. This is what “learning from others’ experience” means.

Relating to Daily Life #

Relating technology principles and knowledge points to everyday life is called “relating to daily life.” It involves using analogies, personification, metaphors, and other techniques to connect dry principles and knowledge to everyday life. By using this learning technique, we can quickly absorb new knowledge. More importantly, this technique extends our memory curve, enabling us to remember things for a longer period.

Frederick Brooks, the author of “The Mythical Man-Month,” once said, “The design of any software architecture cannot escape human social organization.” Similarly, for the constantly emerging technologies and the vast array of knowledge points, we can find their reflections in everyday life. Therefore, in our daily learning, we can boldly open our minds and try to connect what we learn to interesting scenes we encounter in life. Once you start doing this, you will find that compared to your peers, it seems like a door has been opened for you to a colorful and interesting new world.

This is why I have spared no effort in trying to explain Spark’s core concepts, key knowledge points, and its working principles using analogies and stories. My intention is simple – to make it easy and quick for you to digest and absorb, while also making that dry theoretical knowledge stick in your memory more securely and for a longer time.

Establishing connections between technology and life naturally requires energy and effort to activate more neurons in the brain. However, don’t think of this process as painful; instead, it should bring you a sense of anticipation and excitement. Because once the connection is established, both sides of the connection – whether it be technology or life – become part of you.

With a producer mindset, following the “2-3-5” principle, and combining the learning technique of “relating to daily life,” I believe your learning efficiency will continue to increase.

Not Leaving Until We Meet #

Finally, I want to sincerely say to you, “Thank you!” Thank you for accompanying me on this journey of Spark tuning. For me, your company is the greatest recognition.

You know, I am neither a Spark Committer nor a Spark Contributor. I am just a veteran developer with unlimited passion and years of practical experience in Spark. All I can do is to extract and present every bit of my knowledge, experience, accumulated thoughts, and techniques from my mind, just like juicing, and present them to you in the form of this column.

My knowledge and perspective are extremely limited. The content of this column may have omissions. I really feel that, in terms of this column, I am just an initiator, like the creator of a GitHub project. The correction, enrichment, polishing, and improvement of the column actually rely on every developer like you who participates in the subscription.

Therefore, even though our column is coming to an end, our learning will not stop here. In the next period of time, I will create a project on GitHub called “potatoes,” which is the project for “potatoes.” I will gather the code, data, results, and common questions related to the column in this project.

Moreover, based on your needs, I will continue to add content such as written test and interview questions, job opportunities, and career development to it, turning the “potatoes” project into our shared Spark academy. Through this academy, we can not only delve deeper into learning Spark but also establish more connections, expand our network, and explore more opportunities and possibilities. The project is available at: https://github.com/wulei-bj-cn/potatoes.git. I’ll be waiting for you there, and we won’t leave until we meet!

As the saying goes, “The way ahead is long, but we keep striving.” Let’s continue learning and encourage each other.

Lastly, I have prepared a graduation survey for you. It has only a few questions, and I hope you can take two minutes to fill it out. After going through this time together, I look forward to hearing your feedback and suggestions for me and this course!