00 Preface How to Efficiently Enter PyTorch #

Hello, I’m Fang Yuan, and welcome to join me in learning PyTorch.

Let’s start with a self-introduction! I have previously worked for both Baidu and Tencent, serving as a senior algorithm researcher. Currently, I am a data scientist at Line China, an internationally renowned internet company, where I work on the development of computer vision and natural language processing related projects, providing deep learning services for millions of users daily.

Think about it, I have been involved in the research and application of machine learning and deep learning for 10 years now, and it has been a very interesting journey. With the rapid development of artificial intelligence, various deep learning frameworks have emerged, including the mainstream ones and the newcomers.

Why do I say this? It actually goes back to my graduate school days. Initially, I simply labeled PyTorch as a newcomer. I remember that at that time, the wave of deep learning was just beginning, and traditional machine learning was transitioning to deep learning. However, our choices of frameworks were very limited.

At that time, Theano was a popular deep learning framework in the academic community, which some of you may not have heard of. It was like a founder’s existence. After its birth in 2008, for a long time, it was the industry standard for deep learning development and research.

To reproduce the algorithms in papers, I started learning Theano. After getting in touch with it, I found that its declarative programming, both in terms of style and logic, was very unique. However, at that time, learning resources were scarce, and I could only rely on the official documentation. As a result, I found Theano very obscure and difficult to learn, with a high barrier to entry.

Later, I joined the core departments of internet giants. At that time, the academic community had already produced a lot of research in the field of deep learning, while the industry was just beginning to apply deep learning technology. Google’s TensorFlow framework was officially open-sourced in 2015, and our team started applying deep learning technology to text processing and other areas.

In 2017, Google released TensorFlow 1.0, and in 2019, it released the new version 2.0. During the time of TensorFlow 1.x, the framework had a large user base. However, there were obvious problems, mainly that the framework environment configuration was not compatible, there were significant differences between functions in old and new versions, and coding was difficult.

Whenever there is a version update, there will always be API changes and incompatibilities between versions. And when I read other people’s code, TensorFlow 1.x was not very readable. All of these issues increased my learning cost. It wasn’t until TensorFlow 2.x that TensorFlow gradually learned from the advantages of PyTorch and improved itself.

During the same period as TensorFlow, there was another deep learning framework with numerous users, which was Keras. Keras’s API is very user-friendly and easy to get started with. If you have any ideas that need quick experimentation and a look at the results, Keras is definitely the best choice.

However, highly modular encapsulation also brings disadvantages. It seems that learning Keras is very easy, but I soon encountered bottlenecks. Highly encapsulated means less flexibility, for example, if you need to modify the underlying structure of the network, the interfaces provided by Keras do not support it. In most of the time using Keras, we mainly stayed at the stage of calling interfaces, making it difficult to truly learn the content of deep learning.

It wasn’t until PyTorch appeared that, with more and more people using it, its technical iteration speed and ecosystem development speed became very rapid. If you find an open-source PyTorch project on GitHub, it can be easily migrated into your own project, allowing you to stand on the shoulders of giants and see the world directly.

Moreover, compared to the mainstream frameworks mentioned earlier, PyTorch has a user-friendly imperative programming style. PyTorch is designed in a more scientific manner, without the need to switch between various APIs, making operations more convenient.

PyTorch’s environment configuration is also very convenient. Various development versions are backward compatible, so there is no problem of old version code not being usable on new versions. Furthermore, PyTorch has a similar style to NumPy and can easily integrate with the Python ecosystem. We only need to master NumPy and basic deep learning concepts to get started, and it is also fast and flexible in terms of building networks.

In addition, PyTorch is very convenient for debugging code, as it can output intermediate tensor results at any time. Using PyTorch is as simple as using print in Python. Just put a pdb breakpoint into the PyTorch model, and you can use it directly.

Because of its elegance, flexibility, and efficiency, PyTorch has attracted more and more people to learn it. If there are still people who only see PyTorch as a newcomer, thinking that PyTorch is just a “challenger” trying to carve out its own territory in the world dominated by TensorFlow, then data can prove that this idea has become outdated. In fact, both in the academic and industrial fields, PyTorch has occupied a significant position.

From the perspective of the academic community, before 2019, TensorFlow was still the mainstream framework chosen by major conferences. However, after 2019, PyTorch has almost taken over the top conferences, with a rapid rise, achieving this in just one year.

It is worth noting that the field of machine learning has always relied on research-driven approaches, and the industry naturally cannot ignore the achievements of scientific research. Take my team as an example, we have gradually migrated to the PyTorch framework, and newly launched projects are implemented using PyTorch as the first choice. It must be said that the application scope of PyTorch has gradually expanded, which has also promoted the development of its ecosystem. As more and more developers are using PyTorch, it is easy to find solutions to errors or bugs on development forums.

In short, once you master PyTorch, you are on the fast track of deep learning and machine learning. You can quickly get started with other deep learning frameworks and have a comprehensive understanding.

If you are about to enter or have already entered the field of deep learning and machine learning, PyTorch can help you quickly validate models and algorithms, quickly deploy deep learning models, provide high-concurrency services, and easily implement interesting experiments such as image generation, text analysis, and sentiment analysis. In addition, many algorithm-related positions also require proficiency in tools like PyTorch.

There are many directions to explore, and I won’t list them all here. So the question is, since PyTorch has so many advantages, how can we get started quickly?

Only understanding the principles is like a castle in the air, and direct practice is relatively difficult for beginners. Therefore, the method I recommend is to first have a general understanding of the framework’s basic syntax and then quickly integrate into an actual project to see how we can solve a problem based on the framework in real tasks. This column is designed along this line of thinking. I provide you with rich code and practical examples in this column to help you quickly get started with PyTorch.

Through this column, you will become proficient in using PyTorch tools to solve your own problems, which is the most basic goal of this column.

In addition to mastering the usage of the tools, my ultimate goal is to give you the ability to analyze and solve problems, and to teach you how to optimize your own algorithms and models. In terms of learning experience, I hope this column will open a window for you to understand the knowledge required to pursue the path of deep learning.

In order to take you from beginner to expert, I divided this column into three progressive parts.

Foundations

A brief introduction to the development trend of PyTorch and the installation method of the framework, as well as the commonly used operations of NumPy, which is an appetizer for PyTorch. We assume the use of PyTorch 1.9.0 and that you have already mastered Python programming and have a basic understanding of machine learning, but don’t worry, I will explain new knowledge from scratch.

Model Training

To quickly master a framework, you need to start with the core modules. In this section, I will explain the automatic differentiation mechanism, building networks, updating model parameters, saving and loading models, visualizing the training process, and distributed training, all related to the series of processes in deep learning model training. This will show you what PyTorch can help us with. After studying this section, you will be able to build network models based on PyTorch.

Hands-on

Throughout this column, we focus on the application of the PyTorch framework in specific project practices. So in the end, I will combine the popular image and natural language processing tasks, linking the content of the previous two modules, to explain in-depth how PyTorch solves real-world problems.

Overall, besides providing you with a systematic framework for learning PyTorch, I also hope to share my experience and thoughts on the path of deep learning.

Finally, I have a piece of advice. When learning PyTorch, it’s a good method to learn and lookup at the same time. Because when we actually work on projects, we will definitely come across functions we have never used before. Looking them up ourselves can strengthen our memory.

Of course, I will do my best to be a guide and help you achieve the course objectives step by step. I also hope that you can actively engage in this learning journey. Now, let’s explore PyTorch together and open the door to deep learning!