Make Rust Your Next Main Programming Language #

Hi, I’m Chen Tian. I’m currently the VP of Engineering at TubiTV, the largest free streaming service in North America, and the author of the WeChat blog Programmer’s Life and Zhihu column Mysteries.

Over the past 18 years, I have been engaged in the R&D of high-performance systems, covering areas such as network protocols, network security, server architecture, blockchain, and cloud services.

Because I like to use the right tools to solve the right problems, at different stages of my career, I have extensively used development languages with very different forms and mechanisms.

I have built various network protocols with C and assembly language and maintained ScreenOS, an embedded operating system that is very well known in the field of network security. I have written my previous startup project TukeQuan with Python/JavaScript. I have built TubiTV’s high-concurrency backend core with Node.js/Elixir. I have built the blockchain framework Forge with Elixir and have also studied Haskell/F#, Clojure/Racket, Swift, Golang, C#, and other languages.

Starting in 2018, I began to pay attention to Rust. At that time I was developing Forge and deeply felt Elixir’s inability to handle computationally intensive functions. Introduced by Handong, who is also the author of “The Rust Programming Language”, I started learning and using Rust.

Also because I had deeply used so many development languages before, as soon as I came into contact with Rust, I realized it was definitely a tool for the future.

First of all, you will feel that Rust is a language that attaches great importance to the developer experience when you start using it. If you rank the programming language user experience in the internet age, Rust is definitely in a class of its own.

You can’t imagine a language whose compiler not only informs you of errors in your code, but also tries its best to recommend correct code to you. It’s like having an all-knowing, kind expert sitting next to you tirelessly reviewing your code and helping you identify problems as you develop.

For example, in the following code, I started a new thread that references a variable from the current thread (code):

let name = "Tyr".to_string();
std::thread::spawn(|| {
  println!("hello {}", name); 
});

This code is extremely simple, but it implicitly contains unsafe thread access to a variable. The name variable held by the current thread may be released before it is used by the newly launched thread, resulting in a use after free error.

Not only can the Rust compiler detect this error at compile time through type safety, it also tells you the cause of this error: “may outlive borrowed value” (we won’t go into what it means for now), and it goes further to recommend adding “move” to resolve this error. To make it easier for you to further understand the error details, it thoughtfully provides a command line “rustc –explain E0373” to allow you to get more information from the knowledge base:

Once you get used to Rust, it’s hard to leave this level of experience behind. Rust’s ultimate user experience is reflected not only in the compiler, but in the whole language toolchain including rustup, cargo, etc., which are all so simple, easy to use, and humanized.

Secondly, the well-known excellent performance and powerful expressiveness allow Rust to flex its muscles in many scenarios.

As of 2021, mainstream internet companies have included Rust as one of their main languages, such as developing operating systems Redox/Fuchsia, high-performance networking Tokio, high-concurrency backend TiKV for applications, and even client software itself (Feishu). We are delighted to see that in addition to the traditional areas of system development such as operating systems, device drivers, and embedded, Rust is also blossoming in server-side high-performance and high-concurrency scenarios.

In recent years, we have heard news of many well-known internet companies using Rust to refactor their tech stacks almost every period of time. For example, Dropbox used Rust to rewrite its file sync engine, and Discord used Rust to rewrite its state service. In fact, these companies are driven by business needs to naturally use Rust.

For example, Discord’s original Golang state service would consume a lot of memory and cause huge latency spikes during peak hours due to garbage collection. After much painful reflection, they chose to rewrite it in Rust. According to Discord’s official statement, in addition to performance improvements, Rust also made code refactoring as the product iterated feel effortless.

Along with performance, Rust has many advantages for an engineering team. For example, its type safety and borrow checker make it very easy to refactor code as product requirements change or new learnings about the language are discovered. Also, the ecosystem and tooling are excellent and have a significant amount of momentum behind them.

Finally, it’s my own feeling from using it that Rust gets more enjoyable the more you use it. From my personal development experience, with many languages, the more deeply you use them or the more widely you use them, the more you feel reluctant to compete, either because you are constrained too much and unable to exert yourself, or because there are too many cumbersome details and it is difficult to have both expressiveness and conciseness.

But I rarely have this feeling when using Rust. Simple bit operations, handling large amounts of parquet, facing CPU out-of-order instructions directly with atomics, and even high-level abstractions like Go’s channels, Rust and its ecosystem have everything to allow you to use the most appropriate tool to solve the most appropriate problem when you want to do something.

Difficulties in Learning Rust #

After experiencing the power and beauty of Rust in 2019, I held a series of lectures to introduce Rust to my team at the time, in order to handle computationally intensive tasks that Elixir had difficulty with. But in this process, I also deeply felt the difficulty of imparting Rust’s core ideas to experienced developers.

Rust is recognized as a very difficult language to learn, with a very steep learning curve.

As a language with its own unique ideas, Rust draws on the strengths of others, learning and enhancing move semantics and RAII from C++, borrowing and developing lifetimes from Cyclone, functional programming and type system from Haskell, etc.

So if you want to migrate to Rust from other languages, you must go through a period of paradigm shift.

The shift from imperative programming to functional programming, from mutability to immutability, from weakly typed languages to strongly typed languages, and from manual or automatic memory management to memory management through lifetimes, the difficulty is multiplied.

The biggest paradigm shift in Rust is the ownership and lifetime of variables, which is an area almost no other programming language has covered.

But once you get past this hurdle, the other knowledge points are just concrete uses of the ownership and lifetime concepts in different areas, such as how ownership and lifetime combine with the type system to ensure concurrency safety, and how lifetime annotations participate in generic programming, and so on.

In the learning process, in addition to ownership and lifetime, engineers with different language backgrounds will also have different difficulties. You can focus on learning:

• C developers: Difficulties in the type system and generic programming
• C++ developers: Main difficulties in the type system
• Python/Ruby/JavaScript developers: Difficulties in concurrency, type system and generic programming
• Java developers: Difficulties in understanding asynchronous processing and concurrency safety
• Swift developers: Almost no additional difficulties, just thoroughly understand Rust’s asynchronous handling

Once you get through this difficult period of paradigm shift, you will understand that Rust is indeed a language that radiates charm from the inside out.

From the perspective of the language kernel, it reshapes our understanding of some basic concepts. For example, Rust clearly defines the lifetime of a variable within a scope, allowing developers to discard memory and performance killers like garbage collection (GC) while still not having to worry about manual memory management, achieving both memory safety and high performance.

From the appearance of the language, it feels a lot like high-level languages like Python/TypeScript, with first-class expressiveness, but performance is in no way inferior to C/C++, thus achieving both expressiveness and high performance.

This experience of combining expressiveness, high performance, and memory safety makes Rust skyrocket soon after its 1.0 release. Since 2016, it has been voted the most beloved language by Stack Overflow users for six consecutive years.

How to Learn Rust Well? #

Rust is so well-loved, has such a wide range of uses, and major Internet companies are embracing Rust. So how can we get through the difficult period of paradigm shift as smoothly as possible?

In the process of learning programming languages over many years and imparting experience to teams, I have summarized an effective method to learn programming languages from beginning to advanced, which is also very applicable to Rust.

I believe any language learning requires targeted learning + deliberate practice.

What I call targeted learning is to dig deep into each advanced surface knowledge point, return to the fundamentals of underlying basic knowledge, and then use methods like analogy and association to connect the dots of the basic knowledge involved. Then build up the knowledge system layer by layer from the bottom design to the top implementation, so that “spread a layer of soil, tamp it down, spread another layer”, allowing you to understand knowledge points more thoroughly and master them more firmly.

For example, Rust’s ownership and lifetime are concepts many students say they learn cloudy from books or other materials, and even if they thoroughly understand several basic rules one by one, they still feel vaguely understood.

But if we think further about the “rules for accessing values in memory” and finally return to the most basic concepts of software development like heap and stack, re-recognizing the storage method and lifetime of values on the stack, and then layer by layer upwards, we will understand it more and more.

This is the importance of returning to the fundamentals, also known as the first principles: returning to the most basic conditions of things, breaking them down into basic elements for analysis, to explore the problem to be solved.

After targeted learning, we need deliberate practice. Deliberate practice is to use exquisitely designed examples to further consolidate what you have learned through practice, and in the process try to discover unknown problems in the learning process, so that you can move from “I don’t know that I don’t know” to “I know that I don’t know”, and eventually be able to fill in the knowledge gaps in the next cycle.

This process is like the method of learning discussed by Zhu Xi in The Doctrine of the Mean: learn extensively, examine thoroughly, think carefully, distinguish clearly, practice earnestly. That’s how we should learn. We should never rest until we’ve truly mastered what we set out to learn, constantly improving ourselves in the cycle of learning - building - reflecting. Learning Rust is the same.

Based on this learning approach, in this column, I will take you step-by-step to explore Rust’s basic concepts and knowledge, development principles and methods, with the goal of mastering the essence of Rust development. At the same time, each part of the content will consolidate your knowledge and fill in gaps through one or more hands-on projects.

Specifically, the entire column will be divided into five modules:

1. Prerequisites

Before formally learning Rust, let’s first review some basic concepts of software development: heap, stack, functions, closures, vtables, generics, synchronization and asynchrony, etc. You should know that to learn any programming language well, you must first digest the concepts involved, because programming languages are merely specific expressions and carriers of these concepts.

2. Basics

We’ll start with a get hands dirty week to intuitively feel where Rust’s charm lies, what it can do, and experience the joy of programming.

Then return to rationality, delve into Rust’s variable ownership and lifetime, and compare several mainstream memory management methods, including Rust’s memory management, C’s manual management, Java’s GC, and Swift’s ARC. Then we’ll discuss Rust’s major language features around ownership and lifetime: functional programming features, type system, generic programming, and error handling.

3. Advanced

Pascal’s father and Turing Award winner Niklaus Wirth has a famous formula: algorithms + data structures = programs. To freely use Rust to build data structures for your systems, an in-depth mastery of the type system is essential.

In Rust, you can use Traits for interface design, generics for compile-time polymorphism, and trait objects for runtime polymorphism. Using Traits and generics well in your code can solve complex problems very efficiently.

Then we’ll introduce unsafe Rust, don’t be scared by the name. So-called unsafe just steps back the strict checking done by the Rust compiler to be like C++, with the developer taking responsibility for the correctness of the code they write.

Finally we’ll talk about FFI, which is the bridge for Rust to interoperate with other languages. Mastering FFI well allows you to provide higher performance for your mainstream languages like Python/JavaScript/Elixir/Swift on critical paths using Rust, and also makes it easy to introduce specific libraries from the Rust ecosystem.

4. Concurrency

No language provides such extensive support for concurrency primitives while still ensuring concurrency safety like Rust does, so Rust dares to call itself Fearless Concurrency. In the concurrency section, I’ll take you from atomics all the way up through Mutex, Semaphore, Channels, to the actor model. Concurrency techniques touted as best practices in other languages are just one concurrency tool among many in Rust.

Rust also has the currently most excellent asynchronous processing model. I believe that in time, this clever use of state machines to achieve zero-cost abstraction asynchronous handling will surely be adopted in more emerging languages.

In the area of concurrency, Rust is like the Sagittarius Cloth, providing you with swords, spears, axes, tridents, hooks, and forks, so you can use the most appropriate tool to solve the most appropriate problem.

5. Practice

Mastering a language’s features and being able to apply those features to write code that solves some small problems is just catching a glimpse of the doorway, like practicing surfing in a swimming pool. To really integrate and apply the language, you still have to temper it in the winds and waves. In this section, we will learn how to apply Rust in production environments, how to use Rust’s programming ideas to solve real problems, and finally how to build complex software systems with Rust.

Throughout the column, I will try to write in a popular and easy to understand way, and draw analogies to different languages for each knowledge point, striving to help you understand the design logic behind Rust’s numerous concepts. In each lecture I will highlight key points, clarify knowledge threads, and then help you integrate each knowledge point through a series of progressive hands-on projects.

I sincerely hope that through learning this column, you can start from the basic concepts, take step-by-step towards overcoming the peak of ignorance, cross the valley of despair, and move towards the origin of perpetuity! With some effort, you can eventually build all kinds of systems with Rust yourself, and make Rust a tool for the future in your career.

After subscribing, click here to join the “Rust Language Beginners Exchange Group” and learn Rust together.