Supplemental Course: Code as Data - Why Do We Need Macro Programming Capabilities? #

Hello, I am Chen Tian.

At the request of many students, today we will talk about macro programming.

When designing the course initially, considering the systematic nature of the knowledge points, Rust’s metaprogramming capabilities, declarative macros, and procedural macros were each arranged in a separate lecture. However, macro programming is advanced content and was later cut out. Actually, if you are just starting to learn Rust, you don’t need to delve too deeply into macros. For most application scenarios, using macros provided by the standard library or third-party libraries is sufficient. Not knowing how to write macros does not affect your day-to-day development.

However, many students are interested in macros, so let’s talk about them in depth today. Before we talk about how to use macros and how to build them, we need to figure out why macros exist.

Why Do We Need Macro Programming Capabilities? #

Let’s start with the very unique design of the Lisp language. In the world of Lisp, there is a famous saying: “Code is data, and data is code.”

If you have some experience with Lisp-related development, or have heard of any Lisp dialect, you might know that unlike other programming languages, Lisp directly exposes the AST (Abstract Syntax Tree) to developers. Every line of code written by the developer is essentially describing the AST of that code.

If this feature is not clear to you, let’s understand it with a specific example. This snippet of code is from 6 years ago, when the 2048 game was very popular, and I wrote an implementation of 2048 using Racket, a dialect of Lisp:

; e.g. '(2 2 2 4 4 4 8) -> '(4 2 8 4 8)
(define (merge row)
  (cond [(<= (length row) 1) row]
        [(= (first row) (second row))
         (cons (* 2 (first row)) (merge (drop row 2)))]
        [else (cons (first row) (merge (rest row)))]))

The algorithm of this code is not hard to understand. Given a row:

• If it only has one value, then return directly;
• If the first two elements are the same, then multiply the first element by 2 and combine it with the result of merging all elements after the first two (this involves recursion), to form and return a new list;
• Otherwise, just combine the first element with the result of merging all subsequent elements into a new list (this also involves recursion).

Looking at this code, I believe you can describe the corresponding syntax tree with some patience:

You’ll find that writing Lisp code is equivalent to directly describing a syntax tree.

From the perspective of syntax trees, programming languages are not that impressive. The data structures that they operate on and execute are just trees like this, the same as the various data structures we developers manipulate in our day-to-day programming.

If a programming language exposes its syntax tree to the developer during the parsing process and allows the developer to prune and graft the syntax tree, then the language possesses metaprogramming capabilities.

The less the language restricts such processing, the stronger the metaprogramming capabilities. But as the flip side of the same coin, the language will be overly flexible, lawless, or even bite back at the language itself; conversely, the more restrictions the language imposes on the manipulation of the syntax tree by developers, the weaker the metaprogramming capabilities. Although it loses flexibility, the language becomes more orderly.

The Lisp language, as the pinnacle of metaprogramming capabilities, exposes the syntax tree to the developer without reservation, allowing developers to change the behavior of the code at will, not only at compile-time but also at runtime. This is also the direct embodiment of the Lisp philosophy of “code as data, data as code”.

In the book “Hackers and Painters” (p196), PG cites the “Greenspun’s Tenth Rule”:

Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.

Although this is Lisp enthusiasts’ extreme sarcasm about other languages, it also points out an unquestionable fact: no matter how ingeniously designed and rich in ecosystem a language is, in actual usage, it inevitably needs to have some capability to generate code from code, to greatly relieve developers from the need of constantly writing repetitive, structurally and pattern-same scaffolding code.

Fortunately, the Rust language offers strong enough macro programming capabilities that allow us to avoid repetitive scaffolding code when necessary, while also imposing enough restrictions to prevent us from overusing macros and losing control of the code.

So what macros does Rust provide exactly?

What Support Does Rust Provide for Macro Programming? #

In past courses, we have encountered a variety of macros, such as the vec! macro for creating Vec, #[derive(Debug, Default, ...)] for adding trait support to data structures, and the #[cfg(test)] macro for conditional compilation, and so on.

Actually, there are only two main types of macros in Rust: declarative macros that do simple code template replacements, and procedural macros that can be deeply customized and generate code.

Declarative Macros #

First are the declarative macros, such as vec![], println!, and info! we’ve seen in the course.

Declarative macros can be described with macro_rules!. Let’s look at a common tracing log macro definition (code):

macro_rules! __tracing_log {
    (target: $target:expr, $level:expr, $($field:tt)+ ) => {
        $crate::if_log_enabled! { $level, {
            use $crate::log;
            let level = $crate::level_to_log!($level);
            if level <= log::max_level() {
                let log_meta = log::Metadata::builder()
                    .level(level)
                    .target($target)
                    .build();
                let logger = log::logger();
                if logger.enabled(&log_meta) {
                    logger.log(&log::Record::builder()
                        .file(Some(file!()))
                        .module_path(Some(module_path!()))
                        .line(Some(line!()))
                        .metadata(log_meta)
                        .args($crate::__mk_format_args!($($field)+))
                        .build());
                }
            }
        }}
    };
}

You can see that it mainly packages the repetitive logic with a simple interface. Then it expands where it’s called without involving the operation of the syntax tree.

If you have used C/C++, Rust’s declarative macros are similar to macros in C/C++, serving the same purpose. However, Rust’s declarative macros are safer. You cannot put expressions where identifiers are required, nor can variables defined within the macro pollute the outside world. For example, in C, you can declare a macro like this:

#define MUL(a, b) a * b

This macro expects the caller to pass in two identifiers to perform the multiplication operation of the values corresponding to these identifiers. However, we can actually pass 1 + 2 for a and 4 - 3 for b, resulting in a completely wrong outcome.

Procedural Macros #

In addition to simple replacement declarative macros, Rust also supports procedural macros that allow us to deeply manipulate and rewrite Rust’s code syntax tree, which are more flexible and powerful.

There are three kinds of procedural macros in Rust:

1. Function-like macros: These look like functions but are processed at compile time. For example, the query macro we used before in sqlx expands into an expand_query function macro. You may not imagine how huge the internal code structure is for a seemingly simple query process.
2. Attribute macros: They can add attributes to other code blocks and provide more functions to those blocks. For example, rocket’s get/put routing attributes.
3. Derive macros: They add new functionalities to the derive attribute. These are the macros we use most often, such as #[derive(Debug)], which provides Debug trait implementation for our data structures, and #[derive(Serialize, Deserialize)], which provides serde related trait implementations.

When Can We Use Macros #

As mentioned earlier, the main purpose of macros is to avoid us having to create a large number of structurally similar scaffolding codes. So when can we use macros?

First, let’s talk about declarative macros. If repetitive code cannot be encapsulated with functions, then declarative macros are a good choice, such as the try! from early versions of Rust, which is the predecessor to the ? operator.

For example, the ready! macro from the futures library:

#[macro_export]
macro_rules! ready {
    ($e:expr $(,)?) => {
        match $e {
            $crate::task::Poll::Ready(t) => t,
            $crate::task::Poll::Pending => return $crate::task::Poll::Pending,
        };
    };
}

Such a structure cannot be encapsulated with a function because it involves an early return, so using a declarative macro is very concise.

Among procedural macros, let’s first talk about the most complex derive macros. Since derive macros are used in specific scenarios, you can consider using them if needed.

For instance, a data structure that we hope can provide the capability of the Debug trait, but implementing the Debug trait for each self-defined data structure is too cumbersome, and the operations performed by the code are the same. At this time, you can consider using a derive macro to simplify this operation.

In general, if the trait you defined has a fixed pattern for others to implement, then you can consider constructing a derive macro for it. One reason serde is so popular and easy to use in the Rust world is largely because basically, you only need to add #[derive(Serialize, Deserialize)] to your data structure, and you can easily serialize it into many different types like JSON, YAML, etc. (or deserialize it from these types).

There are no specific scenarios for function macros and attribute macros. sqlx uses function macros to handle SQL queries, and tokio uses an attribute macro #[tokio::main] to introduce runtime. They can help make the logic implementation of the target code simpler, but generally, unless it is particularly necessary, I do not recommend writing them.

Okay, by now you have learned enough about the basics of macros. Feel free to share your understanding of macros in the comments.

If you are interested in writing macros, in the next lecture, we will handcraft declarative and procedural macros to understand in-depth what exactly macros do. See you in the next lecture!