04 Get Hands Dirty to Write a Practical CLI Tool

get hands dirty: Writing a Useful CLI Utility #

Hi, I’m Chen Tian.

In the previous lesson we already covered Rust’s basic syntax. You must be itching to start writing some Rust code to practice, but also feeling a bit overwhelmed on where to start!

So this week we’ll try something new - do a week long “learning by example” challenge to try writing three very practical small apps in Rust to experience its charm and ability to solve real problems.

Are you worried that I’ve only just learned the basic syntax and don’t know anything yet, how can I start writing apps already? What if I run into concepts I don’t understand?

Don’t worry, you’ll definitely run into syntax you don’t fully grasp, but for now don’t stress about understanding everything, just treat it like copying classical Chinese text, even if you don’t get it all. As long as you follow along, through writing, compiling, and running you’ll be able to intuitively experience Rust’s charm, just like memorizing Tang poems as a child.

Alright, let’s start today’s challenge.

HTTPie #

To cover most learners’ needs, the examples chosen are ubiquitous in work - writing a CLI tool to aid various tasks.

We’ll use implementing HTTPie as an example to see CLI development in Rust. HTTPie is developed in Python, a user-friendlier cURL-like CLI tool that can help diagnose HTTP services better.

Below is the interface after sending a POST request with HTTPie. You can see compared to cURL, it does a lot to improve usability, including syntax highlighting of different info:

image

Think about if you were to implement HTTPie in the language you’re most familiar with, how would you design it, what libraries are needed, and roughly how many lines of code? And how many lines would it take in Rust?

With your own thoughts in mind, let’s start implementing this tool in Rust! Our goal is to implement this functionality in around 200 lines of code.

Functional Analysis #

To build a tool like HTTPie, let’s first go over the key functions needed:

  • Parse command line options, handle subcommands and various parameters, validate user input, and convert the input into internal parameters we understand
  • Based on the parsed parameters, send an HTTP request and get response
  • Finally, output the response in a user friendly way

The flow is shown again in the diagram:

image

Let’s look at the libraries needed to implement these functions:

  • For CLI parsing, Rust has many libraries that fit the need, today we’ll use the officially recommended clap
  • For HTTP client, we used reqwest in the previous lesson, we’ll continue using it, but try out its async interface this time
  • For formatted output, to make the output look lively and readable like Python HTTPie, we can use a terminal color library, here we pick the simple colored
  • In addition, we need some extra libraries: use anyhow for error handling, jsonxf to format JSON responses, mime to handle MIME types, and include tokio for async handling

CLI Handling #

Alright, with the basic approach in mind, let’s create a project called httpie:

cargo new httpie
cd httpie

Then open the project directory in VSCode and edit Cargo.toml to add the needed dependencies (Note: the code below uses beta versions of crates that may have breaking changes in the future. If unable to compile locally, refer to code in the GitHub repo):

[package]
name = "httpie"  
version = "0.1.0"
edition = "2018"

[dependencies]  
anyhow = "1" # Error handling
clap = "3.0.0-beta.4" # CLI parsing
colored = "2" # Terminal color
jsonxf = "1.1" # JSON pretty print  
mime = "0.3" # Handle MIME types
reqwest = { version = "0.11", features = ["json"] } # HTTP client
tokio = { version = "1", features = ["full"] } # Async lib

Let’s first add CLI-related code in main.rs:

use clap::{AppSettings, Clap};

// Define HTTPie's main CLI entry point with subcommands
// /// comments below are docs, used by clap for help

/// A naive httpie implementation with Rust, can you imagine how easy it is?  
#[derive(Clap, Debug)]
#[clap(version = "1.0", author = "Tyr Chen <[[email protected]](/cdn-cgi/l/email-protection)>")] 
#[clap(setting = AppSettings::ColoredHelp)]
struct Opts {
    #[clap(subcommand)]
    subcmd: SubCommand,
}

// Subcommands correspond to different HTTP methods, currently only get/post
#[derive(Clap, Debug)]
enum SubCommand {
    Get(Get),
    Post(Post),
    // We don't support other HTTP methods for now
}

// get subcommand 

/// Feed get with a url and we'll retrieve the response for you
#[derive(Clap, Debug)]
struct Get {
    /// The HTTP URL
    url: String,
}

// post subcommand. Takes a URL and optional key=value pairs for the JSON body.

/// Feed post with a url and optional key=value pairs. We will post the data
/// as JSON, and retrieve the response for you
#[derive(Clap, Debug)]  
struct Post {
    /// The HTTP URL
    url: String,
    /// HTTP request body   
    body: Vec<String>,
}

fn main() {
  let opts: Opts = Opts::parse();
  println!("{:?}", opts);
}

The code uses macros provided by clap to simplify CLI definition. The macro can generate additional code to help parse the CLI. With clap, we just need to describe the data structure to capture CLI data, then call T::parse() to parse various command line options. We did not define the parse() function, it’s auto-generated by #[derive(Clap)].

Currently we’ve defined two subcommands. In Rust subcommands can be defined via enum, with the parameters for each subcommand defined via their own data structures Get and Post.

Let’s run it:

❯ cargo build --quiet && target/debug/httpie post httpbin.org/post a=1 b=2
Opts { subcmd: Post(Post { url: "httpbin.org/post", body: ["a=1", "b=2"] }) }

By default the binary compiled by cargo build is under target/debug in the project root. We can see command line parsing works as expected.

Add Validation #

But right now we haven’t done any validation on the user input. With input like this, the URL is completely wrong:

❯ cargo build --quiet && target/debug/httpie post a=1 b=2
Opts { subcmd: Post(Post { url: "a=1", body: ["b=2"] }) }

So we need to add validation. There are two inputs to validate - one is the URL, the other is the body.

Let’s first validate the URL is valid:

use anyhow::Result;
use reqwest::Url;

#[derive(Clap, Debug)]
struct Get {
  /// The HTTP URL  
  #[clap(parse(try_from_str = parse_url))]
  url: String,
}

fn parse_url(s: &str) -> Result<String> {
  // Just check if the URL is valid here
  let _url: Url = s.parse()?;

  Ok(s.into())
}

clap allows adding a custom parse function for each parsed value. We’ve defined parse_url here to check.

Then we need to ensure each body item is in key=value format. We can define a KvPair data structure to store this information, and also a custom parse function to put the parsed result into KvPair:

use std::str::FromStr;
use anyhow::{anyhow, Result};

#[derive(Clap, Debug)]
struct Post {
  /// The HTTP URL
  #[clap(parse(try_from_str = parse_url))]
  url: String,
  /// HTTP request body
  #[clap(parse(try_from_str=parse_kv_pair))]
  body: Vec<KvPair>, 
}

/// Key=value pairs from command line can be parsed into KvPair  
#[derive(Debug)]
struct KvPair {
  k: String,
  v: String,
}

/// By implementing FromStr we can parse strings into KvPair using str.parse()
impl FromStr for KvPair {
  type Err = anyhow::Error;

  fn from_str(s: &str) -> Result<Self, Self::Err> {
    // Split into an iterator using = 
    let mut split = s.split("=");
    let err = || anyhow!(format!("Failed to parse {}", s));
    Ok(Self {
      // Take first item from iterator as key
      // We convert Option into Result and handle errors with ?
      k: (split.next().ok_or_else(err)?).to_string(),
      // Take second item from iterator as value 
      v: (split.next().ok_or_else(err)?).to_string(),
    })
  }
} 
  
/// Since we implemented FromStr for KvPair, we can directly parse
fn parse_kv_pair(s: &str) -> Result<KvPair> {
  Ok(s.parse()?) 
}

Here we implemented the FromStr trait to convert qualifying strings into KvPair. By implementing FromStr, we can call the generic parse() method on strings to easily handle string to type conversion.

With these changes, our CLI is now more robust. We can test again:

❯ cargo build --quiet
❯ target/debug/httpie post https://httpbin.org/post a=1 b
error: Invalid value for '<BODY>...': Failed to parse b

For more information try --help
❯ target/debug/httpie post abc a=1  
error: Invalid value for '<URL>': relative URL without a base

For more information try --help

target/debug/httpie post https://httpbin.org/post a=1 b=2
Opts { subcmd: Post(Post { url: "https://httpbin.org/post", body: [KvPair { k: "a", v: "1" }, KvPair { k: "b", v: "2" }] }) }

Cool, we’ve completed basic validation, but it’s obvious we haven’t crammed all the validation code together in the main flow. Instead, validation is done via separate, decoupled validation functions and traits. This newly added code is highly reusable and independent, without modifying the main flow.

This aligns well with software development’s open-closed principle - Rust can help us more easily write well-structured, maintainable code through tools like macros, traits, generic functions, trait objects, etc.

You probably don’t understand all the code details yet, but don’t worry, keep writing for now, just get the code running today. You don’t need to fully grasp every concept, we’ll cover them all systematically later.

HTTP Request #

Next let’s continue with HTTPie’s core functionality: HTTP request handling. We’ll add subcommand handling logic to the main() function:

use reqwest::{header, Client, Response, Url};

#[tokio::main]  
async fn main() -> Result<()> {
  let opts: Opts = Opts::parse();
  // Create an HTTP client  
  let client = Client::new();
  let result = match opts.subcmd {
    SubCommand::Get(ref args) => get(client, args).await?,
    SubCommand::Post(ref args) => post(client, args).await?,
  };

  Ok(result)
}

Note we changed main() to an async fn, indicating it’s an async function. For async main, we need the #[tokio::main] macro to automatically add async runtime handling.

Inside main(), based on subcommand type we call get and post functions to handle specifics. Implementations:

use std::{collections::HashMap, str::FromStr};

async fn get(client: Client, args: &Get) -> Result<()> {
  let resp = client.get(&args.url).send().await?;
  println!("{:?}", resp.text().await?);
  Ok(()) 
}

async fn post(client: Client, args: &Post) -> Result<()> {
  let mut body = HashMap::new();
  for pair in args.body.iter() {
    body.insert(&pair.k, &pair.v);
  }
  let resp = client.post(&args.url).json(&body).send().await?;
  println!("{:?}", resp.text().await?);
  Ok(())
}

Here the parsed KvPair list needs to be put into a HashMap and passed to the client’s JSON method. This completes HTTPie’s basic functionality.

Right now the printed data is very unfriendly for users. We need to further print the HTTP headers and body in different colors for readability like Python HTTPie. This code is pretty simple so won’t go into details.

Finally, the full code:

// Code omitted for brevity

#[tokio::main]
async fn main() -> Result<()> {
  // Code omitted

  let client = reqwest::Client::builder()
    .default_headers(headers)
    .build()?;
  
  let result = match opts.subcmd {
    SubCommand::Get(ref args) => get(client, args).await?,
    SubCommand::Post(ref args) => post(client, args).await?, 
  };

  Ok(result)
}

#[cfg(test)] 
mod tests {
  use super::*;

  #[test]
  fn parse_url_works() {
    // Tests omitted
  }

  #[test]
  fn parse_kv_pair_works() {
     // Tests omitted 
  }
}

At the end I also wrote some unit tests that can be run with cargo test. Rust supports conditional compilation, here #[cfg(test)] means the entire mod tests only compiles under cargo test.

The tokei line count tool shows we used 139 lines of code total, including around 30 lines of test code:

❯ tokei src/main.rs 
-------------------------------------------------------------------------------
 Language            Files        Lines         Code     Comments       Blanks
-------------------------------------------------------------------------------
 Rust                    1          200          139           33           28
-------------------------------------------------------------------------------
 Total                   1          200          139           33           28
-------------------------------------------------------------------------------

You can use cargo build –release to compile a release version, copy it somewhere in your $PATH, then try it out:

image

And we have a fully functional HTTPie with complete help ready for use.

Let’s test the effects:

image

This is almost identical to official HTTPie. Today’s code can be found here.

Wow, huge success with this example! We implemented HTTPie’s core in just over 100 lines, way under the expected 200. Could you vaguely feel Rust’s ability to solve real problems? With today’s HTTPie for example:

  • To parse command line input into data structures, we just need some simple annotations on the data structures.
  • Data validation can be done in separate functions completely decoupled from main flow.
  • As a CLI parsing library, clap’s overall experience is very similar to Python’s click, but simpler than Golang’s cobra.

This demonstrates Rust’s capabilities - despite targeting system level development, it can provide abstraction and experience similar to Python. Once you get used to Rust, it feels amazing to use.

Summary #

Now you should vaguely understand why I said in the opening Rust has powerful expressiveness.

You may still feel confused learning this way, like blind men touching an elephant. Beginners often think they must fully grasp all syntax before writing any code. That’s not the case.

The goal of this week’s challenges writing 3 examples is to let you intuitively experience Rust’s way of handling and solving problems through fumbling your way through writing code, while also comparing with languages you know - whether Go/Java or Python/JavaScript, how would I solve this in my familiar language, what support does Rust provide, what do I feel is missing.

This process will spark all kinds of thoughts in your mind, which will inevitably lead to more and more questions. This is a good thing - bringing these questions to future lessons will let you learn more purposefully and profoundly.

Today’s challenge is not too difficult, you may want more. Don’t worry, next lesson we’ll write a slightly more difficult web service commonly used at work to continue experiencing Rust’s charm.

Review Questions #

We’ve only implemented syntax highlighting to differentiate HTTP headers and body, but the HTTP body still looks a bit ugly. We could further implement syntax highlighting. If you’ve completed today’s code and feel up for an extra challenge, feel free to try improving our HTTPie using syntect. syntect is a very powerful Rust syntax highlighting library.

Please share your thoughts in the comments below. Congrats on completing your 4th Rust learning checkpoint, see you next lesson!

Note #

Note: The code in this article uses beta crate versions which may have breaking changes in the future. If unable to compile locally, refer to code in the GitHub repo. For any similar issues going forward, also refer to latest code on GitHub. Happy learning!