34 RESTful & Socket to Build the Transaction Execution Core #

Hello, I’m Jingxiao.

In the previous section, we briefly introduced the history of quantitative trading, its precise definition, and its basic components. With this high-level basic knowledge, we will now break it down into modules and start explaining specific parts of the quantitative trading system.

Starting from this lesson, we will actually start from the code and step by step design a clear, complete, and easy-to-understand quantitative trading system.

A quantitative trading system can be seen as a black box. This black box connects to the data obtained from the exchange, performs strategy calculations, and then connects to the exchange to place orders. As we mentioned in the lesson on input and output, the characteristics of the black box are input and output. Every student who designs network interactions needs to form a clear interaction state diagram in their mind:

• Knowing how packets are transmitted between networks;
• Knowing how each node processes different input packets, then outputs and distributes them to the next level.

When you don’t understand, you can first draw an interaction topology diagram on a draft paper, clearly label the input and output formats of each node, and then think about how the network flows. This is crucial for network programming.

Now, I assume that you only have a basic understanding of network programming. So, next, I will start with the definition of REST, and then transition to specific interaction methods - how to interact with exchanges through Python, and thus perform network interactions such as placing orders, canceling orders, and querying orders.

Introduction to REST #

What is a REST API? What is a socket? If you have experience in network programming, you must be familiar with these two terms.

The full name of REST is “Representational State Transfer.” Originally, it referred to a method of operating resources. However, you don’t need to get caught up in this complicated name. In simpler terms, the essence of REST can be understood as: using URLs to locate resources and using verbs such as GET, POST, PUT, DELETE, etc. to describe operations. Interfaces that meet the requirements of REST are called RESTful interfaces.

To help you better understand these concepts, let me give you an analogy. Xiao Ming is not very smart but very obedient. Every day, he makes tea for his mother when she comes home from work. At first, his mother made this request:

Use a red cup to make a cup of Pu’er tea with sugar at 37.5 degrees in the kitchen.

But Xiao Ming is not smart enough and finds it difficult to understand this long sentence. So, his mother simplified the instructions to make it easier for him to understand:

Make tea in the kitchen with the following requirements:

1. Type = Pu’er;
2. Cup = red;
3. Add sugar = true;
4. Temperature = 37.5 degrees.

Here, “tea” is the resource, and “tea in the kitchen” is the address (URI) of the resource. “Make” is the verb, and the requirements mentioned afterwards are the interface parameters. This kind of interface is a RESTful interface provided by Xiao Ming.

If Xiao Ming were a machine, it would be very easy to parse this request. As maintainers, we can also easily check Xiao Ming’s code. When Xiao Ming exposes this interface online, it becomes a RESTful interface.

In general, RESTful interfaces usually appear in the form of HTTP GET and POST. However, not all GET and POST request interfaces are RESTful interfaces.

This might sound a bit confusing, so let’s look at an example. In the previous lesson, we obtained the ticker interface for the BTC to USD price on the Gemini exchange:

GET https://api.gemini.com/v1/pubticker/btcusd

Here, “GET” is the verb, and the URI after it is the address of the “Ticker” resource. So, this is a RESTful interface.

But the following interface is not strictly RESTful:

POST https://api.restful.cn/accounts/delete/:username

Because the URI contains the verb “delete,” it does not point to a resource. To make it a strict RESTful interface, we can change it to:

DELETE https://api.rest.cn/accounts/:username

Then, let’s take a look at the order cancellation interface for Gemini:

POST https://api.gemini.com/v1/order/cancel

You will find some non-RESTful aspects of this interface:

• The verb is inaccurately designed, using “POST” instead of reusing the HTTP verb “DELETE.”
• The URI includes the verb “cancel.”
• The order ID, which represents a resource, is placed in the parameter list instead of the URI, so the URI does not point to the resource.

Therefore, strictly speaking, this is not a RESTful interface.

Additionally, if we examine other private interfaces of Gemini (private interfaces require additional authentication information to access), we will find that their design is not strictly RESTful either. Moreover, most exchanges, such as Bitmex, Bitfinex, OKCoin, etc., provide “REST interfaces” that are also not strictly RESTful. These interfaces can still be called “REST interfaces” because they mostly satisfy another important requirement of REST interfaces: statelessness.

Being stateless means that each REST request is independent and does not require the server to cache intermediate state in a session to complete the request. In simple terms, if server A receives a request and goes offline, but this request can still be sent to server B at the exchange to continue the process, then this interface is stateless.

Here, let me give you an example of a simple stateful interface. The server requires the client to send two different HTTP requests when requesting to cancel an order. And the first request is to make the server “wait for cancellation,” and the second request is to “confirm cancellation.” Even if this interface meets the separation principle of REST verbs and resources, it is not a REST interface because it is stateful.

Of course, when dealing with exchange REST interfaces, you don’t need to get too caught up in the concept of “RESTful.” Otherwise, it is easy to get confused by these terms. The most important thing you need to understand is: one HTTP request completes one complete operation.

Introduction to Exchange APIs #

Now that you should have a general understanding of REST and WebSocket, let’s move on to something more interesting.

First, let me introduce what an exchange is. A blockchain exchange is a matching trading platform that combines traditional matching rule matching engines with blockchain-based fund custody and settlement methods. A cryptocurrency exchange, on the other hand, is a centralized platform that allows users to deposit digital assets into a designated wallet address (created by the exchange) and then place buy or sell orders on the platform to exchange digital assets.

Simply put, an exchange is like a marketplace where people buy and sell goods. There are people shouting at their stalls, saying, “I have two pounds of lamb, two pounds of lamb, I’m willing to exchange it for four pounds of beef!” These people are called makers (order placers). There are also people who casually walk around different stalls and take away one pound of lamb while placing two pounds of beef. These people are called takers (order takers).

The purpose of an exchange is twofold: to provide enough space for makers and takers to operate, and to allow a thing called a matching engine to match orders as much as possible, while charging a certain percentage of fees, um, I mean transaction fees, to ensure the continuation of the game.

Clearly, the market is a great invention, but let’s not delve into further philosophical discussions here.

Next, let me introduce an exchange called Gemini. Gemini is the world’s first licensed and regulated digital currency exchange and custodian, and the first to launch futures contracts. It focuses on matching large trades. Located in New York, Gemini is regulated directly by the New York State Department of Financial Services (NYDFS).

Gemini has a clear interface, comprehensive and user-friendly APIs, and, most importantly, a complete test network. This means that the functionality is exactly the same as the real Gemini. However, their trades use virtual currency, which is very convenient for industry practitioners to conduct integration testing on the platform.

Another well-established exchange is Bitmex. Their API UI and test network are also top-tier in the cryptocurrency industry. However, as Bitmex is a futures exchange, it may have a certain learning curve for beginners in quantitative trading, so we choose Gemini as it is more convenient.

Before we get into the main topic, let me introduce four basic concepts using the example of trading between Bitcoin and the US dollar (we won’t go into the concept of an order book here, so you don’t need to worry about it, you just need to know the price of Bitcoin).

• Buy: The act of buying Bitcoin with US dollars.
• Sell: The act of exchanging Bitcoin for US dollars.
• Market Order: A type of order in which you provide the exchange with a direction (buy or sell) and an amount, and the exchange converts the given amount of US dollars (or Bitcoin) into Bitcoin (or US dollars).
• Limit Order: A type of order in which you provide the exchange with a price, a direction (buy or sell), and an amount, and the exchange converts the given amount of US dollars (or Bitcoin) into Bitcoin (or US dollars) when the price reaches the given price.

These concepts are easy to understand. The main difference between market orders and limit orders is that limit orders include a specified price. How can we understand this? Let’s take a look at the following example.

On a certain day at 12:00:00, Xiao Ming tells the exchange that he wants to buy Bitcoin with $1000. The exchange receives the message and at 12:00:01 replies to Xiao Ming, saying that his account now has 0.099 Bitcoin and is short $1000, and the transaction is successful. This is a market buy order.

On a certain day at 11:59:00, Xiao Qiang tells the exchange that he wants to place an order for 0.1 Bitcoin, with a price of $10000 per Bitcoin, and he will not sell below this price. The exchange receives the message and at 11:59:01 tells Xiao Qiang that the order has been successfully placed and $100 worth of Bitcoin in his account has been frozen. After another minute, the exchange tells Xiao Qiang that his order has been fully executed, and now his account has gained $1000 and lost 0.1 Bitcoin. This is a limit sell order.

(Someone might notice that something is not right here: it seems that some Bitcoin is missing. Well, hehe, why don’t you try guessing yourself?)

Clearly, market orders are executed immediately after being submitted to the exchange, but the execution price is not under your control. They are fast but also very insecure. On the other hand, limit orders specify the trading price and quantity, with higher security. The disadvantage, of course, is that if the market moves in the opposite direction, your order may not be matched by anyone and you end up shouting but no one buys. Since I haven’t explained the order book, the wording here may not be completely accurate, but it should be enough for beginners to understand today’s content.

Having accumulated all this basic knowledge, you must be eager to try it out, right? In the next section, we will formally dive into the topic and guide you step by step on how to place orders using the API.

Step-by-Step Guide to Using API for Placing Orders #

Manually placing orders is slow and does not align with the purpose of algorithmic trading. Let’s see how we can automate order placement using code.

The first step you need to take is to register a Gemini Sandbox account. Don’t worry, this test account doesn’t require you to deposit any funds. After registration, you will be given a large amount of virtual cash. It sounds like a line from a video game promotion, doesn’t it? Haha, but that’s how it is, so hurry up and register.

After registering, out of curiosity, you can try placing orders using the web interface. However, in reality, you won’t be able to place orders without unlocking the feature, so this attempt doesn’t have much significance.

So the second step is to configure the API Key. Go to User Settings -> API Settings in the menu bar, and then click on GENERATE A NEW ACCOUNT API KEY. Take note of the Key and Secret, as once the window disappears, you won’t be able to find them again and will need to generate them again.

Configuration is now complete. Let’s move on to the specific implementation.

First, it’s important to emphasize that when developing a quantitative trading system, you must have a clear data flow diagram in mind. Placing an order is a simple RESTful process, similar to your operations on a web page. You construct your request order, encrypt the request, and then POST it to the Gemini exchange.

However, there are many knowledge points involved, so it’s not practical to guide you step by step from scratch. Therefore, we will use the “read and understand, memorize and use” method to learn. Here is the code for that:

import requests
import json
import base64
import hmac
import hashlib
import datetime
import time

base_url = "https://api.sandbox.gemini.com"
endpoint = "/v1/order/new"
url = base_url + endpoint

gemini_api_key = "account-zmidXEwP72yLSSybXVvn"
gemini_api_secret = "375b97HfE7E4tL8YaP3SJ239Pky9".encode()

t = datetime.datetime.now()
payload_nonce = str(int(time.mktime(t.timetuple())*1000))

payload = {
   "request": "/v1/order/new",
   "nonce": payload_nonce,
   "symbol": "btcusd",
   "amount": "5",
   "price": "3633.00",
   "side": "buy",
   "type": "exchange limit",
   "options": ["maker-or-cancel"]
}

encoded_payload = json.dumps(payload).encode()
b64 = base64.b64encode(encoded_payload)
signature = hmac.new(gemini_api_secret, b64, hashlib.sha384).hexdigest()

request_headers = {
    'Content-Type': "text/plain",
    'Content-Length': "0",
    'X-GEMINI-APIKEY': gemini_api_key,

'X-GEMINI-PAYLOAD': b64,
'X-GEMINI-SIGNATURE': signature,
'Cache-Control': "no-cache"
}
    
response = requests.post(url,
                         data=None,
                         headers=request_headers)
    
new_order = response.json()
print(new_order)

########## Output ##########

{'order_id': '239088767', 'id': '239088767', 'symbol': 'btcusd', 'exchange': 'gemini', 'avg_execution_price': '0.00', 'side': 'buy', 'type': 'exchange limit', 'timestamp': '1561956976', 'timestampms': 1561956976535, 'is_live': True, 'is_cancelled': False, 'is_hidden': False, 'was_forced': False, 'executed_amount': '0', 'remaining_amount': '5', 'options': ['maker-or-cancel'], 'price': '3633.00', 'original_amount': '5'}

Let’s take a closer look at this code.

This is a RESTful POST request implemented using requests.post. The post function takes three parameters: url, data, and headers.

Here, the url is equivalent to "https://api.sandbox.gemini.com/v1/order/new", but it is split into two parts in the code. The first part is the exchange’s API address, and the second part, starting with a forward slash, represents a unified API endpoint. You may see similar syntax used in APIs of other exchanges, where both parts combined form the final url.

The purpose of the rest of the code is to construct request_headers.

Let me briefly explain HTTP requests here. HTTP is the basic protocol in the internet world based on TCP. HTTP, short for Hyper Text Transfer Protocol, is a protocol used to transfer hypertext from a WWW (World Wide Web) server to a local browser. TCP (Transmission Control Protocol), on the other hand, is a reliable, connection-oriented, byte-stream protocol at the transport layer.

As a side note, if you’re developing network programs, I suggest you take the time to read the book “Computer Networking: A Top-Down Approach” by Kurose and Ross. It is widely used as a textbook in computer science programs worldwide. By learning while applying, you can improve your skills comprehensively.

Back to HTTP, it has the main features of being simple and flexible. It can use a “simple request, receive a response and disconnect” approach and is a stateless protocol, which aligns well with the RESTful philosophy.

HTTP requests require a request header, which is represented as a dictionary in Python. In this case, it is the request_headers dictionary, where some fields have special meanings. 'Content-Type': "text/plain" and 'Content-Length': "0" describe the type and length of the content, which corresponds to the data parameter. However, in this Gemini request, the data has no use and has a length of 0.

There are also other fields, such as 'keep-alive', which indicate whether the connection can be kept alive. It’s a good practice to pay attention to these details, as many bugs in network programming occur in seemingly insignificant areas.

Let’s continue with the code. The payload is an important dictionary used to store all the information needed for the order operation, which is the business logic information. Here, we place a limit buy order with a price of $3633.

Also, please note the nonce, which is a crucial field commonly used in network communication.

Since network communication is prone to errors, with a packet possibly being lost or duplicated, both of these situations can have serious consequences in financial operations. If a packet is lost, we can simply resend it; however, when a packet is duplicated, we need to deduplicate it. Although TCP can ensure reliability to a certain extent, Gemini requires all communication payloads to have a nonce to further reduce the chance of errors in the application layer.

A nonce is a monotonically increasing integer. When the nonce of a later request is smaller or equal to the nonce of a previously successfully received request, Gemini rejects the request. This ensures that duplicated packets are not executed twice. Moreover, this also helps prevent man-in-the-middle attacks:

• Firstly, the inclusion of a nonce makes the encrypted text of the same order completely chaotic.
• Secondly, it makes it impossible for a man-in-the-middle to construct a duplicate order by sending the same packet.

Isn’t this design ingenious? It’s like adding an identity marker to each packet, greatly enhancing security. I hope you also pay attention to and contemplate these clever uses.

The following code becomes clear. We need to encrypt the payload using base64 and sha384 asymmetric encryption, where gemini_api_secret is the private key. The exchange stores the public key, which decrypts the requests you send. Finally, the code packages the encrypted request into request_headers, sends it to the exchange, and receives a response, completing the order.

Summary #

In this lesson, we introduced what RESTful API is and how the RESTful API of an exchange works. We also showed you how to place orders using the RESTful API. Additionally, I briefly discussed some techniques and operations in network programming. I hope you will pay attention to every detail in network programming and have a clear understanding of the business logic and specific technical details before writing code.

In the next lesson, we will start with the definition of Web Socket and explain the specific implementation of data modules in quantitative trading.

Thought Question #

Finally, here’s a thought question. Can we use a timestamp instead of a nonce in today’s content? Why? Feel free to leave a comment with your thoughts, and also feel free to share this article.