39 Django to Build a Monitoring Platform

39 Django to Build a Monitoring Platform #

Hello, I am Jingxiao.

Through the previous lessons, I believe you have a basic understanding of quantitative trading systems and can build a simple quantitative trading system using your own code to make profits.

In the previous lessons, our focus was on backend code, middleware, distributed systems, and design patterns. In this lesson, we will focus on frontend interaction.

Monitoring and operations are crucial components in the Internet industry. The purpose of monitoring is to prevent problems before they occur. Through monitoring, we can promptly understand the operational status of enterprise networks. Once security risks arise, you can issue timely warnings or notify operations and maintenance personnel in other ways, giving them time to handle and resolve the issues. This helps avoid disrupting the normal use of business systems and nip all problems in the bud.

In large Silicon Valley Internet companies, monitoring and operations are referred to as SRE (Site Reliability Engineering), which is a crucial part of the company’s normal operations. Facebook, as a billion-level company, naturally has various monitoring systems and operations tools, some targeting business data, others focusing on server health, and others controlling information for databases and microservices.

Nevertheless, despite the ever-changing landscape, the most important aspect of operations work is maintaining system stability. In addition to familiarizing oneself with various tools to improve operational efficiency, managing cloud resource costs, security, and monitoring all require a considerable amount of effort and time. Operations monitoring is not achieved overnight but rather evolves and progresses alongside business development.

As the final section of quantitative practice, today we will use the Django web framework to build a simple quantitative monitoring platform.

Introduction and Installation of Django #

Django is a free and open-source web framework developed with Python. It can be used to quickly build elegant and high-performance websites. It follows the “Model-View-Controller” (MVC) framework pattern, where M stands for Model, V stands for View, and C stands for Controller.

One of the biggest features of Django is that it simplifies the complex relationships between web pages and databases into simple relationships in Python. Its design goal is to make common web development tasks fast and easy. Django is an open-source project focused on solving a series of issues encountered in web development, rather than being a commercial or research project. Therefore, Django continues to improve every day to meet the evolving development needs. This not only saves development time but also improves efficiency in later maintenance.

After talking about all this, let’s further understand Django by getting hands-on experience. Let’s start with how to install and use Django. You can follow the code block below to install Django:

pip3 install Django
django-admin --version

Output:

2.2.3

Next, let’s create a new Django project:

django-admin startproject TradingMonitor
cd TradingMonitor/
python3 manage.py migrate

Output:

Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying admin.0003_logentry_add_action_flag_choices... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying auth.0010_alter_group_name_max_length... OK
Applying auth.0011_update_proxy_permissions... OK
Applying sessions.0001_initial... OK

At this point, you will see that the file system is roughly like this:

TradingMonitor/
├── TradingMonitor
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── db.sqlite3
└── manage.py

Let me explain briefly what each file means:

  • TradingMonitor/TradingMonitor represents the initial Python package of the project.
  • TradingMonitor/__init__.py is an empty file that declares the directory as a Python package.
  • TradingMonitor/settings.py manages the project’s configuration information.
  • TradingMonitor/urls.py declares the mapping relationship between requested URLs.
  • TradingMonitor/wsgi.py represents the communication protocol between Python programs and web servers.
  • manage.py represents a command-line tool used for interacting with Django projects.
  • db.sqlite3 represents the default database, which can be replaced with other databases in the settings.

In addition, you may have noticed the python3 manage.py migrate command in the above commands. This command is used to create or update the database schema. Whenever the model source code is modified and needs to be applied to the database, this command should be executed.

Next, let’s add an admin account for this system:

python3 manage.py createsuperuser

Output:

Username (leave blank to use 'ubuntu'): admin
Email address:  
Password: 
Password (again): 
Superuser created successfully.

Then, let’s start Django’s debugging mode:

python3 manage.py runserver

Finally, open a browser and go to: http://127.0.0.1:8000. If you can see the following page, it means that Django has been successfully deployed.

Home Page

Isn’t the installation of Django very simple? This is actually Python’s consistent philosophy: simplicity and a simplified learning curve.

OK, now let’s navigate to http://127.0.0.1:8000/admin. You will see Django’s admin website, but I won’t go into too much detail here.

Admin Page - Admin Login Page

At this point, Django has been successfully installed and started running.

MVC Architecture #

As I mentioned earlier, the MVC architecture is the essence of the design pattern in Django. Next, let’s take a closer look at this architecture and build a server using Django.

Designing Models #

In the previous lesson on logging and storage systems, I introduced the peewee library, which allows us to operate MySQL without using complex SQL statements, but instead directly use Python classes for transformation. In fact, this is also the approach taken by Django.

Django can be used without a database by using an object-relational mapper (ORM), which allows us to describe data structures using only Python code.

Let’s first take a look at the following Model code:

# TradingMonitor/models.py

from django.db import models


class Position(models.Model):
    asset = models.CharField(max_length=10)
    timestamp = models.DateTimeField()
    amount = models.DecimalField(max_digits=10, decimal_places=3)

The models.py file mainly uses a Python class to describe a data table, called a model. With this class, you can create, retrieve, update, and delete records in the database using simple Python code, without having to write SQL statements, which is what we previously mentioned about avoiding database operations using SQL.

Here, we create a Position model to represent our trading position information. In this model:

  • asset represents the code of the currently held asset, such as btc;
  • timestamp represents the timestamp;
  • amount represents the position information at the timestamp.

Designing Views #

After the model is defined, we can reference it in the views. Generally, the views retrieve data based on parameters, load a template, and use the retrieved data to render the template.

Views are where we implement business logic. Let’s take a look at the render_positions code, which takes two parameters: request and asset. For now, we won’t worry about request. Here, asset specifies an asset name, such as btc, and then this function returns a rendered page.

# TradingMonitor/views.py

from django.shortcuts import render
from .models import Position

def render_positions(request, asset):
    positions = Position.objects.filter(asset=asset)
    context = {'asset': asset, 'positions': positions}
    return render(request, 'positions.html', context)

But how does this function work? Let’s go through it line by line.

positions = Position.objects.filter(asset=asset) This line of code performs a query on the database, where filter is used to select all the necessary asset information from the database. However, I’m just using this as an example; when actually monitoring, we usually read information from the database in a more targeted way instead of reading all the information at once.

context = {'asset': asset, 'positions': positions} This line of code simply packages a dictionary. As for the purpose of this dictionary, it will be explained below.

return render(request, 'positions.html', context) Finally, this line of code returns a page. We adopt the template design here, which is also the development approach highly recommended by Django. It separates the template from the data, so the data only needs to be filled in.

The template file used here is position.html. You should notice that context is passed to the template as a variable. Now let’s take a look at the content of the template.

Designing Templates #

A template file is actually a combination of HTML code and some code. You can think of it as if this HTML needs to be preprocessed by us before it is sent to the user. The way to preprocess it is to find the corresponding places for replacement.

Let’s take a look at the following example code:

<!-- TradingMonitor/templates/positions.html -->

<!DOCTYPE html>
<html lang="en-US">
<head>
<title>Positions for {{asset}}</title>
</head>

<body>
<h1>Positions for {{asset}}</h1>

<table>
<tr>
    <th>Time</th>
    <th>Amount</th>
</tr>
{% for position in positions %}
<tr>
    <th>{{position.timestamp}}</th>
    <th>{{position.amount}}</th>
</tr>
{% endfor %}
</table>
</body>

I will explain a few key points. First is <title>Positions for {{asset}}</title>, where double curly braces enclose the asset variable, which corresponds to the asset key in the previous context dictionary. Django’s rendering engine will replace asset with the corresponding content in context, which in this case is replaced with btc.

Next, {% for position in positions %}, this is a crucial part. We need to handle a list, so we iterate over positions using for. Here, positions also corresponds to positions in the context.

The {% endfor %} at the end naturally indicates the end. This way, we encapsulate the data into a list.

Designing URLs #

Finally, we need to provide URL interfaces for our operations. I put the specific operations in the following code, which is relatively simple, so I won’t go into detail.

# TradingMonitor/urls.py

from django.contrib import admin
from django.urls import path
from . import views

urlpatterns = [
    ...
]

path('admin/', admin.site.urls),
path('positions/<str:asset>', views.render_positions),
]

Here, we can access it through http://127.0.0.1:8000/positions/btc!

Testing #

Of course, besides the main process, I also need to emphasize a few simple but very important details, otherwise, our changes cannot be applied properly.

Step 1: Under TradingMonitor/TradingMonitor, create a new folder called migrations; and in this folder, create an empty file called __init__.py.

mkdir TradingMonitor/migrations
touch TradingMonitor/migrations/__init__.py

At this point, your directory structure should look like this:

TradingMonitor/
├── TradingMonitor
│   ├── migrations
│       └── __init__.py
│   ├── templates
│       └── positions.html
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   ├── models.py
│   ├── views.py
│   └── wsgi.py
├── db.sqlite3
└── manage.py

Step 2: Modify TradingMonitor/settings.py:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'TradingMonitor',  # Add our app here
]



TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [os.path.join(BASE_DIR, 'TradingMonitor/templates')],  # Add the templates directory here
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

Step 3: Run python manage.py makemigrations:

python manage.py makemigrations

########## Output ##########

Migrations for 'TradingMonitor':
  TradingMonitor/migrations/0001_initial.py
    - Create model Position

Step 4: Run python manage.py migrate:

python manage.py migrate

########## Output ##########


Operations to perform:
  Apply all migrations: TradingMonitor, admin, auth, contenttypes, sessions
Running migrations:
  Applying TradingMonitor.0001_initial... OK

I have shown the specific operations for these steps with code and comments, so you can follow along and perform the operations. After completing the operations, our data structure has been successfully synchronized to the database.

Finally, enter python manage.py runserver and then open the browser and enter http://127.0.0.1:8000/positions/btc, and you will see the effect. Here

Now, let’s take a look back at the MVC pattern. Through this diagram I drew, you can see that the M, V, and C are connected together in a plugin-like and loosely coupled manner:

Here

Of course, what I have guided you to write is just a simple Django application. For a real quantitative platform monitoring system, this is only a simple beginning.

In addition, for a monitoring system, there are actually many open-source plugins that can be used. Some of them have very cool interfaces and some can achieve high stability and ease of use. Many of them can work well with Django. Some typical ones include:

  • Graphite, a plugin for storing time series data and displaying them in graphs through a Django web application.
  • Vimeo, a dashboard based on Graphite with additional features and a smooth design.
  • Scout, which monitors the performance of Django and Flask applications, providing automatic detection of views, SQL queries, templates, etc.

Summary #

The content of this lesson is closer to the upstream application layer. Taking Django, a Python backend, as an example, we explained the process of setting up a server. You should have noticed that building a server using the RESTful Framework is such a simple process that you can start your own exchange (lol). Compared to specific technologies, the MVC framework and the philosophy of Django that I talked about today are more worthy of your in-depth learning and understanding.

Thought Question #

Today I want to give you a challenging assignment. How is secure authentication implemented in Django for RESTful APIs? Can you learn and master this knowledge point through searching and self-study? I hope to see your diligent learning records and summaries in the comments section, and I will provide suggestions one by one. You are also welcome to share this article with your friends and colleagues for discussion and progress together.