21 Business Caching Metadata Services How to Implement

21 Business Caching Metadata Services How to Implement #

Hello, I am Xu Changlong.

When you casually open Weibo or a comprehensive news website, you can see a rich variety of media files, including images, text, audio, and video. A single webpage may even be composed of hundreds or thousands of files.

So where are these files stored? Typically, small amounts of text data, less than 1KB, are saved in databases, while larger text or multimedia files (such as MP4, TS, MP3, JPG, JS, CSS, etc.) are usually stored on hard disks. Managing these files is not too complicated.

However, if the number of files exceeds one million, managing them on hard disks becomes problematic. When a user request arrives at the server, dozens of servers need to find out where the files are saved from hundreds of hard disks. Additionally, regular backup and tracking of access records need to be done. All these bring great challenges to our development work.

The emergence of object storage technology has solved many of these issues by abstracting away many details, greatly improving development efficiency. In this class, we will discuss the evolution of storage, giving you a deeper understanding of the principles and practices of server storage and object storage.

Distributed File Storage Service #

Before discussing object storage, let’s first understand the foundation that supports it - distributed file storage service. This is also the data support foundation for internet media resources.

Let’s analyze in detail the functions provided by distributed file storage and what needs to be done for file management in databases. Because the database stores file paths, these records need to be synchronized and updated during migration, archiving, and replica backup.

When the number of files reaches millions or more, in order to respond to file lookup demands with high performance, it is necessary to shard and partition the index information of files, and also provide additional services such as file retrieval, management, access statistics, and hot data migration.

So how do these indexes and storage actually work? Please refer to the diagram below:

From the above diagram, we can see that just managing the indexes of files is already challenging for development, let alone the complex tasks of file storage, transmission, and replica backup. Before using distributed storage services, the common method for implementing static file services was to use Nginx + NFS to mount NAS. However, this method has obvious disadvantages, such as having only one copy of the file and requiring manual periodic backup.

In order to ensure data integrity, improve file service availability, and reduce repetitive development work, most of the industry chooses to use distributed storage systems to centrally manage file distribution and storage. Through distributed storage, it is possible to automatically achieve dynamic scaling, load balancing, file sharding or merging, file archiving, and management acceleration of hot and cold files. This makes managing a large number of files more convenient.

To help you understand how common distributed storage services work, we will analyze it using FastDFS as an example. Please refer to the diagram below:

In fact, the solution for distributed file storage is not perfect.

Taking FastDFS as an example, it has many mandatory rules. For example, the access path for newly saved files is generated by the distributed storage service, and developers need to maintain the mapping between files and actual storage. So every time you want to display a picture, you need to query the database or provide a random hash path to the frontend. In this way, if the file is missing, the frontend will not know what the file actually is.

The file names generated by FastDFS are difficult to understand, and the demonstration path is shown below:

# A demonstration path generated by FastDFS found online
/group1/M00/03/AF/wKi0d1QGA0qANZk5AABawNvHeF0411.png

You might have also noticed that this address is long and difficult to understand, which makes it inconvenient for us to manage files since we are accustomed to classifying and managing various image resources based on hierarchical paths. If this path were /active/img/banner1.jpg, it would be easier to manage.

Although I just mentioned one distributed storage system, other distributed storage systems also have their own problems. Here I would like to remind you that even if you use distributed storage services, our operations and development work is not easy.

Why do I say this? Based on my practical experience, we also need to pay attention to the following five issues:

Disk monitoring: Monitoring the lifespan, capacity, and remaining inodes of disks, as well as fault monitoring warnings and daily maintenance.
File management: Using the distributed storage controller to perform regular work such as cold/hot conversions, regular cleaning, and file archiving.
Ensuring service stability: We also need to focus on the synchronization status of distributed storage replicas and service bandwidth. If the service traffic is high, operations and development also need to handle the caching of hot-access files.
Customization for business requirements: Some slightly personalized requirements, such as attaching business-type tags to files or adding automatic TTL cleaning, may not be directly supported by existing distributed storage services. We may need to read the relevant source code and further improve it to achieve the required functionality.
Hardware updates: The lifespan of hard disks used in servers is generally not long, especially for hard disks used for storage services, which need to be replaced regularly (e.g. every three years).

Object Storage #

Looking back on the past experiences after using distributed storage, I suddenly realized that the hierarchical disk storage structure brought a lot of additional work to the development team. For example, services that mount disks need to provide relative and absolute paths on hundreds of servers and disks, as well as the ability to provide file retrieval, traversal, and file access permissions.

These are actually management functions, and they are not frequently used functions for our external business needs. This design leads to heavy investment in development, which has exceeded the scope that development should originally focus on.

These troubles can be greatly improved by using object storage services. Object storage perfectly solves many problems. This design elegantly abstracts the underlying implementation details, isolating the work of business and operations, making our paths more elegant, simple, and easy to understand, and allowing development to save more time to focus on business.

What are the specific advantages of object storage? I mainly want to emphasize the following three aspects.

Firstly, from the perspective of file indexing. In object storage, the hierarchical directory structure is weakened, or even omitted.

When I say weakened, it means that the hierarchical directory structure can still be retained in object storage. When we need to perform operations based on the hierarchical directory structure, we can use prefix searching to perform prefix searches on these keys to achieve directory lookup and management. Overall, it is very convenient to use, without worrying about slow index searching due to large data volume.

I want to emphasize that object storage does not actually store files according to the specified paths. In fact, the file path is just a key. When we query file objects, we actually perform a hash query, which is much faster than prefix matching queries using strings in databases. Moreover, due to not maintaining a complete tree index, the KV method is faster in both query and storage, and it is also easier to maintain.

Secondly, the management of reading and writing has changed from managing files through disk files to managing file objects through API. The simplified interface in this way will make data reading and writing easier, more flexible, and more convenient to use, without us having to consider too much knowledge about disks.

In addition, object storage also provides file indexing management and mapping, providing more possibilities for managing data and media files. In the past, our files were generally images, audio, and video files, which are generally independent entities for business systems. With object storage, we can manage some data as small files.

However, if we put the data into storage, there will be a large number of small files that need to be managed, and these small files are fragmented, requiring more management strategies and tools. Let’s take a look at how object storage manages small files under this way of thinking.

How Object Storage Manages Small Files #

As I mentioned earlier, in object storage, the actual storage path has been changed to a hash-based storage. Therefore, we can design our object storage paths using a RESTful-like approach, such as:

user\info\9527.json stores the user’s public information
user\info\head\9527.jpg is the avatar of the corresponding user
product\detail\4527.json directly retrieves product information

As you can see, with this design, we don’t need to access the database for every request to retrieve specific object information. The frontend only needs to concatenate the paths to get all the required files. This approach helps us reduce a lot of maintenance costs for caching.

At this point, you may have a question: if this technique is so useful, why wasn’t it popularized before?

This is because in previous implementations, the path requested for access was the actual physical storage path of the file. For Linux, a directory cannot contain too many files. If there are too many files in one directory, it will be difficult to manage. Taking the example above, if we have 3 million users, storing 3 million avatar files in the same directory would cause even a simple ls command to make the server stuck for ten minutes. To avoid similar problems, many servers store these small files by hashing the file name and taking the last four characters of the hash result as the name of a double-layer subdirectory. This ensures that there are not too many files in one directory. However, this approach is inconvenient for frontend development as it requires hash calculation, and it also makes it difficult to search for and manage disk data. Therefore, many people reject this method.

However, even when switching to distributed storage services, the storage of small files still poses a challenge, as both replication synchronization and storage are done on a per-file basis. If the files are very small and thousands of them are uploaded in a second, the replication synchronization process will experience delays due to a large number of allocation and scheduling requests, making it difficult to manage the replication synchronization traffic.

To solve the above problems, object storage and distributed storage services have made optimizations by compressing and storing multiple files in file blocks instead of storing them independently.

The diagram below illustrates file block management:

For example, 100 files are compressed and stored in a 10MB file block for unified management, which is much simpler than managing the files individually. However, it is foreseeable that updating data in this way will be troublesome. Therefore, when updating data in small files, we usually create a new file to update the content. Only when organizing data periodically, we merge the new and old data and write them into a new block while removing the old data.

By the way, it is worth mentioning that the same approach can be used for managing large files. Large files can also be sliced into multiple smaller file blocks for easier management.

How object storage manages large text #

Earlier, we discussed the advantages of object storage in managing small files. Now let’s see how object storage manages large text, which can be more abstractly referred to as using object storage instead of caching.

When do we have a need to manage large text data? A typical scenario is a news portal, especially one with a large amount of news content. These portals often use object storage to directly provide text services.

This design is mainly because large text data over 1KB in size is not suitable for storage in databases or caches. Why is that? Let’s analyze it with the help of the diagram below.

In the above diagram, the left side illustrates the common way of providing data query services through caching, while the right side represents the approach using object storage, which is structurally more convenient and simpler to use and maintain.

Let’s estimate the cost. First, let’s calculate the bandwidth requirements. Assuming our request traffic is 10,000 QPS (queries per second), a caching service for 1KB data would require 1KB x 10,000 QPS, which is approximately equal to 10MB x 8 (conversion from bytes to bits for network bandwidth units) = 80MB/s (network bandwidth unit) of external bandwidth. To leave some room for flexibility, let’s say we need a bandwidth size of around 100MB/s. Additionally, we would need multiple high-performance servers and a large-capacity caching cluster to implement our service.

If we do the math, the cost seems high, right? For systems like news portals that have high read-to-write ratios, not being able to reduce maintenance costs means more resource investment. The common solution is to generate static files from the news content. However, while this approach controls the cost of internet traffic, it increases the costs of operations and development. Are there any better alternatives?

Comparatively, using object storage to maintain the specific pages of resources is a better option.

Let’s analyze the main process in detail: all traffic requests are directed to the object storage service of the cloud provider and are cached and accelerated by the CDN. If the CDN cannot find the requested file, it will make a request to the object storage to find it, and if the object storage cannot find it either, it will request the server to generate it. This greatly reduces the pressure on network traffic. If combined with version control features, it can also allow for reverting to the historical versions of files, thus improving service availability.

Here, I’ll briefly add some implementation details. If there are restrictions on accessing the news content, such as only allowing members to read it, we can set permissions for specific objects so that only a token that will expire shortly can access the content of the file.

Cloud File Transfer #

In addition to the traditional way of downloading data from the server, there is another common method for users to exchange data with each other.

For example, when User A wants to transfer a file to User B, the normal process would involve establishing a TCP connection between the two clients or using a server as an intermediary. However, these methods often result in low transmission efficiency.

By utilizing object storage, file transfer and exchange can be achieved quickly. The main process is as follows: the file transfer service generates a temporary authorization token for the sender, then uploads the file to the object storage. Once the upload is successful, the receiver can obtain the authorization token through the address and perform a multi-threaded download. After the temporary file expires, it will be automatically cleared.

In fact, this method can not only be used for data exchange among users, but also for data exchange and backup storage across different data centers in our business.

Many object storage service providers have already built-in features such as multi-threaded chunked upload and download, GSLB-based CDN route optimization for accelerated upload, etc., in their client SDKs. Using such services can significantly speed up data transmission.

Finally, let’s mention disaster recovery. It can be said that most object storage service providers offer disaster recovery services. We can enable features such as local dual-active backup, global acceleration, disaster scheduling, primary/backup switching, and more for all of our data.

Summary #

In this lesson, we learned about object storage. By comparing it with traditional storage methods, we can easily see the advantages of object storage. Firstly, its simplified design completely abstracts the underlying implementation details, reducing the operational and maintenance costs of disk maintenance.

We can move frequently accessed data from databases to object storage, and reduce costs through CDN and local caching. Applying these classic designs together can save us a lot of time and resources.

I hope this lesson has sparked your interest in exploring object storage. Some commonly used object storage projects in the industry include Alibaba Cloud’s OSS, Tencent’s COS, Huawei Cloud’s OBS, and open-source projects such as Ceph and MinIO.

By understanding these projects, you will have a deeper understanding of the future development trends in the storage industry. In fact, the industry has started to focus on providing large-scale high-speed storage services for major cloud service providers, and this centralized management can save costs more efficiently.

Finally, I have prepared a table for you to examine the characteristics of different storage technologies from multiple perspectives:

As you can see, they have different design directions and concepts. NFS leans towards server services, distributed storage leans towards file management, while object storage leans towards business applications.

Thought-provoking Questions #

In distributed storage, file blocks are used to save and manage small files. When we update the content of a file, how do we refresh the content of that file?

That’s all for today’s lesson. I look forward to discussing with you in the comments section.