28 Security Always Consider Anti Scraping Limitation Measures When Dealing With Money

28 Security Always Consider Anti-Scraping Limitation Measures When Dealing with Money #

Today, I want to share with you the topic of ensuring safety when dealing with code that involves money. It is crucial to consider anti-fraud measures, limitations, and duplicate prevention in order to provide a solid safety net.

There are three main types of code that involve money.

First, there is code that utilizes third-party services that require payment. If the code lacks proper authorization and usage control, it can be exploited and result in excessive calls, leading to financial losses for the company. Some third-party services may settle payments after they are used, so if any issues go unnoticed, the company could receive a substantial bill during the next settlement period.

Second, there is code that involves the distribution of virtual assets, such as points or coupons. Although these assets do not directly correspond to cash, they can generally be exchanged within platforms for tangible assets of real value. For example, coupons can be used during checkout, and points can be exchanged for goods in a points store. Therefore, virtual assets can be seen as money with certain value. However, since they do not directly involve actual money or external financial channels, there is a risk of vulnerabilities due to arbitrary distribution.

Third, there is code that involves the inflow and outflow of actual money. For instance, deducting funds from users’ accounts. If abnormal multiple deductions occur, it can result in user complaints, loss of users, or even regulatory intervention demanding business suspension and rectification, which can greatly impact the operations. Another example is the payment function for providing users with cashback. If a loophole causes duplicate payments, it may only have a minor effect on businesses involving B2B transactions, but duplicate payments involving C2C users may never be recovered.

Recently, Pinduoduo experienced an incident where a large quantity of 100 RMB coupons without any spending threshold were exploited overnight, highlighting issues with limitations and anti-fraud measures.

Today, through these three examples, we will demonstrate how to establish a solid safety net at the code level.

Consideration of Anti-Brushing for Open Platform Resources #

Let me share with you a case where I encountered SMS service brushing, and talk about anti-brushing.

Once, when the monthly bill for SMS services was generated, I discovered that the monthly SMS expenses, which used to be a few thousand yuan, suddenly became tens of thousands of yuan. After checking the database records, I found that the number of daily SMS verifications, which used to be a few thousand, suddenly increased to tens of thousands from a certain day, but the number of registered users did not increase significantly. Obviously, the SMS interface was being brushed.

We know that the SMS verification code service is an open service triggered by users. Since it is a registration verification code, it does not require login to use. If our SMS sending interface, without any anti-brushing protection, directly calls a third-party SMS channel, it is equivalent to “exposing oneself”, which can easily be exploited by SMS bombing platforms:

@GetMapping("wrong")
public void wrong() {
    sendSMSCaptcha("13600000000");
}

private void sendSMSCaptcha(String mobile) {
    // Call the SMS channel
}

For open interfaces like SMS verification codes, there needs to be anti-brushing logic within the program logic. A good anti-brushing logic should have no impact on regular users, only users with suspicious usage patterns should be affected. For SMS verification codes, there are four feasible ways to prevent brushing.

The first method is to only allow sending verification codes with specific request headers.

In other words, we can determine whether the request is initiated by an App by analyzing certain additional parameters passed from the webpage or App client to the server through the request headers. This method can be considered as a basic defense against scraping for programs that can only obtain the URL but not the additional request headers required to send the SMS.

The second method is to only allow sending verification codes after the user has visited the registration page.

For regular users, whether they register through an App or an H5 page, they must first enter the registration page to see the send verification code button and then click to send it. We can make a fixed pre-interface request when the page or interface is opened, to open a window that allows sending verification codes for this device, prior to sending the verification code. This method can defend against direct bypassing of the fixed process by sending verification code requests through the interface, without disturbing regular users.

The third method is to control the number of times and the frequency of sending verification codes to the same phone number.

Unless the SMS cannot be received, users are unlikely to request a verification code and not complete the registration process, and then request it again. Therefore, we can limit the maximum number of requests for the same phone number per day. Since it takes time for the verification code to arrive, very short sending intervals are meaningless, so we can also control the shortest interval for sending. For example, we can limit the same phone number to only send 10 verification codes per day, with a minimum sending interval of 1 minute.

The fourth method is to add a pre-captcha.

SMS bombing platforms usually collect many free SMS interfaces. An interface only sends one SMS to one user, so controlling the number of times and the interval of sending to the same phone number is not effective enough. In this case, we can consider adding a slightly intrusive pre-captcha as the last resort, even if it slightly affects the user experience.

In addition to captchas, we can use other more user-friendly methods of human-machine verification (such as sliding, clicking captchas, etc.), or even introduce more modern and seamless captcha solutions (such as judging whether the input rhythm of the phone number by the user indicates a human or a bot) to improve the user experience.

Furthermore, we can consider triggering human-machine detection only when anomalies are detected. For example, human-machine detection can be triggered when a large number of verification codes are sent from the same remote IP address within a short period of time.

In conclusion, we need to ensure that only regular users can access open platform resources through the normal process, and the usage of resources should be within a reasonable range according to business requirements. In addition, it is necessary to properly monitor the real-time volume of SMS sending and timely raise an alarm when there is a significant increase in sending volume.

Next, let’s take a look at the issue of limiting the usage.

Virtual assets cannot be created out of thin air for unlimited use #

Although virtual assets are produced and controlled by the platform itself, there is a possibility of immediate cashing in if they can be immediately used. For example, if there is a platform bug that allows users to claim high-value coupons and immediately use them.

From the perspective of merchants, this may just be an order paid by a user, and they may not be aware that the user used a coupon from the platform. At the same time, because the platform and merchants settle accounts afterwards, the merchants will arrange for delivery immediately. Once the delivery is made, it is basically irreversible, causing a large amount of financial loss overnight.

Let’s simulate an example of a coupon being abused from the code level.

Assume there is a CouponCenter class responsible for generating and distributing coupons. The following is an incorrect approach where unlimited coupons can be created out of thin air whenever the caller needs it:

@Slf4j
public class CouponCenter {

    // Used to count how many coupons have been sent
    AtomicInteger totalSent = new AtomicInteger(0);

    public void sendCoupon(Coupon coupon) {
        if (coupon != null)
            totalSent.incrementAndGet();
    }

    public int getTotalSentCoupon() {
        return totalSent.get();
    }

    // No restrictions, generate as many coupons as requested
    public Coupon generateCouponWrong(long userId, BigDecimal amount) {
        return new Coupon(userId, amount);
    }
}

In this way, by using the CouponCenter’s generateCouponWrong method, users can generate as many coupons as they want:

@GetMapping("wrong")
public int wrong() {
    CouponCenter couponCenter = new CouponCenter();

    // Send 10,000 coupons
    IntStream.rangeClosed(1, 10000).forEach(i -> {
        Coupon coupon = couponCenter.generateCouponWrong(1L, new BigDecimal("100"));
        couponCenter.sendCoupon(coupon);
    });

    return couponCenter.getTotalSentCoupon();
}

The better approach is to treat the coupon as a resource, not something that can be created out of thin air, and that it requires a prior application. The reasons being that:

  • If virtual assets can eventually be converted into real money through discounts, the amount of coupons that can be issued should depend on the operation and financial accounting, and there should be a planned and limited amount. Care should be taken with regard to the mentioned unrestricted coupons. The use of coupons with restrictions will at least bring about a large amount of real consumption, while orders placed with unrestricted coupons may not involve any payment from the user.
  • Even if virtual assets have no monetary value, a large influx of non-standard virtual assets into the market can disrupt the economic system of virtual assets and lead to a rapid devaluation of virtual currencies. Quantity control is the key to value.

The application for assets needs to be justified and may even require a process, allowing for tracing of which activities require them and who made the requests, and the program will distribute them based on batch requests.

Next, let’s improve the program based on this line of thinking.

Firstly, define a CouponBatch class. To generate coupons, a batch of coupons must first be applied for from the operation. The batch will contain a fixed number of coupons and information such as the reason for the application:

// Coupon batch
@Data
public class CouponBatch {
    private long id;
    private AtomicInteger totalCount;
    private AtomicInteger remainCount;
    private BigDecimal amount;
    private String reason;
}

When there is a need to distribute coupons, we first apply for a batch and then distribute the coupons through the batch:

@GetMapping("right")
public int right() {
    CouponCenter couponCenter = new CouponCenter();

    // Apply for a batch
    CouponBatch couponBatch = couponCenter.generateCouponBatch();

    IntStream.rangeClosed(1, 10000).forEach(i -> {
        Coupon coupon = couponCenter.generateCouponRight(1L, couponBatch);
        // Distribute the coupon
        couponCenter.sendCoupon(coupon);
    });

    return couponCenter.getTotalSentCoupon();
}

As seen, in the generateCouponBatch method, we applied for a batch and set this batch to contain 100 coupons. In the generateCouponRight method, each coupon distributed will decrement the remaining count of coupons from the batch. If there are no more coupons, the method will return null:

public Coupon generateCouponRight(long userId, CouponBatch couponBatch) {
    if (couponBatch.getRemainCount().decrementAndGet() >= 0) {
        return new Coupon(userId, couponBatch.getAmount());
    } else {
        log.info("Insufficient remaining coupons in batch {}", couponBatch.getId());
        return null;
    }
}

public CouponBatch generateCouponBatch() {
    CouponBatch couponBatch = new CouponBatch();
    couponBatch.setAmount(new BigDecimal("100"));
    couponBatch.setId(1L);
    couponBatch.setTotalCount(new AtomicInteger(100));
    couponBatch.setRemainCount(couponBatch.getTotalCount());
    couponBatch.setReason("XXX activity");
    return couponBatch;
}

After this improvement, a batch can only distribute a maximum of 100 coupons.

img

Because this is just a demo, we simply made a new Coupon object. In real production-level code, there would be a certain amount of coupon records inserted into the database based on CouponBatch, and each coupon would have a unique ID for tracking and cancellation.

Finally, let’s consider preventing duplication.

Money in and out must be tied to orders and achieve idempotence #

When it comes to money transactions, there are two important points to consider.

First, any financial operation needs to generate a business order with corresponding attributes on the platform side. The order can be for coupon issuance, cashback, or loan, but it must be created before any financial operation is performed. At the same time, the generation of the order needs to have business attributes. Business attributes mean that the order is not created out of thin air, otherwise it would have no control significance. For example, a cashback order must be associated with the original purchase order; likewise, a loan order must be associated with the same loan contract.

Second, it is crucial to prevent duplicate operations and achieve idempotent processing, which must be implemented across the entire process. Here, “across the entire process” means that there should be a consistent business order number from the beginning to the end to achieve final payment duplication prevention.

Regarding these two points, you can refer to the following code examples:

     
// Incorrect: Using a UUID as the order number

@GetMapping("wrong")
public void wrong(@RequestParam("orderId") String orderId) {
    PayChannel.pay(UUID.randomUUID().toString(), "123", new BigDecimal("100"));
}

// Correct: Using the same business order number

@GetMapping("right")
public void right(@RequestParam("orderId") String orderId) {
    PayChannel.pay(orderId, "123", new BigDecimal("100"));
}

// Third-party payment channel

public class PayChannel {
    public static void pay(String orderId, String account, BigDecimal amount) {
        ...
    }
}

For payment operations, we usually call interfaces of third-party payment companies or banks for processing. In general, these interfaces have the concept of a merchant order number, and it is not possible to perform duplicate financial processing for the same order number. Therefore, the interfaces of third-party companies can achieve idempotent processing with a unique order number.

However, a common mistake made when implementing financial operations in business systems is not using a single order number as the merchant order number throughout the entire process, passing it to the third-party payment interface. The reason for this issue is that larger internet companies usually have a separate payment department. The payment department may perform aggregated operations for payments and internally manage a payment order number, which is then used to interact with the third-party payment interface. As a result, although there is only one purchase order, there are multiple payment order numbers, leading to multiple payments for the same purchase order.

Even if duplicate deductions occur during payments, we can refund the user. However, once duplicate payments are made to the user, it is difficult to recover the money, so we need to be even more cautious.

This is the significance of maintaining consistency throughout the entire process. From the beginning, a business order needs to be generated, and the same business order number should be used consistently throughout the payment process to prevent duplicate financial operations.

Key Recap #

Today, I started with the topic of security and shared with you the three important aspects that need to be done for money-related businesses: anti-fraud, quantity limitation, and duplicate prevention.

Firstly, when using open and user-facing platform resources, we need to consider anti-fraud measures. This mainly includes methods such as normal usage process identification, human-machine recognition, individual and global quantity limitations.

Secondly, virtual assets cannot be generated out of thin air. There must be a distribution plan and application batches, which are then used to produce the assets. This way, we can achieve limitations, auditing, and traceability.

Thirdly, we need to be extra careful with operations involving real money and implement duplicate prevention. We should not manipulate user accounts arbitrarily. Each operation should be based on a real order and full-link idempotent control can be achieved using the business order number.

If the program logic involves valuable resources or real money, we must have a sense of awe. After the program is launched, humans have resting time, but the program keeps running. If there is a security vulnerability, it could potentially explode overnight, resulting in significant financial losses due to widespread exploitation.

In addition to implementing anti-fraud, quantity limitation, and duplicate prevention controls in the process, we also need to monitor and set up alarms for important data such as the API call volume, virtual asset usage, transaction volume, and transaction amount on third-party platforms. This way, even if problems arise, they can be discovered immediately.

I have put the code we used today on GitHub, you can click this link to view it.

Reflection and Discussion #

Both anti-duplication and anti-brushing are proactive measures. If our system is being attacked or exploited, do you have any way to detect the problem in a timely manner?

Regular reconciliation is usually conducted when using any third-party resources. If we find that the number of calls recorded in our system is lower than the usage recorded by the other system during reconciliation, what do you think is the general problem?

Regarding security safeguards, do you have any insights? I am Zhu Ye. Feel free to leave a comment in the comment section to share your thoughts with me. You are also welcome to share today’s content with your friends or colleagues for further discussion.