39 Add Meal Multiple Return Values No Need to Fear Merger Combiners to the Rescue

39 Add Meal Multiple Return Values No Need to Fear Merger Combiners to the Rescue #

Hello, I’m Yang Sizheng. Today I’m going to share with you the topic of the Merger.

In the previous lesson, when we analyzed the specific implementation of MergeableClusterInvoker, we explained the following: In MergeableClusterInvoker, the merger parameter value in the URL is read. If the merger parameter starts with “.”, it means that the content after “.” is a method name. This method name is a method in the return type of the remote target method. After MergeableClusterInvoker obtains the result objects returned by all Invokers, it will iterate through each result and invoke the method specified by the merger parameter to merge these result values.

In addition to the above method of specifying a merger method name for merging, Dubbo also provides many default Merger implementations internally. This is the content we will analyze in detail in this lesson. In this lesson, we will introduce the MergerFactory, Merger interface, and the Merger implementations for common data types in Java.

MergerFactory #

When MergeableClusterInvoker uses the default Merger implementation, it will choose the appropriate Merger implementation based on the MergerFactory and the return type of the service interface.

In MergerFactory, a ConcurrentHashMap collection is maintained (i.e., the MERGER_CACHE field) to cache the mapping relationship between the return type of the service interface and the Merger instance.

The MergerFactory.getMerger() method will find the corresponding Merger implementation from the MERGER_CACHE cache based on the passed-in returnType type. Let’s take a look at the specific implementation of this method:

public static <T> Merger<T> getMerger(Class<T> returnType) {

    if (returnType == null) { // If returnType is null, throw an exception

        throw new IllegalArgumentException("returnType is null");

    }

    Merger result;

    if (returnType.isArray()) { // If returnType is an array type

        // Get the type of the elements in the array

        Class type = returnType.getComponentType();

        // Get the Merger implementation corresponding to the element type

        result = MERGER_CACHE.get(type);

        if (result == null) {

            loadMergers();

            result = MERGER_CACHE.get(type);

        }

        // If Dubbo does not provide a Merger implementation for the element type, return ArrayMerger

        if (result == null && !type.isPrimitive()) {

            result = ArrayMerger.INSTANCE;

        }

    } else {

        // If returnType is not an array type, directly find the corresponding Merger instance from the MERGER_CACHE cache

        result = MERGER_CACHE.get(returnType);

        if (result == null) {

            loadMergers();

            result = MERGER_CACHE.get(returnType);

        }

    }

    return result;

}

The loadMergers() method loads the names of all extension implementations of the Merger interface through Dubbo SPI, and fills them into the MERGER_CACHE collection. The specific implementation is as follows:

static void loadMergers() {

    // Get the names of all extension implementations of the Merger interface

    Set<String> names = ExtensionLoader.getExtensionLoader(Merger.class)

            .getSupportedExtensions();

    for (String name : names) { // Iterate through all Merger extension implementations

        Merger m = ExtensionLoader.getExtensionLoader(Merger.class).getExtension(name);

        // Record the mapping relationship between the Merger instance and the corresponding returnType in the MERGER_CACHE collection

        MERGER_CACHE.putIfAbsent(ReflectUtils.getGenericClass(m.getClass()), m);

    }

}

ArrayMerger #

Dubbo provides Merger implementations for handling different types of return values, including Merger implementations for basic type arrays such as boolean[], byte[], char[], double[], float[], int[], long[], short[], and Merger implementations for collection classes such as List, Set, Map, etc. The inheritance relationship is shown in the following diagram:

Lark20201208-135542.png

Merger inheritance diagram

Let’s first take a look at the implementation of ArrayMerger: When the return value of the service interface is an array, ArrayMerger is used to merge multiple arrays into one array, i.e., flattening the two-dimensional array into a one-dimensional array. The specific implementation of the ArrayMerger.merge() method is as follows:

public Object[] merge(Object[]... items) {

    if (ArrayUtils.isEmpty(items)) {

        // If the result set passed in is empty, return an empty array

        return new Object[0];

    }

    int i = 0;

    // Find the first non-null result

    while (i < items.length && items[i] == null) {

        i++;

    }

    // If all results in the items array are null, return an empty array

    if (i == items.length) {

        return new Object[0];

    }

    Class<?> type = items[i].getClass().getComponentType();

    int totalLen = 0;

    for (; i < items.length; i++) {

        if (items[i] == null) { // Ignore null results

            continue;

        }

        Class<?> itemType = items[i].getClass().getComponentType();

        if (itemType != type) { // Ensure the types are the same

            throw new IllegalArgumentException("Arguments' types are different");

        }

        totalLen += items[i].length;

    }

    if (totalLen == 0) { // Determine the length of the final array

        return new Object[0];

    }

    Object result = Array.newInstance(type, totalLen);

    int index = 0;

    // Traverse all result arrays and add each element from the items two-dimensional array to the result to form a one-dimensional array

    for (Object[] array : items) {

        if (array != null) {

            for (int j = 0; j < array.length; j++) {

                Array.set(result, index++, array[j]);

            }

        }

    }

    return (Object[]) result;

}

The implementation of other basic data type array Merger implementations is very similar to ArrayMerger. They all flatten the corresponding type’s two-dimensional array into a one-dimensional array. Here’s an example of IntArrayMerger:

public int[] merge(int[]... items) {

    if (ArrayUtils.isEmpty(items)) {

        // Check that the multiple int[] passed in is not empty

        return new int[0];
```java
}

// Use Stream API directly to flatten multiple int[] arrays into one int[] array

return Arrays.stream(items)
        .filter(Objects::nonNull)
        .flatMapToInt(Arrays::stream)
        .toArray();
}

The remaining implementations of Merger for other basic types, such as FloatArrayMerger, IntArrayMerger, LongArrayMerger, BooleanArrayMerger, ByteArrayMerger, CharArrayMerger, and DoubleArrayMerger, are not repeated here. If you are interested, you can refer to the source code for learning on this GitHub repository.

MapMerger #

SetMerger, ListMerger, and MapMerger are implementations of Merger specifically for Set, List, and Map return types. They merge multiple Set (or List, Map) collections into a single Set (or List, Map) collection. The core implementation of MapMerger is as follows:

public Map<?, ?> merge(Map<?, ?>... items) {
    if (ArrayUtils.isEmpty(items)) {
        // Return an empty Map when the result set is empty
        return Collections.emptyMap();
    }

    // Add all key-value pairs from all Map collections in items to the result Map
    Map<Object, Object> result = new HashMap<>();
    Stream.of(items).filter(Objects::nonNull).forEach(result::putAll);
    return result;
}

Now let’s take a look at the core implementations of SetMerger and ListMerger:

public Set<Object> merge(Set<?>... items) {
    if (ArrayUtils.isEmpty(items)) {
        // Return an empty Set collection when the result set is empty
        return Collections.emptySet();
    }

    // Create a new HashSet collection and add all passed-in Set collections to it
    Set<Object> result = new HashSet<>();
    Stream.of(items).filter(Objects::nonNull).forEach(result::addAll);
    return result;
}

public List<Object> merge(List<?>... items) {
    if (ArrayUtils.isEmpty(items)) {
        // Return an empty List when the result set is empty
        return Collections.emptyList();
    }

    // Flatten all the passed-in List collections into one List collection using the Stream API and return it
    return Stream.of(items).filter(Objects::nonNull)
            .flatMap(Collection::stream)
            .collect(Collectors.toList());
}

Custom Merger Implementation #

After introducing the built-in Merger implementations in Dubbo, let’s try to write our own custom Merger implementation. We will modify the Provider and Consumer in the dubbo-demo-xml module as an example.

First, in the dubbo-demo-xml-provider module, we publish two services that belong to groupA and groupB respectively. The dubbo-provider.xml configuration is as follows:

<beans xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:dubbo="http://dubbo.apache.org/schema/dubbo"
       xmlns="http://www.springframework.org/schema/beans"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-4.3.xsd
       http://dubbo.apache.org/schema/dubbo http://dubbo.apache.org/schema/dubbo/dubbo.xsd">

    <dubbo:application metadata-type="remote" name="demo-provider"/>
    <dubbo:metadata-report address="zookeeper://127.0.0.1:2181"/>
    <dubbo:registry address="zookeeper://127.0.0.1:2181"/>
    <dubbo:protocol name="dubbo"/>

    <!-- Configure two Spring Beans -->
    <bean id="demoService" class="org.apache.dubbo.demo.provider.DemoServiceImpl"/>
    <bean id="demoServiceB" class="org.apache.dubbo.demo.provider.DemoServiceImpl"/>

    <!-- Publish demoService and demoServiceB as services, belonging to groupA and groupB respectively -->
    <dubbo:service interface="org.apache.dubbo.demo.DemoService" ref="demoService" group="groupA"/>
    <dubbo:service interface="org.apache.dubbo.demo.DemoService" ref="demoServiceB" group="groupB"/>
</beans>

Next, in the dubbo-demo-xml-consumer module, we reference the services. The content of the dubbo-consumer.xml configuration file is as follows:

<beans xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:dubbo="http://dubbo.apache.org/schema/dubbo"
       xmlns="http://www.springframework.org/schema/beans"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-4.3.xsd
       http://dubbo.apache.org/schema/dubbo http://dubbo.apache.org/schema/dubbo/dubbo.xsd">

    <dubbo:application name="demo-consumer"/>
    <dubbo:registry address="zookeeper://127.0.0.1:2181"/>

    <!-- Reference DemoService, specifying group as * to allow referencing Providers from any group, and enable merger to perform result merging -->
    <dubbo:reference id="demoService" check="false" interface="org.apache.dubbo.demo.DemoService" group="*" merger="true"/>
</beans>

Then, in the /resources/META-INF/dubbo directory of the dubbo-demo-xml-consumer module, add a Dubbo SPI configuration file named org.apache.dubbo.rpc.cluster.Merger, with the following content:

String=org.apache.dubbo.demo.consumer.StringMerger

The StringMerger implements the Merger interface mentioned earlier. It concatenates the String results returned from multiple Provider nodes. The specific implementation is as follows:

public class StringMerger implements Merger<String> {
    @Override
    public String merge(String... items) {
        if (ArrayUtils.isEmpty(items)) { // Check for empty return values
            return "";
        }

        StringBuilder result = new StringBuilder();
        for (String item : items) { // Concatenate the return values from multiple Providers with a vertical line
            result.append(item).append("|");
        }

        return result.toString();
    }
}

Finally, we start ZooKeeper, the dubbo-demo-xml-provider module, and the dubbo-demo-xml-consumer module in order. In the console, we will see the following output:

result: Hello world, response from provider: 172.17.108.179:20880|Hello world, response from provider: 172.17.108.179:20880|

Summary #

In this lesson, we focused on Merger, the key component in MergeableCluster.

  • First, we introduced the core functionality of the MergerFactory, which selects the corresponding Merger implementation to merge the results based on the return type of remote method invocations.
  • Then, we analyzed the built-in Merger implementations in Dubbo, which cover the merging of one-dimensional arrays of various basic types in Java, such as IntArrayMerger, LongArrayMerger, and so on. They all flatten multiple arrays of a specific type into one array of the same type.
  • In addition to these Merger implementations for basic type arrays, Dubbo also provides Merger implementations for collection types like List, Set, and Map, which organize the elements from multiple collections into one collection of the same type.
  • Finally, we took StringMerger as an example to demonstrate how to customize a Merger implementation.