020 SIGIR 2018 Paper Review - How to Enhance the Generality of Ranking Models with Adversarial Learning #
Today, we continue with an in-depth analysis of a paper from SIGIR 2018 (International Conference on Research and Development in Information Retrieval). The paper we’ll be discussing today is the best short paper from the conference, titled “Cross Domain Regularization for Neural Ranking Models using Adversarial Learning” (Link to the paper).
It’s important to first briefly introduce the group of authors behind this paper, and it can be said that they are quite an “all-star lineup”.
The first author, Daniel Cohen, is a Ph.D. student in the Computer Science department at the University of Massachusetts Amherst. In 2017, he interned at Microsoft Research in Cambridge. This award-winning paper is a summary of his internship project. Cohen has already published multiple papers in the application of deep learning models in information retrieval.
The second author, Bhaskar Mitra, is a Principal Scientist at Microsoft Research in Cambridge. In recent years, Mitra has been very active in the field of information retrieval and has been a driving force behind the development of deep learning in this field. He has published numerous papers in this field and has also chaired several lectures on the combination of deep learning and information retrieval at various academic conferences in the past few years.
The third author, Katja Hofmann, is also a scientist at Microsoft Research in Cambridge. Hofmann’s main contribution in the field of information retrieval is research on online learning to rank.
The last author of the paper, W. Bruce Croft, is an academic authority in the field of information retrieval and also Cohen’s Ph.D. advisor. He is an ACM Fellow and has received the highest academic honor in the field of information retrieval, the Gerard Salton Award.
Note: The translation retains the original formatting in Markdown.
Main Contributions of the Paper #
This paper mainly involves the combination of two important concepts. The first concept is “Cross Domain” information retrieval, and the second concept is “Adversarial Learning”.
Cross Domain information retrieval mainly refers to the need to search multiple domains. Here, the term “domain” mainly refers to collections of documents that are not very similar or very different. For example, if we want to search for sports news, financial news, etc., “sports” and “finance” are different domains.
The core challenge of cross domain information retrieval is how we can effectively search for different domains. For example, if our ranking algorithm or its characteristics depend on specific domain-specific information, such as searching for sports, which requires the names of athletes, this information will definitely appear very rarely or not at all in another domain. Therefore, it is difficult to achieve true cross domain information retrieval by relying on domain-specific information.
The contribution of this paper is that the authors believe that to retrieve cross-domain information well, it is necessary to train such a ranking model: not easily dominated by a specific domain, and also not biased towards specific domain-specific information.
How to achieve this goal? The authors used a technique called “Adversarial Learning”. This is also an important reason why this paper won the award.
Here, I will briefly introduce the principle of adversarial learning. The initial idea of adversarial learning came from using deep generative models to solve a series of problems in computer vision. The basic mode of adversarial learning is mainly used to generate data, and it is complex data, such as images.
Adversarial learning has two modules, one module is called the generator, and the other module is called the discriminator. The role of the generator is to generate data. The role of the discriminator is to judge whether the generated data is “real data”. The ultimate goal of the generator is to generate data that can deceive the discriminator’s judgment ability. The ultimate goal of the discriminator is to continuously improve its judgment ability to distinguish between real and fake data.
Of course, initially, the data generated by the generator comes from random noise, so the discriminator can easily judge the truth of the data. However, over time, the data generated by the generator will become increasingly close to real data, and the discriminator will also be optimized to quickly judge the truth of the data. Of course, this is a dynamic process, and the states of the discriminator and generator will eventually stabilize.
When the idea of adversarial learning was initially proposed, it was mainly applied in the field of computer vision to generate realistic images. Later, this technology was widely used in various fields of artificial intelligence.
The biggest contribution of this paper is that it uses the idea of adversarial learning to enhance the generality of the learned ranking model, thereby minimizing the learning of domain-specific information that is only useful for one specific domain.
Core Method of the Paper #
Specifically, this article proposes the following method. Firstly, we have two sets of models: one set is used to learn the relationship between query keywords and documents, and the other set is an adversarial learning model. Then, the primary task of these two sets of models is to model the relevant and irrelevant documents more accurately. This is the main objective function of the entire framework.
The new module proposed in this article for the framework is to use adversarial learning to generate relevant and irrelevant documents separately. Specifically, a certain type of document is similar to the image we mentioned earlier, and we hope to use a generator to generate this type of data. Of course, we still need a discriminator to guide the generator’s work.
In this article, the concept of relevance primarily refers to whether a document is from a certain field. In other words, we hope that adversarial learning can help identify whether a document comes from a specific domain. When the adversarial learning model is well-trained, for the model that handles query keywords and document relevancy, we will use a technique called " gradient reversal " to forcefully deviate the model’s inclination to fit a certain domain.
From the perspective of network structure, the model proposed in the article requires both query keywords and documents to undergo transformations such as convolutional layers and extraction layers, and then perform a well-known " Hadamard product “, which is essentially the element-wise multiplication. This combines the extracted hidden features of documents and query keywords. This result is then transformed by a series of dense layers to ultimately predict a relevance label. For the adversarial learning model, the discriminator in adversarial training extracts some intermediate layers from the aforementioned architecture as input and then determines whether this document belongs to a certain domain.
Experimental Results #
The authors conducted experiments on a Yahoo search dataset and two additional datasets to evaluate the proposed model. The experiments aimed to investigate whether a model trained in one domain would perform well in another domain.
An unsurprising result was that training the model on the entire dataset yielded good performance across all domains. However, the paper also demonstrated that by using the method proposed in the paper, it was possible to achieve excellent performance in a specific domain, such as sports documents, by completely excluding all other documents during training. The model still performed well on the test set. The experimental results showed an improvement of over 5% compared to not using adversarial training.
Conclusion #
Today I talked to you about the best short paper at SIGIR this year.
Let’s recap the key points: First, this paper mainly involves two concepts, namely cross-domain information retrieval and adversarial learning. We explained in detail the main contribution of this article, which is to use the idea of adversarial learning to enhance the universality of the learned ranking model. Second, we briefly introduced the core content of the method proposed in the article, training two sets of models to generate relevant and irrelevant documents using adversarial learning. Third, we briefly introduced the experimental results of the paper, showing that adversarial training leads to better performance.
Lastly, I’ll leave you with a thought-provoking question: Besides using adversarial training, can you think of other methods that can effectively learn ranking models for information not belonging to a specific domain?