005 Fine Reading of the Second Best Long Paper at EMNLP 2017 #

Every year, EMNLP selects two best long papers, and we have already analyzed the first one, “Men Also Like Shopping: Reducing the Amplification of Gender Bias via Corpus-level Constraints.” Today, I will continue to discuss the second paper.

The second best long paper from EMNLP 2017 is titled “Depression and Self-Harm Risk Assessment in Online Forums.” This paper explores the use of natural language processing techniques to address a social problem. In recent times, the application of machine learning, data science, and other technologies to solve and handle social problems has been gradually becoming an interdisciplinary field of research for both social sciences and machine learning.

Authors’ Information #

The first author, Andrew Yates, is a computer science PhD graduate from Georgetown University in Washington, D.C., United States. He is currently a postdoctoral researcher at Max Planck Institute for Informatics in Germany. During his PhD, Andrew has published multiple papers on deep learning techniques and their application in information retrieval and natural language processing.

The second author, Arman Cohan, is a doctoral student in the computer science department at Georgetown University. Arman has published several papers in the field of information retrieval and natural language processing. In 2016, he interned at Medstar Health in Washington, D.C., and published two papers. In the summer of 2017, he interned at Adobe Research in San Jose, California.

The third author, Nazli Goharian, is also affiliated with the computer science department at Georgetown University, where he serves as a professor of computer science. He was the advisor of the first author and is currently supervising the second author. With a career spanning over 20 years, Nazli has worked in both industry and academia, bringing a wealth of academic and industrial experience. He has published over 20 papers in the field of information retrieval and text analysis.

Main Contributions of the Paper #

Before understanding the main contributions of this paper, let’s first clarify the problem that this paper primarily addresses.

In modern society, people are facing increasing pressures in their lives and work. Research has shown that many individuals may suffer from various mental conditions. In the current era of developed internet technology, online platforms provide a wealth of resources and information for these individuals seeking help, particularly in specialized online support communities or larger online communities such as Twitter or Reddit.

Therefore, studying the behavior of these individuals in various online communities is of great help in designing systems that better meet their needs. For many social researchers, analyzing the mental states of these individuals is crucial in providing long-term assistance to them.

This paper presents a relatively general framework for analyzing the online behavior of individuals with mental disorders. Within this framework, it is possible to accurately analyze whether individuals who post information exhibit self-harm behaviors, as well as to easily analyze which users are likely to have depression.

The entire framework utilizes the increasingly mature deep learning techniques developed in recent years for text analysis, making the application approach valuable for reference and use in other scenarios.

Core Method of the Paper #

Before introducing the method proposed in this article, the authors spend a considerable amount of space introducing the dataset used in the article and how the data labels were generated.

First, the authors found posts related to mental illness from the famous online community Reddit. These posts were selected according to a prepared corpus to ensure a relatively high accuracy in identifying posts related to mental illness. By using sentence patterns from the corpus, such as “I have been diagnosed with depression,” it can be ensured that the posts found largely come from individuals with mental illness.

If a user posts such a post but has fewer than 100 previous posts, this user would not be included in the database. The authors may have made this selection to ensure that the data includes a comprehensive representation of the user’s behavior.

The authors mined all the posts that met the criteria from the Reddit community over a period of ten years from 2006 to 2016, and manually labeled 9,210 users with mental illness struggles. These users serve as positive examples for machine learning.

So how were negative examples found? The authors could certainly use all the users, but this would likely result in a lack of comparability in the study. If there are significant differences between positive and negative examples, it would be difficult to determine whether these differences are caused by mental illness or other factors. Therefore, the authors proposed a method to find the closest negative example for each positive example.

In practice, the authors implemented a stricter approach, where the negative examples must not have posted any content related to mental illness and must be similar to the positive examples in other aspects. Under these conditions, the authors found 107,274 negative examples.

For the users in the dataset, each user on average posted 969 posts, with an average length of over 140 words. This dataset, composed of these users, can be considered a major contribution of this article and is used for analyzing depression.

Regarding self-harm behavior, the authors used data from an online community called ReachOut, collecting 65,024 forum posts, among which 1,227 posts mentioned self-harm. The level of self-harm mentioned in the data was categorized into five levels to represent different levels of urgency.

This paper primarily introduces a text analysis framework based on convolutional neural networks (CNN), used for two tasks: detecting users with depression and detecting the degree of self-harm tendencies. Although these two tasks use different data and have different model details, they both utilize the same framework. Now let’s discuss the main idea of this framework.

First, the authors use the posting information of each user to model that user. The basic idea is to use neural networks to model each post of the user, extract useful information, and then summarize these information pieces to represent the user. With this idea in mind, let’s see how it is implemented.

The words within a certain range in each post go through a convolutional layer to extract features, and then these features go through a max pooling layer to aggregate the information. This step basically applies the standard convolutional layer used in image processing to text information. Each post, after this transformation, becomes a feature vector. With these feature vectors, multiple feature vectors of a user are combined to form an overall representation of the user based on the specific task.

For the task of detecting depression, the authors adopt an “averaging” approach, where the feature vectors of all the posts are directly averaged. For the task of detecting self-harm, the authors adopt a more complex approach, where all the posts are flattened together, and the previous posts are used as negative examples. It is important to note that this approach does not involve averaging but fully flattening the posts, thus expressing the overall features of the user.

After this information extraction process, the subsequent step is to construct a classifier. This step is actually a common practice in deep learning, which is to use multiple fully connected layers to transform the extracted information into target labels.

The method proposed by the authors is clear and understandable in terms of the overall approach. It also provides a basic pattern for text mining using deep learning models, which is to use convolutional networks to extract features and use fully connected layers to learn classifiers.

Experimental Results of the Method #

The authors conducted thorough experiments on the aforementioned dataset and also compared several basic methods, such as directly using text features and then using support vector machines for classification.

In the task of identifying depression, the method proposed in this paper achieved a comprehensive F1 score of 0.51, with a recall rate of 0.45. In contrast, the direct use of support vector machines achieved a precision rate of 0.72, but the recall rate was very low at only 0.29.

In the task of detecting self-harm, the proposed method achieved an accuracy rate of 0.89 and an F1 score of 0.61, both significantly higher than other methods.

It can be said that based on these notable numbers, the method presented in this paper has achieved good results.

Summary #

Today I talked to you about the second best long paper at EMNLP 2017. This paper introduces an application of deep learning models for analyzing textual information in online forums, specifically in identifying users with mental disorders.

Let’s review the key points: First, I briefly introduced the authors of this paper. Second, this paper leverages natural language processing techniques to address a social problem, and the dataset constructed for this research is valuable. Third, the paper applies the standard convolutional layers commonly used in image processing to textual information, proposing a convolutional neural network-based framework for text analysis, which achieves good results in identifying depression and detecting self-harm tendencies.

Lastly, I’ll leave you with a question to ponder. If using convolutional layers for image information makes sense,why is the same operation effective for text information? What is the physical significance of convolutional operations on text?

Further reading: Depression and Self-Harm Risk Assessment in Online Forums