Research on Sentiment Analysis of Microblog Text based on Web Crawler

Abstract

Sentiment analysis of Microblog text has become an important research direction in the field of natural language processing (NLP), thanks to the rapid development of social media platforms such as Microblog. However, the unique characteristics of Microblog data—such as short text length, informal language, the inclusion of emoticons, and dynamic changes in sentiment over time—pose significant challenges to traditional sentiment analysis methods. This paper proposes a comprehensive Microblog sentiment analysis framework that combines improved web scraping techniques, advanced data preprocessing methods, and deep learning-based sentiment classification models to enhance the accuracy and robustness of the analysis.

Microblog data collection: We explore the challenges of Microblog web scraping, where anti-scraping mechanisms limit large-scale data collection. We propose an optimised scraping strategy to efficiently collect text data and overcome access restrictions.

Microblog data preprocessing: Due to noise in raw Microblog data, we employ regular expressions, Jieba word segmentation, and stopword removal techniques to extract clean, sentiment-relevant text for analysis.

Deep learning-based sentiment analysis: We introduced a CNN-BiLSTM model with an attention mechanism, which can capture local features through CNN, process long-term dependencies through BiLSTM, and assign weights to key sentiment clues through the attention mechanism. Experiments on real Microblog datasets show that our model outperforms baseline methods in classification accuracy.

This study not only advances innovations in sentiment analysis methods but also provides practical insights for applications such as public opinion monitoring and social media analysis. Future research could explore multimodal sentiment analysis (integrating text and emojis) and domain adaptability across different Microblog scenarios.

-