Collect public data from major Chinese social platforms in one go
When you need to automatically gather posts found by a specific keyword and the comments beneath them, this tool is useful. It uses browser automation to maintain logged-in states on each platform, removing the need to solve complex encryption algorithms directly. The collected data is saved in various formats, ready for immediate use in analysis or archiving.
Logging in and scraping data via browser automation
The core of this tool is the use of the Playwright browser automation framework. After a user logs in by scanning a QR code, the tool takes over that browser session to collect data. This eliminates the need to manually generate complex signature parameters in JavaScript required by each platform. It also supports Chrome's remote debugging mode (CDP), allowing the reuse of cookies and extension states from an existing browser, which helps lower the risk of platform security detection.
Supported platforms and data collection scope
Currently, the tool supports Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Baidu Tieba, and Zhihu. It provides features for keyword search, specifying particular post IDs, collecting secondary comments, and gathering data from specific creator pages on each platform. Collected data can be saved in formats such as CSV, JSON, Excel, SQLite, and MySQL, and image or video files can be downloaded alongside the data if needed. Specifically for Bilibili, if ffmpeg is installed, the tool can merge and save videos in the highest quality.
Preparation before starting to use the tool
To run the tool, a Python environment and Node.js (version 16 or higher) must be installed. The package manager uv is recommended, and dependencies can be easily installed using the uv sync command. Browser drivers need to be installed separately only when using the standard Playwright mode; for CDP mode, it is sufficient to configure an existing Chrome browser (version 144 or higher). You can directly modify the configuration file config/base_config.py to adjust settings such as the target platform, login method, and whether to collect comments.
Clear boundaries on legal responsibility and usage purpose
The README strongly warns that this tool should be used only for learning and research purposes. Commercial use, large-scale crawling, and actions that infringe on others' legitimate rights and interests are strictly prohibited, and all legal responsibility for such actions lies with the user. The developer explicitly states that they bear no responsibility for data loss, equipment damage, or legal lawsuits that may arise from using the tool. Therefore, it is safe to check the laws of the relevant country and the terms of service of each platform before using the tool, and to limit its use strictly to personal learning.
