Trace the logic behind X's recommendation feed selection and ordering
Readers who have wondered why specific posts appear in the 'Recommended' tab can now read the internal logic directly. This repository exposes the core algorithm code that builds X's recommendation feed, showing how posts become candidates, how they are scored, and which ones are excluded. It allows you to see how complex machine learning models and filtering rules combine to create the final feed and to trace the basis for the algorithm's decisions.
Recent updates: Election filter removal and model improvements
As elections have concluded, posts from accounts previously subject to that filter are no longer automatically excluded. On the 13th of the same month, the search model was trained to better identify posts that users read to the end. These changes are moving in the direction of improving feed quality and relevance.
Two paths for feed composition: Follows and Recommendations
The recommendation feed is created by combining recent posts from accounts the user follows with posts discovered from accounts they do not follow. Posts from followed accounts are retrieved from recent posts stored in memory, while posts from unfollowed accounts are found through machine-learning-based search and similarity clustering. Posts from these two sources are scored and ordered together by the same model. It takes the user's recent actions (likes, comments, etc.) as input and predicts the probability that the user will perform a specific action for each post.
Score calculation and the actual meaning of weights
The final score is calculated as the sum of the predicted probabilities for each action multiplied by their respective weights. Positive actions (likes, shares, etc.) are assigned positive weights, while negative actions (reports, blocks, etc.) are assigned negative weights. The key point here is that weights do not directly adjust the original engagement counts, but rather adjust the predicted probability values. For example, interpreting a higher weight for reports than for likes as 'one report offsets 468 likes' is a misunderstanding. Weights are multipliers applied to predicted probabilities based on the user's own action patterns, and code comments have been added to clarify this understanding.
Visibility filtering and labeling system
Whether a post can be displayed in the feed is determined by a 'visibility filtering' system, separate from ranking. This system classifies posts as 'allow', 'show notification', 'show in middle screen', or 'exclude' by considering post labels, user block/mute settings, and account status. Labels are continuously generated by models that classify spam, adult content, and violence, as well as systems that evaluate account trustworthiness.
What to know before reading this repository
This repository is written in Rust and features a structure combining JAX-based machine learning models with a Rust serving layer. Not all components run within a single request path; content understanding and labeling are performed continuously outside the request path. Some rules are not included in the repository to prevent gaming (evasion). Additionally, this repository may not reflect all production environment settings exactly, so you should check the relationship between defaults and production execution in the experiments and settings sections. Because the code structure is complex with high interdependencies, it is recommended to grasp the overall architecture diagram before reading individual components.