Researchers at MIT say they’ve found an algorithm that can tell what’s going to be trending on Twitter hours before it happens, opening the door to modeling other kinds of data. Twitter’s trending topics — the ones shown on its home page — are selected automatically, based on total number of tweets and a jump in how often a topic is mentioned. But Associate Professor Devavrat Shah and student Stanislav Nikolov say that by building a statistical model based on a sample set of 400 topics, they can tell with 95 percent accuracy whether or not a topic will end up trending. In tests, topics can be predicted an average of an hour and a half before they trend, with some predicted up to four or five hours beforehand.
Reverse-engineered Twitter algorithm can predict trending topics before they happen
is a senior tech and policy editor focused on online platforms and free expression. Adi has covered virtual and augmented reality, the history of computing, and more for The Verge since 2011.
95 percent accuracy, an average of an hour and a half beforehand
While this development could be used to optimize advertising linked to topics on Twitter, Shah also thinks it could also be used to predict other things that update at regular intervals. Unlike machine learning that looks for different parameters that might factor into a trend, this system essentially recognizes statistical patterns. If the pattern of a new topic resembles ones found in its library of past data, the topic is seen as more likely to trend.
As long as there’s enough computing power available and a consistent relationship between past and present data, this method should produce solid results, even if researchers don’t know what factors are in play. Of course, since the team is essentially reverse-engineering an algorithm written by Twitter, it’s likely much easier to find trends in it than in less predictable real-world data. More details will be presented at the Interdisciplinary Workshop on Information and Decision in Social Networks in November.
Follow topics and authors from this story to see more like this in your personalized homepage feed and to receive email updates.
