Twitter is opening a worm hole that will help people go back
in time by building a tweet index that span 8 years. This new index should
allow people to look for any public tweet since 2006, which equates to nearly
half a trillion entries.
The company explains how they were able to build a system
that can serve up queries with an average latency of 100ms in a very long and
detailed manner that could make your nose bleed. For those of you who are into
engineering, the system consists of 4 main parts: a batched data aggregation
and preprocess pipeline, an inverted index builder, Earlybird shards and
Earlybird roots.
At the moment, searching for a certain query will show the
full results of the index in the “All” tab of search results both on the web
and mobile app version of Twitter. The company promises that over time, search
results will appear in the Top section of the results page and in new features
coming to the search index.
