MASc seminar - Rania Ibrahim

Candidate

Rania Ibrahim

Title

Scalable Topic Detection Approaches from Twitter Streams

Supervisor

Fakhreddine Karray

Abstract

Real time topic detection in Twitter streams is an important task that helps discovering natural disasters in a real time from users’ posts and helps political parties and companies understand users’ opinions and needs. In 2014 the number of active users on Twitter is reported to be more than 288 million users who are posting around 500 million tweets daily. Therefore, detecting topics from Twitter streams in a real time needs scalable and efficient techniques. In this work, we scale an Exemplar-based technique that detects topics from Twitter streams, where each of the detected topics is represented by one tweet (i.e, exemplar). Using exemplar tweets to represent the detected topics, makes these topics easier to interpret as opposed to representing them by uncorrelated terms as in other topic detection algorithms. The approach is implemented using Apache Giraph and is being extended to efficiently support sliding windows. Experimental results show that Giraph Exemplar-based approach achieves the best topic recall and term precision against other distributed related work, while maintaining a good term recall and running time.

Moreover, another clustering technique called Local Variance-based Clustering (LVC) is proposed in this work for detecting topics from Twitter streams. Local Variance-based Clustering (LVC) defines the data points densities based on their similarities. The proposed local variance measure is calculated based on the variance of the data points similarity histogram and is shown to well distinguish between core, border, connecting and outliers points. Experimental results show that LVC can detect topics from Twitter with higher topic recall by 15% and higher term precision by 3% over DBSCAN.