Summary

Conventional social media such as Facebook and Twitter with user-generated content are an interesting way to analyze information about individuals and organizations. The APIs of these platforms are useful to gather and analyse a lot of this data emanating from these platforms. on the other hand, online forums without structured freely available APIs provide a great sources of topical discussions. The main difference is mostly the anonymous nature of the forums, which encourages individuals to have long and deep discussions on various topics (technology, politics, culture, and so on), unlike on other social media where one is often identified through pseudonyms and actual identifiers. These millions of rich conversations on Internet forums have become extremely interesting for companies to learn about the trends of their consumers.

To harness conversational data, we explored the techniques to create crawlers using the Scrappy framework. After extraction and storage of this data, we...