Getting the data

Forums do not provide programmatic interfaces (APIs) to capture data. However, you can connect to the website as a user to see all the conversations and collect data. The process of data extraction automatically from websites is called 'web scraping'.

Introduction to scraping

Since the beginning of the web, web scraping has been the main challenge for anyone who wanted to exploit the richness of information available on the Internet. In the very beginning, very few APIs were available and people used to copy the content of websites by just using copy-paste schema. Then, some programmatic tools were created to follow links (crawling) and extract the content from web pages (scraping). The information was structured by using text patterns (regex) or DOM (Document Object Model) parsing methods. More recently, the development of semantic analysis tools and artificial intelligence enabled alternative approaches, which are much more efficient and closer to human understanding and...