Creating a Logstash configuration file

In this section, we will develop a configuration file that will contain input and output. Here, input will be twitter and output will be elasticsearch, as we need to store data in Elasticsearch for visualization in Kibana. We will not use a filter as we want to store the tweets in the same way as they are tweeted.

The configuration file will look like this:

input {
  twitter {
    consumer_key =>  "XXXXXXXXXXXXXXXXXXX"
    consumer_secret =>  "XXXXXXXXXXXXXXXXXX"
    oauth_token =>  "XXXXXXXXXXXXXXXXXXXXXXXX"
    oauth_token_secret =>  "XXXXXXXXXXXXXXXXXXXX"
    keywords => ["#DragMeDownDay,"#BePositive"]
    full_tweet => "true"
  }
}
output {
  elasticsearch {
    protocol => "http"
    host => "localhost"
    port => "9200"
    index => "twitter"
    document_type => "realtime"
  }
}

Save this configuration as twitter.conf inside the bin folder of the downloaded Logstash folder.

Let's decode each parameter for better...