Kibana offers a simple way to visualize logs/data coming from Logstash. With Twitter as one of the standard inputs of Logstash, it is very easy to visualize Twitter trends in Kibana. In this post, I would be demonstrating how you can get Twitter data for few search terms, pass it through Elasticsearch to get indexes and customize Kibana dashboard to get real time changes in the tweets. I’m using Ubuntu 14.04 on AWS. If you haven’t used AWS before, you might want to look at my previous post on getting started with AWS. Let’s have a look at what we are gonna get at the end of this tutorial –

I assume that you have instantiated an EC2 instance (Ubuntu 14.04) and logged into it. Combination of Logstash, Elasticsearch and KIbana is so popular to track and manage logs that it is termed as ELK stack. As this tutorial is for learning purpose, we would be setting up everything on a single machine. Let’s get started –

1. Install Java –

   sudo add-apt-repository -y ppa:webupd8team/java
   sudo apt-get update
   sudo apt-get -y install oracle-java7-installer
   

2. Install Elasticsearch –

   wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -
   echo 'deb http://packages.elasticsearch.org/elasticsearch/1.1/debian stable main' | sudo tee /etc/apt/sources.list.d/elasticsearch.list
   sudo apt-get update
   sudo apt-get -y install elasticsearch=1.1.1
   

3. Edit Elasticsearch config file – Open file “/etc/elasticsearch/elasticsearch.yml” and line “script.disable_dynamic: true” at the end of file. Search for line with “network.host:” and change it to “network.host: localhost”. Now, start elasticsearch using –

   sudo service elasticsearch restart

4. Install Kibana –

   cd ~; wget https://download.elasticsearch.org/kibana/kibana/kibana-3.0.1.tar.gz
   tar xvf kibana-3.0.1.tar.gz
   

5. Edit Kibana config file “kibana-3.0.1/config.js” and change line with keyword “elasticsearch” to “elasticsearch: “http://”+window.location.hostname+”:80″,”. Now move files to proper location –

   sudo mkdir -p /var/www/kibana3
   sudo cp -R kibana-3.0.1/* /var/www/kibana3/
   

6. Install nginx –

   sudo apt-get install nginx
   cd ~; wget https://gist.githubusercontent.com/thisismitch/2205786838a6a5d61f55/raw/f91e06198a7c455925f6e3099e3ea7c186d0b263/nginx.conf
   

7. Now edit this config file (nginx.conf) and change “server_name” value to the Elastic IP of the node and “root” to “root /var/www/kibana3;”. Now, copy the file to the right location. Provide a username in place of <username> below (and proper password, when asked) –

   sudo cp nginx.conf /etc/nginx/sites-available/default
   sudo apt-get install apache2-utils
   sudo htpasswd -c /etc/nginx/conf.d/kibana.myhost.org.htpasswd <username>
   sudo service nginx restart
   

8. Install Logstash –

   echo 'deb http://packages.elasticsearch.org/logstash/1.4/debian stable main' | sudo tee /etc/apt/sources.list.d/logstash.list
   sudo apt-get update
   sudo apt-get install logstash=1.4.2-1-2c0f5a1
   

9. Let’s configure Logstash now. Create a file “logstash.conf” in home directory and put the following content in it. “term1″ (eg. modi) is any term you want to search in tweets, “term2″ (eg. “obama”) is any other term that that you want to compare the results with. “tweets1″ (eg. “moditweets”) and “tweets2″ (eg. “obamatweets”) can be anything, you can give some meaningful names to it. We would refer to both the kinds of tweets using these name in Kibana. Other values of “consumer_key”, “consumer_secret”, “oauth_token” and “oauth_token_secret” should be taken from a Twitter app which you need to create using the Twitter developer account –

   input {
    twitter {
    consumer_key => "<proper_value>"
    consumer_secret => "<proper_value>"
    keywords => ["<term1>"]
    oauth_token => "<proper_value>"
    oauth_token_secret => "<proper_value>"
    type => "tweets1"
   }
   
   twitter {
   consumer_key => "<proper_value>"
   consumer_secret => "<proper_value>"
   keywords => ["term2"]
   oauth_token => "<proper_value>"
   oauth_token_secret => "<proper_value>"
   type => "tweets2"
   
   }
   }
   
   output {
     elasticsearch { host => localhost }
     stdout { codec => rubydebug }
   }
   

10. Once this is done, you should see Kibana dashboard if you point your browser to the Elastic IP address of the EC2 node.
11. Configuring Kibana – To visualize the tweets in real time as shown in the video we need to make few changes –
    1. Add two queries in the “QUERY” section. Click + to add another query. Enter “tweets1″ in one query and “tweets2″ in another query.
    2. In the top right corner, click on “Configure dashboard”. Click on Timepicker and change “Relative time options” to “1m” and “Auto-refresh options” to “1s”.
    3. Now go to “FILTERING” and close all filterings. In the top right section, there is one drop down to select time filtering. Click it and select “Last 1m”. Click it again and select “Auto-Refresh -> Every 1s”.
    4. You can configure the main graph area the way you want. For eg. I converted bars to lines. There are many options you can change and select the one best suited to your needs.
12.  Now is the time to see the magic. Let’s start Logstash and wait for sometime. We would start observing the trends for the two kinds of tweets in Kibana dashboard. To start Logstash, run –

    sudo /opt/logstash/bin/logstash -f ~/logstash.conf
    

Share and Enjoy

• Facebook • Twitter • Delicious • LinkedIn • StumbleUpon • Add to favorites • Email • RSS

The post Visualize Real Time Twitter Trends With Kibana appeared first on Harsh Tech Talk.