(EFK) Elasticsearch, Fluentd, Kibana Setup – [Step By Step Guide]

http://www.techstricks.com/elasticsearch-fluentd-kibana-efk-setup-guide/

The EFK (Elasticsearch, Fluentd and Kibana) stack is an open source alternative to paid log management, log search and log visualization services like Splunk, SumoLogic and Graylog (Graylog is open source but enterprise support is paid). These services are used to search large amounts of log data for better insights, tracking, visualisation and analytical purposes. The EFK stack i.e. Elasticsearch, Fluentd and Kibana are separate open source projects that together make an amazing open source centralized log management stack that is not only free to use and easy to setup/install but also scalable and can handle really large amounts of log data in realtime. This article documents how to setup Elasticsearch, Fluentd and Kibana and putting it all together to get the best out of your boring log data. So Without further ado, Let’s jump right into the setup/installation process.

Let us first create a folder that we’ll put our EFK stack into. I chose to use work in /Users/amyth/installs/efk. You can choose any location that you would like to work on, Really.

1. Installing & Running Elasticsearch

1.1 Java Installation

Let us get started by installing Java as it is one of the core dependencies of elastichsearch.

Once the installation is finished, Confirm it by checking the java version using the following command.

and you should see something like the following.

1.2 Installing Elasticsearch

Next, download elasticsearch (v2.1.0)  and uncompress the downloaded package.

Now let’s run the an elasticsearch instance by cd’ing into the elasticsearch folder and running the elasticsearch script in the bin folder. To run elasticsearch as a daemon use the -d argument while calling the script.

or to run it as a daemon

After running Elasticsearch, confirm you have a running instance by navigating to 127.0.0.1:9200 and you should see something like the following:

2. Installing & Running Kibana

Now let us install and configure Kibana. First download kibana from this download page. Once downloaded move the download file to our efk install location and uncompress the downloaded file.

Next, lets run kibana using the following command

Now in your web browser navigate to http://0.0.0.0:5601 and you should see the kibana dashboard. Something like the following image.

EFK, Elasticsearch, Fluentd, Kibana

kibana

Now before we create indices, let’s get the third and final pillar to our stack up and running.

3. Installing & Running Fluentd

For installation of Fluentd, it provides a bash script that automates the installation process. These Bash scripts are available for:

  • ubuntu: Trusty, Precise and Lucid
  • debian: Jessie, Wheezy and Squeeze.

Simply get and run these scripts using one of the following commands below (based on your operating system)

Once Installed, Let’s start the td-agent.

To make sure you have td-agent running, try the status command

4. Put together EFK, Elasticsearch, Fluentd and Kibana stack.

4.1 Get Required Fluentd Plugins

Now let us put all of it together to make it work. First we need a few fluentd plugins installed. Let’s install them by using the following commands.

4.2 Send Syslog to Elasticsearch via Fluentd

Next, we want to send some log data through fluentd to elasticsearch. In this case we’ll configure fluentd to forward the syslog data to ES. In order to do so, opent file /etc/td-agent/td-agent.conf and replace the existing configuration with the configuration below.

Now let’s launch fluentd using the following commands.

We would also require to tell syslog/rsyslog to stream the log data to fluentd. So let’s open the syslog configuration file.

and add the following line to it. This tells syslog to forward the log data to host 127.0.0.1 which is our local host on port 5140. As fluentd listens to port 5140 by default.

Now to reload the configuration so that it include our recent changes, Let’s restart the syslog/rsyslog service.

Now let’s create an elasticsearch index named kibana where dynamic mapping is enabled.

Now go to your kibana dashboard by navigating in your web browser to ‘http://0.0.0.0:5601′ and choose the settings tab and enter kibana* in the “index name or pattern” field. Then uncheck “Index contains time-based events” and click on the create button.

EFK

Kibana

 

Now go to the Discover link and you should see and search the logs from your syslog.

Moving from JDK 1.7 to JDK 1.8 on Ubuntu

http://stackoverflow.com/questions/30177455/moving-from-jdk-1-7-to-jdk-1-8-on-ubuntu

http://dev.haufe.com/log-aggregation/

http://logz.io/blog/kubernetes-log-analysis/

cd /tmp
$ wget http://download.oracle.com/otn-pub/java/jdk/8u74-b02/jdk-8u74-linux-x64.tar.gz\?AuthParam\=1458001079_a6c78c74b34d63befd53037da604746c
$ tar xzf jdk-8u74-linux-x64.tar.gz?AuthParam=1458001079_a6c78c74b34d63befd53037da604746c
$ sudo mv jdk1.8.0_74 /opt
$ cd /opt/jdk1.8.0_74/
$ sudo update-alternatives --install /usr/bin/java java /opt/jdk1.8.0_91/bin/java 2
$ sudo update-alternatives --config java // select version
$ sudo update-alternatives --install /usr/bin/jar jar /opt/jdk1.8.0_91/bin/jar 2
$ sudo update-alternatives --install /usr/bin/javac javac /opt/jdk1.8.0_91/bin/javac 2
$ sudo update-alternatives --set jar /opt/jdk1.8.0_91/bin/jar
$ sudo update-alternatives --set javac /opt/jdk1.8.0_74/bin/javac
$ java -version // you should have the updated java

MINSTALL – An all in one script designed to minimize a server install and manage all services

https://github.com/KnightSwarm/Minstall

Description
An all in one script designed to minimize a server install and manage all services.

Compatibility

  • Operating Systems:
    • Debian 6 (Squeeze) 64 Bit
    • Debian 6 (Squeeze) 32 Bit
    • Debian 7 (Wheezy) 64 Bit (WIP)
    • Debian 7 (Wheezy) 32 Bit (WIP)
  • Platforms:
    • Hardware
    • KVM
    • OpenVZ
    • VirtualBox
    • VMware
    • Xen HVM

Notes
Run on a freshly installed server under root, may not work under an already setup server!

Install LNMP with script Tuxlite on Debian 8

https://github.com/thadroe/TuxLite-Debian-8

image-2016111009-001

Quick Install (Git)

# Install git and clone TuxLite
aptitude install git
git clone https://github.com/Mins/TuxLite.git
cd TuxLite

# Edit options to enter server IP, MySQL password etc.
nano options.conf

# Make all scripts executable.
chmod 700 *.sh
chmod 700 options.conf

# Install LAMP or LNMP stack.
./install.sh

# Add a new Linux user and add domains to the user.
adduser johndoe
./domain.sh add johndoe yourdomain.com
./domain.sh add johndoe subdomain.yourdomain.com

# Install Adminer or phpMyAdmin
./setup.sh dbgui

# Enable/disable public viewing of Adminer/phpMyAdmin
./domain.sh dbgui on
./domain.sh dbgui off

Install LNMP with script Tuxlite

http://forum.webhostbug.com/discussion/85/install-lnmp-with-tuxlite-script

https://github.com/Mins/TuxLite

 

Hi. You bought a new VPS. And if you want to build LNMP server faster. I recommended you use TuxLite script.

LNMP stand for Linux Nginx MySQL and PHP and this case we use Ubuntu (or Debian) Nginx MySQL PHP5. First we need correct and update linux system ( Ubuntu and Debian)

Correct hostname

  1. apt-get update
  2. apt-get install git
  3. git clone https://github.com/Mins/TuxLite.git
  4. cd TuxLite

Then edit config file

Open file install.sh and delete those line

  1. echo ""
  2. echo "Installing updates & configuring SSHD / hostname."
  3. sleep 5
  4. ./setup.sh basic

Open file options.conf

  1. #Fill your server detail, example
  2. HOSTNAME=sv1
  3. HOSTNAME_FQDN=sv1.webhostbug.net
  4. SERVER_IP=192.241.140.234
  5. SSHD_PORT=22
  6. # phpMyAdmin = 1, Adminer = 2
  7. DB_GUI=2
  8. # Enable or disable AWStats. Options = yes|no
  9. AWSTATS_ENABLE=no

Open file nginx.conf then uncomment this line server_names_hash_bucket_size 64; Ok Now we will make all scripts executable.

  1. chmod 700 *.sh
  2. chmod 700 options.conf

Install LAMP or LNMP stack.pls wait to finishing.

./install.sh

Add a new Linux user and add domains to the user.

  1. adduser johndoe
  2. ./domain.sh add johndoe yourdomain.com
  3. ./domain.sh add johndoe subdomain.yourdomain.com

Install Adminer or phpMyAdmin

./setup.sh dbgui

Enable/disable public viewing of Adminer/phpMyAdmin

  1. ./domain.sh dbgui on
  2. ./domain.sh dbgui off

#Install and enable Varnish. Do not use Varnish unless you know #how to configure the VCL.

  1. ./varnish.sh install
  2. ./varnish.sh on
  3. ./varnish.sh off

#Post Install

#Your install is complete and your site works. Great! Now what? As #a responsible sysadmin you will need to perform some maintenance #and post install procedures.

  1. #Occasionally update your server using
  1. aptitude update
  2. aptitude safe-upgrade
  1. #Set up cronjobs to backup your databases and sites. The #backup.sh script can do this for you. Remember to download and #keep copies of your sites locally.
  2. #Enable/Disable phpMyAdmin and AWstats from being publicly #accessible. This is to prevent brute force attacks on your #server. Enable only when you need them.
  3. #Consider tweaking configurations to better utilize the amount #of memory available to your server. Read the readme for #respective web stacks for more information on config files to edit.
  4. #Install Fail2Ban to protect your server from brute force SSH #login attempts.
  5. `aptitude install fail2ban`
  6. #Install DDOS Deflate script to mitigate denial of service attacks. Refer here http://deflate.medialayer.com/

Install LNMP with TuxLite script

 

Fluentd + Elasticsearch + Kibana – Free Alternative to Splunk

http://docs.fluentd.org/articles/free-alternative-to-splunk-by-fluentd

Splunk is a great tool for searching logs, but its high cost makes it prohibitive for many teams. In this article, we present a free and open source alternative to Splunk by combining three open source projects: Elasticsearch, Kibana, and Fluentd.

Elasticsearch is an open source search engine known for its ease of use. Kibana is an open source Web UI that makes Elasticsearch user friendly for marketers, engineers and data scientists alike.

By combining these three tools (Fluentd + Elasticsearch + Kibana) we get a scalable, flexible, easy to use log search engine with a great Web UI that provides an open-source Splunk alternative, all for free.

In this guide, we will go over installation, setup, and basic use of this combined log search solution. This article was tested on Mac OS X Mountain Lion. If you’re not familiar with Fluentd, please learn more about Fluentd first.

Table of Contents

Prerequisites

Java for Elasticsearch

Please confirm that your Java version is 7 or higher.

$ java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

Now that we’ve checked for prerequisites, we’re now ready to install and set up the three open source tools.

Set Up Elasticsearch

To install Elasticsearch, please download and extract the Elasticsearch package as shown below.

$ curl -O https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.tar.gz
$ tar zxvf elasticsearch-1.7.1.tar.gz
$ cd elasticsearch-1.7.1/

Once installation is complete, start Elasticsearch.

$ ./bin/elasticsearch

Set Up Kibana

To install Kibana, download it via the official webpage and extract it. Kibana is a HTML / CSS / JavaScript application. Dowload page is here. In this article, we download Mac OS X binary.

$ curl -O https://download.elastic.co/kibana/kibana/kibana-4.1.1-darwin-x64.tar.gz
$ tar zxvf kibana-4.1.1-darwin-x64.tar.gz
$ cd kibana-4.1.1-darwin-x64/

Once installation is complete, start Kibana and run ./bin/kibana. You can modify Kibana’s configuration via config/kibana.yml.

$ ./bin/kibana

Access http://localhost:5601 in your browser.

Set Up Fluentd (td-agent)

In this guide We’ll install td-agent, the stable release of Fluentd. Please refer to the guides below for detailed installation steps.

Next, we’ll install the Elasticsearch plugin for Fluentd: fluent-plugin-elasticsearch.

Then, install fluent-plugin-elasticsearch as follows.

  • td-agent v2: /usr/sbin/td-agent-gem install fluent-plugin-elasticsearch
  • td-agent v1: /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-elasticsearch

We’ll configure td-agent (Fluentd) to interface properly with Elasticsearch. Please modify /etc/td-agent/td-agent.conf as shown below:

<source>
  @type syslog
  port 42185
  tag syslog
</source>

<source>
  @type forward
</source>

<match syslog.**>
  @type elasticsearch
  logstash_format true
  flush_interval 10s # for testing
</match>

fluent-plugin-elasticsearch comes with a logstash_format option that allows Kibana to search stored event logs in Elasticsearch.

Once everything has been set up and configured, we’ll start td-agent.

$ sudo /etc/init.d/td-agent start

Set Up rsyslogd

In our final step, we’ll forward the logs from your rsyslogd to Fluentd. Please add the following line to your /etc/rsyslog.conf, and restart rsyslog. This will forward your local syslog to Fluentd, and Fluentd in turn will forward the logs to Elasticsearch.

*.* @127.0.0.1:42185

Please restart the rsyslog service once the modification is complete.

$ sudo /etc/init.d/rsyslog restart

Store and Search Event Logs

Once Fluentd receives some event logs from rsyslog and has flushed them to Elasticsearch, you can search the stored logs using Kibana by accessing Kibana’s index.html in your browser. Here is an image example.

To manually send logs to Elasticsearch, please use the logger command.

$ logger -t test foobar

When debugging your td-agent configuration, using out_copy + out_stdout will be useful. All the logs including errors can be found at /etc/td-agent/td-agent.log.

<match syslog.**>
  @type copy
  <store>
    # for debug (see /var/log/td-agent.log)
    @type stdout
  </store>
  <store>
    @type elasticsearch
    logstash_format true
    flush_interval 10s # for testing
  </store>
</match>

Conclusion

This article introduced the combination of Fluentd and Kibana (with Elasticsearch) which achieves a free alternative to Splunk: storing and searching machine logs. The examples provided in this article have not been tuned.

If you will be using these components in production, you may want to modify some of the configurations (e.g. JVM, Elasticsearch, Fluentd buffer, etc.) according to your needs.

Learn More

Fluentd and Graylog2 for End-to-End Log Analysis

http://www.fluentd.org/guides/recipes/graylog2

This article shows how to set up Fluentd with Graylog2. Graylog2 is a popular log analysis framework powered by Elasticsearch and MongoDB. Fluentd can output data to Graylog2 in the GELF format to take advantage of Graylog2’s analytics and visualization features.

Prerequisites

  • A basic understanding of Fluentd
  • Ubuntu/Debian Linux
  • sudo/root access

Setup

Dependencies

Install the dependencies with the following command.

sudo apt-get update
sudo apt-get install mongodb-server openjdk-7-jre-headless uuid-runtime adduser --yes

Elasticsearch

Latest Graylog2 requires Elasticsearch 1.7 or later, which can be installed with the following commands.

NOTE: Graylog2 doesn’t support Elasticsearch 2.0 yet.

wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.7.1.deb
dpkg -i elasticsearch-1.7.1.deb

Then, open /etc/elasticsearch/elasticsearch.yml, find the line that says

cluster.name: elasticsearch

Uncomment and update it to cluster.name: graylog2.

Also, for better security, disable dynamic scripting for Elasticsearch by adding the following line to /etc/elasticsearch/elastcisearch.yml.

script.disable_dynamic: true

Elasticsearch is ready. Start it with

sudo /etc/init.d/elasticsearch restart

Graylog2

In this article, we use Graylog2 1.2.

Get the GPG key.

wget -qO - https://raw.githubusercontent.com/Graylog2/graylog2-puppet/master/files/RPM-GPG-KEY-graylog2 | apt-key add -

Update the package list.

echo 'deb http://packages.graylog2.org/repo/debian/ trusty 1.2' > /etc/apt/sources.list.d/graylog2.list

Finally, install Graylog’s components.

sudo apt-get update
sudo apt-get install graylog-server graylog-web graylog2-stream-dashboard --yes

Edit /etc/default/graylog-server and /etc/default/graylog-web and update RUN=no to RUN=yes.

Edit /etc/graylog/server/server.conf and cocnfigure the following parameters:

  • password_secret
  • root_password_sha2

For root_password_sha2, run echo -n YOUR_PASSWORD | shasum -a 256 and copy and paste the output string. This “YOUR_PASSWORD” will be used later to log into Graylog2’s web interface.

Edit /etc/graylog/web/web.conf and configure the following parameters:

Finally, start Graylog2!

sudo /etc/init.d/graylog-server restart
sudo /etc/init.d/graylog-web restart

Nginx (Optional)

If you are sharing access to Graylog2’s web interface outside of your network, You need to set up a proxy server. Nginx is recommended.

First, install Nginx.

sudo apt-get install nginx --yes

Edit /etc/nginx/sites-available/default so that it looks exactly like this:

server {
  listen 80;
  location / {
    proxy_pass http://localhost:9000/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_connect_timeout 150;
    proxy_send_timeout 100;
    proxy_read_timeout 100;
    proxy_buffers 4 32k;
    client_max_body_size 8m;
    client_body_buffer_size 128k;
  }
}

Then, restart nginx.

sudo /etc/init.d/nginx restart

Graylog2 GELF input

Go to . If Nginx was configured in the previous section, it should be served from port 80. Otherwise, it should be at port 9000.

To log in, use “admin” as the username and “YOUR_PASSWORD” from the Graylog2 server setup as your password.

Once logged in, click on “System” in the top nav. Next, click on “Inputs” from the left nav. (Or, you can just go to <YOUR_ GRAYLOG2 URL>/system/inputs).

Then, from the dropdown, choose “GELF UDP” and click on “Launch new input”, which should pop up a modal dialogue, Since the default values are good, fill the “Title” and just click “Launch”.

Now, Graylog2 is ready to accept messages from Fluentd over UDP. It’s time to configure Fluentd.

Fluentd

See here for all the download options. Here, we are using the Trusty deb package.

curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh

Then, install the out_gelf plugin to send data to Graylog2. Currently, the GELF plugin is not published to Rubygems, so we download the plugin file and place it in /etc/td-agent/plugin. We also need to gem-install GELF’s Ruby client.

/usr/sbin/td-agent-gem install gelf
cd /etc/td-agent/plugin
sudo wget https://raw.githubusercontent.com/emsearcy/fluent-plugin-gelf/master/lib/fluent/plugin/out_gelf.rb

Finally, edit /etc/td-agent/td-agent.conf as follows.

<source>
  type syslog
  tag graylog2
</source>
<source>
  type tail
  format nginx
  path /var/log/nginx/access.log
  tag graylog2.nginx
</source>
<match graylog2.**>
  type copy
  <store>
    type gelf
    host 0.0.0.0
    port 12201
    flush_interval 5s
  </store>
  <store>
    type stdout
  </store>
</match>

That configuration listens to two data sources:

  1. Nginx access log of Graylog2 web app itself. Admittedly, this is a toy example, but it showcases the in_tail plugin.
  2. It listens to syslog input. Fluentd automatically parses common fields such as ident.

We need to make a couple of tweaks to successfully ingest Nginx access logs and syslogs.

  1. Run sudo chown -R td-agent:td-agent /var/log/nginx so that Fluentd can read Nginx access logs.
  2. Open /etc/rsyslog.conf and add the following line to the beginning of the file: *.* @127.0.0.1:5140 Then, restart rsyslogd by running sudo /etc/init.d/rsyslog restart.

Finally, start Fluentd with the following command.

sudo /etc/init.d/td-agent restart

Confirming the setup inside Graylog2’s UI

When you log back into Graylog2, you should be seeing a graph like this (wait for events to flow in)

What’s Next?

Interested in other data sources and output destinations? Check out the following resources: