How to install ELK stack on Linux mint-18.3

to Install and Configure the ELK Stack on Linux mint-18.3

ELK stands for Elasticsearch, Logstash, and Kibana and is a robust open source solution for searching, analyzing and visualizing data. Elasticsearch is a distributed, restful search and analytics engine based on Lucene, Logstash is a data processing pipeline for managing events and logs and Kibana is a web application for visualizing data in Elasticsearch. This ELK Stack tutorial should work on other Linux VPS systems as well but  tested and written for a Linux mint-18.3 VPS

Requirements

For this tutorial to work, there are a couple of requirements   
&bull       A Linux Mint-18.3 VPS
&bull       A user with sudo privileges

Installation

Let' s start with updating the system repository

linuxhelp user # apt-get update
Hit:1 http://archive.canonical.com/ubuntu xenial InRelease
Hit:2 http://security.ubuntu.com/ubuntu xenial-security InRelease                              
Hit:3 http://archive.ubuntu.com/ubuntu xenial InRelease                                        
Ign:4 http://packages.linuxmint.com sylvia InRelease                     
Hit:5 http://packages.linuxmint.com sylvia Release                      
Hit:7 http://archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:8 http://archive.ubuntu.com/ubuntu xenial-backports InRelease
Reading package lists... Done

And then we need to Install necessary packages for the smooth installation

linuxhelp user # apt-get install apt-transport-https software-properties-common wget
Reading package lists... Done
Building dependency tree       
Reading state information... Done
apt-transport-https is already the newest version (1.2.24).
wget is already the newest version (1.17.1-1ubuntu1.3).
The following NEW packages will be installed:
  software-properties-common
.
.
Fetched 5,878 B in 0s (10.0 kB/s)               
Selecting previously unselected package software-properties-common.
(Reading database ... 223888 files and directories currently installed.)
Preparing to unpack .../software-properties-common_1.7.2_all.deb ...
Unpacking software-properties-common (1.7.2) ...
Setting up software-properties-common (1.7.2) ...

Java is required for the installation so Install Oracle Java JDK via PPA

linuxhelp user # add-apt-repository ppa:webupd8team/java    
You are about to add the following PPA:
 Oracle Java (JDK) Installer (automatically downloads and installs Oracle JDK7 / JDK8 / JDK9). There are no actual Java files in this PPA.
Important ->  Why Oracle Java 7 And 6 Installers No Longer Work: http://www.webupd8.org/2017/06/why-oracle-java-7-and-6-installers-no.html
Ubuntu 16.10 Yakkety Yak is no longer supported by Canonical (and thus, Launchpad and this PPA). The PPA supports Ubuntu 17.10, 17.04, 16.04, 14.04 and 12.04.
More info (and Ubuntu installation instructions):
- for Oracle Java 7: http://www.webupd8.org/2012/01/install-oracle-java-jdk-7-in-ubuntu-via.html
- for Oracle Java 8: http://www.webupd8.org/2012/09/install-oracle-java-8-in-ubuntu-via-ppa.html
.
.
Oracle JDK 9 is now considered stable. There are currently only 64bit builds (no other builds are available for download: http://www.oracle.com/technetwork/java/javase/downloads/index.html )
 More info: https://launchpad.net/~webupd8team/+archive/ubuntu/java
Press Enter to continue or Ctrl+C to cancel

Executing: /tmp/tmp.5HlimHy0Rs/gpg.1.sh --keyserver
hkp://keyserver.ubuntu.com:80
--recv-keys
EEA14886
gpg: requesting key EEA14886 from hkp server keyserver.ubuntu.com
gpg: key EEA14886: public key " Launchpad VLC"  imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

After adding repository we need to update the system repository.

linuxhelp user # apt-get update
Hit:1 http://archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://security.ubuntu.com/ubuntu xenial-security InRelease                                                                                           
Hit:3 http://archive.canonical.com/ubuntu xenial InRelease                                                                                   
Hit:4 http://archive.ubuntu.com/ubuntu xenial-updates InRelease                                                                      
Hit:5 http://archive.ubuntu.com/ubuntu xenial-backports InRelease                                               
.
.
Get:9 http://ppa.launchpad.net/webupd8team/java/ubuntu xenial/main i386 Packages [2,460 B]
Get:10 http://ppa.launchpad.net/webupd8team/java/ubuntu xenial/main Translation-en [1,260 B]
Ign:11 http://packages.linuxmint.com sylvia InRelease                  
Hit:12 http://packages.linuxmint.com sylvia Release
Fetched 25.6 kB in 5s (4,273 B/s)
Reading package lists... Done

we are ready  to install Oracle Java now.

linuxhelp user # apt-get install oracle-java8-installer
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Suggested packages:
  visualvm ttf-baekmuk | ttf-unfonts | ttf-unfonts-core ttf-kochi-gothic | ttf-sazanami-gothic ttf-kochi-mincho | ttf-sazanami-mincho ttf-arphic-uming
Recommended packages:
  gsfonts-x11 oracle-java8-set-default
The following NEW packages will be installed:
  oracle-java8-installer
0 upgraded, 1 newly installed, 0 to remove and 182 not upgraded.
Need to get 33.1 kB of archives.
.
.
Selecting previously unselected package oracle-java8-installer.
(Reading database ... 223891 files and directories currently installed.)
Preparing to unpack .../oracle-java8-installer_8u161-1~webupd8~0_all.deb ...
Unpacking oracle-java8-installer (8u161-1~webupd8~0) ...
Processing triggers for hicolor-icon-theme (0.15-0ubuntu1) ...
Processing triggers for shared-mime-info (1.5-2ubuntu0.1) ...
Processing triggers for desktop-file-utils (0.22+linuxmint1) ...
Processing triggers for mime-support (3.59ubuntu1) ...
Setting up oracle-java8-installer (8u161-1~webupd8~0) ...
No /var/cache/oracle-jdk8-installer/wgetrc file found.
Creating /var/cache/oracle-jdk8-installer/wgetrc and
using default oracle-java8-installer wgetrc settings for it.
.
.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://edelivery.oracle.com/otn-pub/java/jdk/8u161-.b12/2f38c3b165be4555a1fa6e98c45e0808/jdk-8u161-linux-x64.tar.gz?AuthParam=1518163488_747c27070f7e553b523dbb466080a883 [following]
--2018-02-09 13:32:48--  http://download.oracle.com/otn-pub/java/jdk/8u161-b12/2f38c3b165be4555a1fa6e98c45e0808/jdk-8u161-linux-x64.tar.gz?AuthParam=1518163488_747c27070f7e553b523dbb466080a883
Connecting to download.oracle.com (download.oracle.com)|23.211.208.37|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 189756259 (181M) [application/x-gzip]
Saving to: ‘ jdk-8u161-linux-x64.tar.gz’ 

     0K ........ ........ ........ ........ ........ ........  1%  997K 3m3s
  3072K ........ ........ ........ ........ ........ ........  3% 1.00M 2m57s
  6144K ........ ........ ........ ........ ........ ........  4% 1.04M 2m51s
  9216K ........ ........ ........ ........ ........ ........  6% 1.04M 2m47s
 12288K ........ ........ ........ ........ ........ ........  8% 1.02M 2m43s
 15360K ........ ........ ........ ........ ........ ........  9% 1.02M 2m40s
 18432K ........ ........ ........ ........ ........ ........ 11%  972K 2m39s
 21504K ........ ........ ........ ........ ........ ........ 13% 1014K 2m36s
 24576K ........ ........ ........ ........ ........ ........ 14% 1.02M 2m33s
.
.
168960K ........ ........ ........ ........ ........ ........ 92% 1015K 13s
172032K ........ ........ ........ ........ ........ ........ 94% 1.00M 10s
175104K ........ ........ ........ ........ ........ ........ 96% 1.02M 7s
178176K ........ ........ ........ ........ ........ ........ 97% 1.00M 4s
181248K ........ ........ ........ ........ ........ ........ 99% 1.01M 1s
184320K ........ .......                                     100% 1.03M=3m4s

2018-02-09 13:35:53 (1005 KB/s) - ‘ jdk-8u161-linux-x64.tar.gz’  saved [189756259/189756259]

Download done.
Removing outdated cached downloads...
update-alternatives: error: no alternatives for java
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/ControlPanel to provide /usr/bin/ControlPanel (ControlPanel) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/java to provide /usr/bin/java (java) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/javaws to provide /usr/bin/javaws (javaws) in auto mode
.
.
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/policytool to provide /usr/bin/policytool (policytool) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/rmid to provide /usr/bin/rmid (rmid) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/rmiregistry to provide /usr/bin/rmiregistry (rmiregistry) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/servertool to provide /usr/bin/servertool (servertool) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/tnameserv to provide /usr/bin/tnameserv (tnameserv) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/unpack200 to .
.
.
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/javadoc to provide /usr/bin/javadoc (javadoc) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/javafxpackager to provide /usr/bin/javafxpackager (javafxpackager) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/javah to provide /usr/bin/javah (javah) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/javap to provide /usr/bin/javap (javap) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/javapackager to provide /usr/bin/javapackager (javapackager) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jcmd to provide /usr/bin/jcmd (jcmd) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jconsole to provide /usr/bin/jconsole (jconsole) in auto mode
.
.
(jmc) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jps to provide /usr/bin/jps (jps) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jrunscript to provide /usr/bin/jrunscript (jrunscript) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jsadebugd to provide /usr/bin/jsadebugd (jsadebugd) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jstack to provide /usr/bin/jstack (jstack) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jstat to provide /usr/bin/jstat (jstat) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jstatd to provide /usr/bin/jstatd (jstatd) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jvisualvm to provide /usr/bin/jvisualvm (jvisualvm) in auto mode
.
.
/usr/bin/wsgen (wsgen) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/wsimport to provide /usr/bin/wsimport (wsimport) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/xjc to provide /usr/bin/xjc (xjc) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libnpjp2.so to provide /usr/lib/mozilla/plugins/libjavaplugin.so (mozilla-javaplugin.so) in auto mode
Oracle JDK 8 installed

#####Important########
To set Oracle JDK8 as default, install the " oracle-java8-set-default"  package.
E.g.: sudo apt install oracle-java8-set-default
On Ubuntu systems, oracle-java8-set-default is most probably installed
automatically with this package.
######################

In order to know the version of java installed, make use of the following command

linuxhelp user # java -version
java version " 1.8.0_161" 
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

Now we are going to Install and configure Elasticsearch
We will install Elasticsearch using the package manager from the Elastic repository.  To get key for add repository use the following command

linuxhelp user # wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
OK

To add the repository to Sourcelist file use the following command

linuxhelp user # echo " deb https://artifacts.elastic.co/packages/5.x/apt stable main"  | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
deb https://artifacts.elastic.co/packages/5.x/apt stable main    

And then don' t forget  to Update the system repository

linuxhelp user # apt-get update
Hit:1 http://archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://archive.canonical.com/ubuntu xenial InRelease                                                      
Hit:3 http://ppa.launchpad.net/webupd8team/java/ubuntu xenial InRelease                                         
Hit:4 http://archive.ubuntu.com/ubuntu xenial-updates InRelease                                                 
Ign:5 http://packages.linuxmint.com sylvia InRelease                                           
Hit:6 http://archive.ubuntu.com/ubuntu xenial-backports InRelease                              
.
.
Ign:10 https://artifacts.elastic.co/packages/5.x/apt stable InRelease
Get:11 https://artifacts.elastic.co/packages/5.x/apt stable Release [4,360 B]
Get:12 https://artifacts.elastic.co/packages/5.x/apt stable Release.gpg [473 B]
Get:13 https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages [32.5 kB]
Get:14 https://artifacts.elastic.co/packages/5.x/apt stable/main i386 Packages [32.5 kB]
Fetched 69.8 kB in 3s (22.0 kB/s)    
Reading package lists... Done

we are ready to  install Elasticsearch now

linuxhelp user # apt-get install elasticsearch
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  elasticsearch
0 upgraded, 1 newly installed, 0 to remove and 182 not upgraded.
Need to get 33.7 MB of archives.
After this operation, 37.6 MB of additional disk space will be used.
.
.
(Reading database ... 223920 files and directories currently installed.)
Preparing to unpack .../elasticsearch_5.6.7_all.deb ...
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Unpacking elasticsearch (5.6.7) ...
.
.
Setting up elasticsearch (5.6.7) ...
Processing triggers for systemd (229-4ubuntu21) ...
Processing triggers for ureadahead (0.100.0-19) ...

After installation open Configuration file

linuxhelp user # vim /etc/elasticsearch/elasticsearch.yml

Here at the line no: 55  add your IP address

network.host: “ IP address” 

Next, you have to restart the elastic search  to make the changes effect

linuxhelp user # systemctl restart elasticsearch.service 
linuxhelp user # systemctl enable elasticsearch.service 

Now Install and configure Kibana

linuxhelp user # apt-get install kibana
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  kibana
0 upgraded, 1 newly installed, 0 to remove and 182 not upgraded.
Need to get 52.8 MB of archives.
.
.
(Reading database ... 224052 files and directories currently installed.)
Preparing to unpack .../kibana_5.6.7_amd64.deb ...
Unpacking kibana (5.6.7) ...
Processing triggers for systemd (229-4ubuntu21) ...
Processing triggers for ureadahead (0.100.0-19) ...
Setting up kibana (5.6.7) ...
Processing triggers for systemd (229-4ubuntu21) ...
Processing triggers for ureadahead (0.100.0-19) ...

After installing open the configuration file

linuxhelp user # vim /etc/kibana/kibana.yml 

Here, Line no: 7   add your IP address

Server.host: “ IP adresss” 

And then you must restart the kibana service to make the changes effect

linuxhelp user # systemctl restart kibana.service 
linuxhelp user # systemctl enable kibana.service 

Next, Install and configure Nginx as a reverse proxy

linuxhelp user # apt-get install nginx
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  nginx-common nginx-core
Suggested packages:
  fcgiwrap nginx-doc
The following NEW packages will be installed:
  nginx nginx-common nginx-core
.
.
Do you want to continue? [Y/n] y
Get:1 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 nginx-common all 1.10.3-0ubuntu0.16.04.2 [26.6 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 nginx-core amd64 1.10.3-0ubuntu0.16.04.2 [428 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 nginx all 1.10.3-0ubuntu0.16.04.2 [3,490 B]
.
.
Unpacking nginx-common (1.10.3-0ubuntu0.16.04.2) ...
Selecting previously unselected package nginx-core.
Preparing to unpack .../nginx-core_1.10.3-0ubuntu0.16.04.2_amd64.deb ...
Unpacking nginx-core (1.10.3-0ubuntu0.16.04.2) ...
Selecting previously unselected package nginx.
Preparing to unpack .../nginx_1.10.3-0ubuntu0.16.04.2_all.deb ...
.
.
Setting up nginx-common (1.10.3-0ubuntu0.16.04.2) ...
Setting up nginx-core (1.10.3-0ubuntu0.16.04.2) ...
Setting up nginx (1.10.3-0ubuntu0.16.04.2) ...
Processing triggers for systemd (229-4ubuntu21) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for ufw (0.35-0ubuntu2) ...

Create a basic authentication file with the OpenSSL command.

linuxhelp user # echo " admin:$(openssl passwd -apr1 YourStrongPassword)"  | sudo tee -a /etc/nginx/htpasswd.kibana
admin:$apr1$6uqqpExh$rIMhJ9FNzGEjCUFZR9cRD

then, Delete the default nginx, virtual host

linuxhelp user # rm /etc/nginx/sites-enabled/default 

create a virtual host configuration file

linuxhelp user # vim /etc/nginx/sites-available/kibana

Populate the file with following lines

server {
    listen 80 default_server 
    server_name _ 
    return 301 https://$server_name$request_uri 
}

server {
    listen 443 default_server ssl http2 
 
    server_name _ 
 
    ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem 
    ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key 
    ssl_session_cache shared:SSL:10m 
 
    auth_basic " Restricted Access"  
    auth_basic_user_file /etc/nginx/htpasswd.kibana 
 
    location / {
        proxy_pass http://localhost:5601 
        proxy_http_version 1.1 
        proxy_set_header Upgrade $http_upgrade 
        proxy_set_header Connection ' upgrade'  
        proxy_set_header Host $host 
        proxy_cache_bypass $http_upgrade 
    }
}

And then Creating a symbolic link

linuxhelp user # ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/kibana

Now, Let' s Test the Nginx configuration 

linuxhelp user # nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

The test is successful so restart nginx service now.

linuxhelp user # systemctl  restart  nginx.service 

At last, Install Logstash using the following command

linuxhelp user # apt-get install logstash
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  logstash
0 upgraded, 1 newly installed, 0 to remove and 182 not upgraded.
Need to get 104 MB of archives.
.
.                                                                                                                 
Selecting previously unselected package logstash.
(Reading database ... 262086 files and directories currently installed.)
Preparing to unpack .../logstash_1%3a5.6.7-1_all.deb ...
Unpacking logstash (1:5.6.7-1) ...
Setting up logstash (1:5.6.7-1) ...
Using provided startup.options file: /etc/logstash/startup.options
Successfully created system startup script for Logstash

With this, the method  to Install and Configure the ELK Stack on Linux mint-18.3 comes to an end.