Setup ELK-Stack with Beats

The ELK-Stack is a popular open-source software stack used for log management and analysis, consisting of Elasticsearch, Logstach and Kibana.

Elasticsearch

1. No-SQL database

2. JSON document

3. Built-in Java

Logstash

1. Log ingestion (responsible to store logs into the elasticsearch)

2. Log parsing (customize data before sending to the elasticsearch)

3. Built-in Java

Kibana

1. Visualization tool (Web UI)

Types of Monitoring

1. Uptime Monitoring

• ICMP - ping

Ping -> To check whether the server is up or not

• HTTP - curl

Curl -> To check whether the application is up or not

• TCP - telnet

Telnet -> To check the network packet response

2. Server Monitoring

1. CPU usage

2. Memory usage

3. Network usage

4. Disk usage

5. Swap usage

6. Process

By using the command htop or top in Linux

3. Log Monitoring

1. Application logs

2. System logs

/var/log/apache2
/var/log/nginx

In Centos/RedHat/Amazon

/var/log/httpd

access logs -> /var/log/apache2/access.log

These details will be available in the access logs

1. Client IP (Public IP)

2. Path (example.com/home)

3. User-agent (device and browser)

4. Event details

5. HTTP status code

error logs -> /var/log/apache2/error.log

1. Error message

4. Security Monitoring

• Login Events

Ubuntu -> /var/log/auth.log

Centos/RedHat/Amazon -> /var/log/secure

• User Activity

Install Audit -> /var/log/audit

• Audit Logs

File/dir -> /home/ubuntu/data.txt

Installing Elasticsearch

To set up the ELK-Stack, you need a good configuration machine of at least t2.large because ELK is a heavy software

sudo apt-get update
sudo apt update -y
sudo apt install openjdk-11-jre -y
curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

sudo apt update -y
sudo apt install elasticsearch -y
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch

To check whether it's installed or not

curl localhost:9200

Installing Kibana with Nginx

sudo apt install nginx kibana -y
sudo systemctl enable kibana
sudo systemctl start kibana
sudo nano /etc/kibana/kibana.yml

Inside this file uncomment server.port:5601 & server.host: localhost

systemctl status kibana
sudo service nginx status
sudo apt install apache2-utils

Setup the username and password for the Kibana

sudo htpasswd -c /etc/nginx/htpasswd.users kibana
sudo nano /etc/nginx/htpasswd.users

sudo mv /etc/nginx/sites-available/default /etc/nginx/sites-available/new-default
sudo nano /etc/nginx/sites-available/new-default
sudo nano /etc/nginx/sites-available/default

Inside this file paste the nginx config and made the reverse proxy

server {
    listen 80;

    server_name <private ip>

    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/htpasswd.users;

    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

sudo systemctl restart nginx
sudo systemctl status nginx
sudo apt install logstash

Install any beat by using the command

• Heartbeat - Uptime Monitoring

• Metricbeat - Server & Application

• Pocketbeat - Network Monitoring

• Filebeat - Log Monitoring

• Winlogbeat - Windows Monitoring

• Auditbeat - Security Monitoring

sudo apt install metricbeat

sudo systemctl start metricbeat
sudo systemctl start logstash
sudo systemctl reload nginx

Type the command to get inside the path

sudo nano /etc/metricbeat/metricbeat.yml

Inside this file search Elasticsearch Output, If it is running on a separate server then we have to paste the IP in the place of localhost but in our case, it is in the same system.

To check the modules list in the metricbeat, use the command

sudo metricbeat modules list

If you want to allow any module then we have to use the command

sudo metricbeat modules enable nginx

To Setup the default dashboard in Kibana

sudo metricbeat setup
sudo systemctl start metricbeat

In the dashboard section, you'll find the multiple dashboards for multiple services

Stack Management > Index Management

System Overview

Host Overview

Now we have to create an Index pattern without creating this we won't be able to visualize the data in Kibana

After adding the index pattern click to discover

Right now, we are able to see the data but it's in the raw format

To Waive the Bill, follow the steps:

• Login to AWS Account

• AWS Support

• Create a Case

  • Account and Billing

  • Billing

  • Charge Enquiry

• Write a message to AWS, that I am trying this for learning purposes and so on.