
Django migration conflicts are common when developing a Django application. In this blog, we will leverage AWS Lambda, Bitbucket webhooks, and CircleCI workflows to prevent the introduction of Django migration conflict to the dev/main branch.

## Infrastructure requirements

Before you read any further, be aware that this blog post assumes that you have these resources available at your disposal:
* Webhook service
* VCS
* CI/CD

For this blog, we will use the following infrastructure:
* AWS Lambda as webhook service
* Bitbucket for VCS
* CircleCI as CI/CD.  

You can use other solutions for webhook service, VCS, and CI/CD. The idea is the same, but you will have to figure out how to make this work on your infrastructure. If you use the same infrastructure, be sure that you have sufficient permissions for API Gateway and Lambda AWS services. You will also need administrator rights to your project's Bitbucket repository.

## Motivation

Since you are reading this article, you may as well already know what Django migration conflict is. Still, we need to set up a common ground before we move on. Assuming that you work in a company that develops Django apps, you can easily get Django migration conflict. All it requires is for you and your colleague to edit the same Django model or create a migration file with the same order number at the same time. You both work on some features and each of you creates a separate branch for your implementation. Your colleague managed to get his branch merged into the main branch sooner than you and had his migrations applied. In such a case, you have to pull the newest changes and resolve conflicts, if there are any. However, it is not that hard for humans to forget, right? If you forget to pull your colleague's changes before you merge your branch into the main branch, you will introduce Django migration conflict into the main branch. Someone will have to pull the main branch, fix it and push the changes, or undo your merge to the main branch. Especially in agile and incremental development, where there are several pull requests merged on daily basis, it can become very tedious to fix forgotten Django migration conflicts in the main branch, especially if they can be prevented.   

## Prevention

As the name of this blog post suggests, we can use CI/CD. Let us go back to the scenario with you and your colleague. For this prevention to work, you need to have set up CI/CD workflows for your branches. Let us assume that you have different workflows set up for different types of branches. It is logical to have a different workflow for the feature/bugfix branch and dev/main branch. With CI/CD in place, we can add Django migration conflict check to feature/bugfix workflow. This way, you and your colleagues can be aware of possible conflicts before propagating them further and simply fix them on your branch. Since these conflict checks will be done on feature/bugfix branches, your main/dev branches will not require these checks.

## Prevention flow
![Flow](https://github.com/remastr/django-migration-conflict-check/raw/main/docs/img/flow.png "Flow diagram" 100x20)

At the first stage, a developer does an action (commit, push, merge, ...) in VCS. This VCS action triggers an event, which is sent to a Webhook service. Webhook service provides you with the means to process an event and send a response. In our case, we will respond to CI/CD associated with our VCS and trigger an action (rebuild project, trigger new pipeline, ...).

## Setup
The setup procedure involves several steps. Please follow them strictly in this order:
* Create API Keys.
* Set up your <Webhook-Service> + <CI/CD> pair.
* Create Bitbucket webhook.
* CircleCI config - Django migration check.

### Create API keys
##### **Create Bitbucket App Password:**
Bitbucket App Password is your API key. With this key, you can interact with your Bitbucket. At the time of creation, you have to assign permissions to the key. Permissions cannot be modified afterward. This key does not require 2-step verification, so be careful when sharing this key. Lastly, please note that you can see the content of the key only at the time of creation.

* Sign in to the Bitbucket Cloud.
* Go to your "Personal settings".
* Under the "ACCESS MANAGEMENT" section, go to "App passwords" and click there on the "Create app password" button.
* Select permissions according to your use case and click on "Create".
* Save your App password in a safe place. You will need it later on.  


##### **Create CircleCI Personal API Token:**
As with the Bitbucket App Password, this is also your API key and you can see the context only when creating the key. However, in this case, you cannot assign any permissions. This key has by default full access rights, so you have to be extremely careful with whom you share this key.

* Sign in to the CircleCI.
* Go to your "User settings".
* Click on "Personal API Tokens" and there click on the "Create New Token" button.
* Name accordingly your token and hit "Add API Token".
* Save your API token in a safe place.

### AWS Lambda + CircleCI
We will use AWS Lambda to run our webhook processing script in a container. Running code in a container without any server is called serverless computing. The first sentence from Wiki explains it perfectly - "Serverless computing is a cloud computing execution model in which the cloud provider allocates machine resources on-demand, taking care of the servers on behalf of their customers." This container will be reachable from the Internet thanks to the HTTP trigger that we are going to set up. HTTP trigger is done via API Gateway. API Gateway is a service used for creating API. It will provide us with a web address. We can configure these web address URL paths and what code to execute on each of them. Since our webhook has only one use case, the root web address will do. Lastly, we need to upload the source code of our webhook processing script into the AWS Lambda function we have created. At our Github repository, we already have a script available that you can use. All that's left is to set up AWS Lambda function environment variables. The script will need your Bitbucket username, API keys you have created, and lastly, a slug of the project on which you want to Django migration check.

##### **Create a function:**  
* Sign in to the AWS Management console.
* Go to Lambda service.
* Click on the "Create function" button.
* Select the "Author from scratch" option.
* Name your function.
* Select Python3.8 runtime.
* Click on the "Create function" button.  


##### **Add HTTP trigger:**  
* On the function page, click on the "Add trigger" button.
* Select "API Gateway" from the dropdown menu.
* Next, from the second dropdown menu, choose "Create an API".
* Select "HTTP API" API type.
* Select "Open" security from the third dropdown menu.
* Click on the orange "Add" button.  


##### **Upload code:**  
* Download the latest zip package from [ releases](https://github.com/remastr/django-migration-conflict-check/releases " releases").
* On the function page, under "Code" section, click on "Upload from" button and choose ".zip file" option.
* Upload the "aws_deployment.zip" you have downloaded in the first step.  


##### **Set Lambda Environment variables:**  
* On the function page, under "Configuration" section, click on "Environment variables".
* Click on the "Edit" button and add following variables with your values:
```
BITBUCKET_API_URL	https://api.bitbucket.org
BITBUCKET_APP_PASSWORD	<Your-Bitbucket-App-Password>
BITBUCKET_USERNAME	<Your-Bitbucket-Username>
CIRCLECI_API_TOKEN	<Your-Personal-CircleCI-API-Token>
CIRCLECI_PROJECT_SLUG   bb/<Your-CircleCI-Organization-Name>/<CircleCI-Project-Name>
```
* Click on the "Save" button.

### Create Bitbucket Webhook
The final step is to configure the Bitbucket webhook. In this step, you will set up Bitbucket to send notification (JSON payload) to your AWS Lambda function API gateway URL. Notifications will be sent only if the trigger specified by you will get triggered. The script that we provided you with will read the JSON payload and trigger new pipelines on project branches that have PR to the branch that just triggered the "PR merged" trigger.

* Go to your Bitbucket repository.
* Click on "Repository settings".
* Click on "Webhooks" in the menu.
* Click on "Add webhook", type in the title and URL you got from the HTTP trigger (API Gateway).
* Under Triggers, select "Pull request merged".
* Click on the "Save" button.

After you are done, it should look similar to this, except for the URL and name you chose for the webhook.
![Bitbucket webhook setup](/img/blog/mitigate-django-migration-conflicts/bitbucket-webhook-setup.png)

## CircleCI config - Django migration check
The last step required is to add a migration check to your CircleCI config.yml. An example of how you can implement this:
```
version: 2
jobs:
  # Some jobs before

  migration_check:
    docker:
      - image: cimg/python:3.7
      - image: circleci/postgres:11.5-alpine-ram
    steps:
      - checkout
      - run:
          name: Merge dev branch to PR branch
          command: |
            git config --global user.email "dummy@mail.com"
            git config --global user.name "dummy_user"
            git merge --no-commit dev
      - restore_cache:
          keys:
            - dependencies-{{ checksum "your-lock-file" }}
            - dependencies-
      - run:
          name: Check for conflicting migrations
          command: poetry run python src/manage.py migrate

  # Some jobs after

workflows:
  version: 2
  your_project:
    jobs:
      # Some jobs before

      - migration_check:
          requires:
            - build
          filters:
            branches:
              ignore:
                - master
                - staging
                - dev

      # Some jobs after
```

### Explanation
In the example CircleCI config.yml above, we set up a job called **migration_check**, this job checks not only Django migrations but also git merge conflicts as a bonus. The thing is, if you remember the scenario in the motivation section, your colleague has pushed the changes to the dev branch. Naturally, you have to pull these changes to your branch and resolve the conflicts if any occur. This job will do that for you and in other words, also check if you have not by accident forgot to pull the changes. If you forgot, then this job will fail and such a result is to be expected. Once it has pulled the changes, it will restore your project dependencies from a cache, if you have any (it is recommended by CircleCI to save dependencies from a build job to a cache). After it has the dependencies loaded, it will run the Django command for migration. Django command will fail only in case there are some Django migration conflicts.  

## Conclusion
There you have it. With this infrastructure, you can find and resolve git merge conflicts and Django migration conflicts and push clean into the dev/main branch. This setup might look like a lot of work, but it is worth the time. Imagine you have 7 PRs to the dev/main branch and your lead developer will start to merge them. The first merge might pass, but it could cause the other 6 PRs to introduce conflicts to your branch. The lead developer will have to roll back the merge, give you notice, and order you to resolve the conflicts, push changes again and then try to merge. This infrastructure saves you time since no rollback will be needed and you instantly know if there are conflicts. Also now you have the template on how to create Bitbucket webhooks and Bitbucket offers a lot more triggers than just "pull request merged". You can tweak this to your needs and enjoy the benefits.


Leverage your CI/CD and AWS Lambda to mitigate the risk of propagating hidden Django migration conflicts into main/dev branches.

Mitigating Django migration conflicts


Whenever you are generating a lot of data, a natural question comes up. How to process the data? Do you need to set up a whole infrastructure to create a new full-fledged database? Perhaps, you do not need to actively work with the data, you just need to occasionally access them and get the necessary information. AWS Athena provides a solution for the latter case.

## AWS Athena - Analytics tool

As the headline suggests, AWS Athena is a part of AWS analytic tool-set. This tool can help you create ad hoc databases or tables and run queries on them. All it needs is data in form of CSV, JSON, or some columnar formats such as Apache Parquet or ORC.

## Athena Use-Case
Query services like Amazon Athena make it easy to run interactive queries against data directly in Amazon S3 without worrying about formatting data or managing infrastructure. For example, Athena is great if you just need to run a quick query on logs to help you debug an application. You just define a table for your data and start querying using standard SQL.

It is not advisable to use Athena for anything related to production, business, or enterprise level. For these cases, there are dedicated services such as Amazon Redshift or EMR.

## Getting data to AWS Athena

#### Adding data source
In short, AWS Glue can extract, transform and load data. It can be used to create a crawler that can infer database and table schema from your data in Amazon S3. Inferred schema is then stored in AWS Glue Data catalog. Athena can access this catalog and load databases and tables and run queries on them. AWS Glue is already integrated into AWS Athena. Tutorial on how to add a data source to Athena can be found [here](https://docs.aws.amazon.com/athena/latest/ug/data-sources-glue.html "here").

#### AWS service integration with Athena
Athena can fetch data from other AWS services thanks to the integration. For example, if you have running AWS CloudTrail, you can create a table from CloudTrail -> Event History, with just one click. Afterward, you can run queries on logs from CloudTrail in the Athena query editor. Some other useful Athena integrations are:
* AWS CloudFormation
* Amazon CloudFront
* Identity and Access Management (IAM)
* Amazon S3 Inventory

You can read the full list of integrations [here](https://docs.aws.amazon.com/athena/latest/ug/athena-aws-service-integrations.html "here").

## Using Athena query editor
![Athena Query Editor](/img/blog/athena/athena_query_editor.png)  
The interface is quite self-explanatory. On the left side, you can select Data source and Database and see what tables are present. On the right side, there is a text editor for SQL queries and buttons to run them or save them. After you run a query, query results will show up under the text editor. Some tips:
* Save your query using "Save as". After that, any change will be saved automatically.
* In the History tab (at the top), you can download query results in CSV file format.
* Once you load data to Athena, it will be always updated. Let's say you are logging some data to S3 bucket and you load that data into Athena. Those are logs we are talking about, they keep coming all the time, right? Common sense is that Athena would show the newest data from logs from the date & time you loaded the data in. This common sense is false in this case. The data is always up to date.

#### Example query: CloudTrail user activity
CloudTrail is an AWS service responsible for logging user activity. Logs are stored in S3 bucket. You can view user activity logs via CloudTrail -> Event History. It has one downside though - Event History can view only the last 90 days of logs. One way of viewing all logs is using AWS Athena:
1. Go to CloudTrail -> Event History.
2. Click on "Create Athena table" to create a DB table out of the logs.
3. Select a bucket where CloudTrail logs are stored.
4. Click on "Create table".
5. Go to AWS Athena.
6. Select the table with CloudTrail logs.

Provided that you arrived at point 6 above, we can start writing a query. Let's see what services have been user **intern_user**:
```
SELECT DISTINCT eventsource
FROM cloudtrail_logs_aws_dl_tutorial_bucket
WHERE useridentity.username LIKE '%intern_user%'
```

We can now run the query and see results under the query editor:  
![Example Query](/img/blog/athena/athena_example_query.png)

## Athena pricing
 Payment is per query. More precisely you are charged based on the amount of data scanned by each query. Pricing may differ per region. Examples:
 * N. Virginia = $5.00 per TB of data scanned
 * Cape Town = $6.00 per TB of data scanned

#### Savings
It's all about the size of data Athena has to read, therefore if you manage to reduce the size of your file, you can make some savings. If you compress files using GZIP, you might see up to 3:1 compression gains.  

It can be also beneficial to load the data in more efficient formats. Columnar formats like Apache Parquet allow more efficient querying. Using such formats allows Athena to read-only columns specified by a query, therefore reducing the amount of data read.


AWS has many hidden gems in the service catalog. AWS Athena is one of them. We will take a look at what the Athena is and what it is capable of.

AWS Athena Analytics Tool


In this tutorial, we are going to learn how to push application logs from our Django application to Elasticsearch storage and have the ability to display it in a readable way in Kibana web tool. The main aim of this article is to establish a connection between our Django server and ELK stack (Elasticsearch, Kibana, Logstash) using another tool provided by Elastic - Filebeat. We will also briefly cover all preceding steps, such as the reasoning behind logging, configuring logging in Django and installing ELK stack.

## Why and how to log
Having reasonable logging messages in the production helped me discover several non-trivial bugs that would otherwise be undiscoverable. It is also a good practice to use logging messages in the local environment to speed up the development, enabling these messages to stay there for production use. 

In order to start logging, just add the following lines at the top of your file. The variable \_\_name\_\_ will be translated into the name of the module that will also appear in the final log messages.

```python
import logging

logger = logging.getLogger(__name__)
```

### Logging levels

Another part of the log structure is the log level. Log level helps us identify the severity of the message and makes it easier to navigate in the log output. There are five levels that can be used for log messages. My usage of the logging levels is as follows:

- DEBUG - messages that can help us track the flow of the algorithm, but are not important for anything else than troubleshooting. For example, 'Task has started', 'Task has ended in 5.4 seconds', 'Email for user id=55 was sent'
- INFO - messages that carry important information, e.g. 'Payment transaction finished with status={status}', 'Email with invoice id={invoice_id} was sent to user id={user_id}'
- WARNING - messages that could help us identify potential errors like 'Referer {ip} is not in allow list'
- ERROR - 'Got error message={message} while downloading file={file} from external API'
- CRITICAL - higher than error, I do not use it

### Creating log messages
In order to create a logging message, we need to call a method of the logger object. There are corresponding names to the logging levels.

One addition to those is an exception that corresponds to the error level but will also add a traceback, which is very handy, especially with having an integrated error monitoring tool like Sentry.
```python
# DEBUG
logger.debug('Email to user id={} sent'.format(user.id))

# INFO
logger.info('Payment transaction finished with status={}'.format(payment.status))

# WARNING
logger.warning('Referer {} is not in allow list'.format(ip)

# ERROR
logger.error(
    'Got status code={}, error message={} while sending request to external API'.format(
        response.status_code, response.text
    )
)

# EXCEPTION
try:
    print(b)
except:
    logger.exception('Exception happened.')
```

### Setup logging in your Django application
Now, we need to set up the output of our log. The most common use cases are output on the screen (e.g. when you run manage.py runserver) or output to the log file. We want to go one step further and get the data into Elasticsearch storage in order to have a user-friendly web UI (Kibana) that allows easy search and filter options to access the logs.

When I did the first research, I found a couple of examples of integration with python-logstash and python-json-formatter, but it eventually did not work with python3. It also felt like a lot of hassle to get things going when there is an option to use a tool called [Filebeat](https://www.elastic.co/products/beats/filebeat "Filebeat")  by Elastic. Filebeat monitors changes in the log file and sends all new records to the Elasticsearch storage through a parser called Logstash. Unfortunately, there is one condition that requires us to have full control over our instance(s), which is used to serve our Django application. If not, we need to find a way of using a service to stream logs to our ELK server or fall back to python-logstash. 

Since we are going to use Filebeat, all we need to achieve is storing the data in a reasonable format in log rotated files. Besides that, I recommend using [Sentry](https://sentry.io/welcome/ "Sentry") for managing our errors. We will set up our Django logger config to send error and exception log messages to Sentry.

This is a sample setup of logging in settings.py. For more information, you can also refer to [Django documentation](https://docs.djangoproject.com/en/dev/topics/logging/ "Django documentation"). 

```python
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'simple': {
            'format': '[%(asctime)s] %(levelname)s|%(name)s|%(message)s',
            'datefmt': '%Y-%m-%d %H:%M:%S',
        },
    },
    'handlers': {
        'applogfile': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': '/webapps/myproject/logs/django/myproject.log',
            'maxBytes': 1024*1024*15,  # 15MB
            'backupCount': 10,
            'formatter': 'simple',
        },
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
            'formatter': 'simple'
        }
    },
    'loggers': {
        'app1': {
            'handlers': ['applogfile', 'console'],
            'level': 'DEBUG',
        },
        'app2': {
            'handlers': ['applogfile', 'console'],
            'level': 'DEBUG',
        }
    }
}
```

The log format is composed to be easily parsable by Logstash. The following messages will start appearing if everything is set correctly:

```python
[2019-11-01 02:59:02] DEBUG|users.forms|Captcha valid for user_id=443
[2019-11-01 09:53:48] ERROR|jobs.views|Errror while redirecting the job=3232
[2019-11-01 10:20:31] INFO|newsletter.services.newsletter_service|Newsletter 55351 added keyword.
[2019-11-01 12:42:09] INFO|newsletter.services.newsletter_service|Keywords successfully loaded for newsletter 60473.
```

## ELK + Filebeat infrastructure
I have already mentioned these terms a couple of times. What we need is the ELK infrastructure - Elasticsearch for storing data, Logstash for parsing and streaming data from our log files to Elasticsearch, and Kibana for data visualization. We can either choose a cloud-based solution 
(e.g. [Elastic](https://www.elastic.co/products/elasticsearch/service/pricing "Elastic"), [logz.io](https://logz.io/pricing/ "logz.io"), [AWS Elasticsearch](https://aws.amazon.com/elasticsearch-service/ "AWS Elasticsearch") or go with the self-hosted solution. It all depends on the use case, for a hobby project or if you have DevOps capacity, I would go for a self-hosted solution. In the case of a commercial project, a cloud-based solution is a better choice. Elastic is progressing really fast with its releases, prepare yourself for a lot of maintenance work.

For the purpose of this article, I have prepared a self-hosted solution that is more demanding to install but will give you more detail on how things work inside the ELK infrastructure. Furthermore, this ELK server is further used for our micro startup called [Working Nomads](https://www.workingnomads.co/jobs "Working Nomads"), so it does not mean all energy was wasted for nothing.

![ELK schema {666x281}](/img/blog/elk/schema.png)

### Installing ELK server

There is a $20 droplet with 4gb memory used in this example which is a minimum requirement for a server that has all the components of ELK installed. Your requirements may be much more demanding and you will need a whole cluster of several instances running a single component, but in this case, we are going to demonstrate the solution on this bare minimum server.

I am not going to go into details with the installation, I will just share the tutorials I have used and give a couple of hints to overcome the issues with which I struggled.

Install ELK on Ubuntu 18.04 based on this [example](https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elastic-stack-on-ubuntu-18-04 "example"). I have used Java 11, and instead of the ELK 6.x version, I have used ELK 7.4 version.
Make sure you [install certificates for your Kibana instance](https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-18-04 "install certificates for your Kibana instance").
Now, you should be able to access the Kibana dashboard, we will configure it later.

### Setting up Logstash

Now it is time to feed our Elasticsearch with data. To achieve that, we need to configure Filebeat to stream logs to Logstash and Logstash to parse and store processed logs in JSON format in Elasticsearch. At this moment, we will keep the connection between Filebeat and Logstash unsecured to make the troubleshooting easier. Later in this article, we will secure the connection with SSL certificates.

We need to configure the port which will be used by Logstash to listen to Filebeat. Furthermore, we need to configure a grok regex to convert the log message to a structured JSON. A new index file will be created every day.

Sample configuration in /etc/logstash/conf.d/logstash.conf

```bash
input {
  beats {
    port => 5044
    ssl => false
  }
}

filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}] %{LOGLEVEL:loglevel}\|%{GREEDYDATA:module}\|%{GREEDYDATA:content}" }
  }
  date {
    locale => "en"
    match => [ "timestamp", "YYYY-MM-dd HH:mm:ss"]
    target => "@timestamp"
    timezone => "America/New_York"
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" 
  }
}
```

Now, restart Logstash and ensure the Filebeat input is working by checking the Logstash logs.

If you need to match a different pattern with grok regex, I recommend using [Grok debugger](https://grokdebug.herokuapp.com/ "Grok debugger") to find out what you actually need.

### Setting up Filebeat
Now, we need to log in to the instance that serves the Django application to [install](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html "install") and configure Filebeat to monitor the log files and stream them to Logstash. Based on the previous part of this tutorial, there is a log being generated and rotated in /webapps/myproject/logs/django/. You will see files like these in the mentioned directory:

```bash
myproject.log
myproject.log.1
myproject.log.2
```

Filebeat is capable to manage these rotated log files out of the box so no extra work is needed.

Configuration in /etc/filebeat/filebeat.yml

```bash
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /webapps/myproject/logs/django/*

output.logstash:
  hosts: ["ip_or_hostname_of_your_ELK:5044"]
```

You can now run Filebeat in a debug mode with this command:
```bash
filebeat -e -d "*"
```

Ideally, what we are aiming for is ensuring that Filebeat is sending the log messages to Logstash, such as:
```bash
Publish event: {
  "@timestamp": "2019-11-14T12:36:15.340Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.4.2"
  },
  "ecs": {
    "version": "1.1.0"
  },
  "host": {
    "name": "django-host"
  },
  "agent": {
    "hostname": "django-host",
    "id": "e992e850-bedd-48b2-a885-31c5a853e34e",
    "version": "7.4.2",
    "type": "filebeat",
    "ephemeral_id": "bd808730-a3a2-452b-a2ea-4ee6bbd55c7f"
  },
  "message": "[2019-11-14 17:36:11] DEBUG|app1.utils|Sample debug message",
  "log": {
    "offset": 2177175,
    "file": {
      "path": "/webapps/myproject/logs/django/myproject.log"
    }
  },
  "input": {
    "type": "log"
  }
}
```
If the push from Filebeat to Logstash is successful, we can turn off the command and run it as a service. Make sure it runs at startup after the machine is rebooted.

### Data retention
Based on our use case, we should set the time period for which the logs are kept. Elastic comes with another tool called Curator. 
Follow this [tutorial](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/installation.html "tutorial") to install it, for a newer version of Elasticsearch you need to install it via pip, otherwise, 
Curator will not be compatible with Elasticsearch. Here is the [configuration](http://www.elastic.co/guide/en/elasticsearch/client/curator/current/ex_delete_indices.html "configuration") of the action file which deletes all indices older than 45 days. 
It works right away, you only need to add the [configuration file](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/configfile.html "configuration file") to /home/user/.elasticsearch/ and change the disable_action flag to False.

You can test your configuration with a dry run. 

```bash
curator --dry-run curator_action.yml
```

And then run it daily with cron job.

```bash
0 1 * * * user /usr/local/bin/curator /home/user/.curator/curator_action.yml  >> /var/log/curator.log 2>&1
```

### Setting up Kibana
Now it is time to look into Kibana and see if the data is there. At this point, Kibana will probably offer you a way to configure your index pattern, if not, navigate to Settings > Kibana > Index Patterns and add index pattern "filebeat-*".

![Kibana pattern {920x641}](/img/blog/elk/kibana_1.jpeg)

In the second step select @timestamp as Time filter field.

![Kibana filter field {920x616}](/img/blog/elk/kibana_2.jpeg)

Now go to the Discover area of Kibana, select an appropriate time range and finally, you should see the logs coming in. Select the fields you have parsed through the grok regex - loglevel, module, and content and you will see the log feed.

![Kibana log feed {920x842}](/img/blog/elk/kibana_3.jpeg)

This is a bare minimum of the configuration, we are going to add some handy features and encrypt the communication between Kibana and Filebeat.

### Support for multiline logs
When the exception occurs, we will get a multiline log message with the traceback. We are using the square bracket as the beginning of our log messages and Filebeat can be configured to recognize that. See the enhanced configuration file.

```bash
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /webapps/myproject/logs/django/*
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after

output.logstash:
  hosts: ["ip_or_hostname_of_your_ELK:5044"]
```

Multiline messages like these will start to pop up in your logs.

![ELK multiline log messages {920x118}](/img/blog/elk/kibana_9.jpeg)

For more information about multiline examples, you can consult the official [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html "documentation").

### Securing the connection between Filebeat and Logstash
Currently, the connection between Filebeat and Logstash is unsecured which means logs are being sent unencrypted. It is heavily recommended to set up SSL certificates to make the connection secure and also to ensure that Logstash will only accept data from trusted Filebeat instances. 
To set this up, follow these tutorials: [#1](https://benjaminknofe.com/blog/2018/07/08/logstash-authentication-with-ssl-certificates/ "#1") and [#2](https://github.com/Busindre/How-to-configure-SSL-for-FileBeat-and-Logstash-step-by-step/blob/master/README.md "#2"). 
You can also consult the official [article](https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ssl-logstash.html "article"), but it does not say how to generate the certificates.

The only pitfall I experienced following those tutorials was setting the ownership of the server.key to the Logstash user.

These are the final configuration files for Filebeat and Logstash:

```bash
input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate_authorities => ["/etc/logstash/ca.crt"]
    ssl_certificate => "/etc/logstash/server.crt"
    ssl_key => "/etc/logstash/server.key"
    ssl_verify_mode => "force_peer"
  }
}

filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}] %{LOGLEVEL:loglevel}\|%{GREEDYDATA:module}\|%{GREEDYDATA:content}" }
  }
  date {
    locale => "en"
    match => [ "timestamp", "YYYY-MM-dd HH:mm:ss"]
    target => "@timestamp"
    timezone => "America/New_York"
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" 
  }
}
```

```bash
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /webapps/yourproject/logs/django/*

  ### Multiline options
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["ip_or_hostname_of_your_ELK:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["/etc/filebeat/ca.crt"]

  # Certificate for SSL client authentication
  ssl.certificate: "/etc/filebeat/client.crt"

  # Client Certificate Key
  ssl.key: "/etc/filebeat/client.key"
```

## Further steps
### Setting up Kibana searches
Kibana offers features to make the interface simple to use. First of all, we can create searches that can be easily accessed. Imagine we have multiple instances reporting to our ELK server, we can configure the search based on filtering host.name and loglevel and save it. 

![Kibana search {506x401}](/img/blog/elk/kibana_4.jpeg)

![Kibana search {399x313}](/img/blog/elk/kibana_5.jpeg)

We can create a list of searches, it is possible to add more filtering parameters based on your preference. We can then quickly select the search we need.

![Kibana search {288x288}](/img/blog/elk/kibana_6.jpeg)

### Visualizations
We can create visualizations based on our log messages. Since we have set up data retention, the log messages are not going to stay forever, it is thus not suitable for analytics. Though, it can help us quickly see a drop in a certain metric that will indicate that some functionality stopped working.

![Kibana visualization {920x1127}](/img/blog/elk/kibana_7.png)

### Dashboard
We can put multiple visualizations together and get simple monitoring of our systems.

![Kibana dashboard {920x582}](/img/blog/elk/kibana_8.png)


In this tutorial, we are going to learn how to push application logs from our Django application to Elasticsearch storage and display the messages in Kibana.

Written by Remastr, experienced developers creating Django & React applications.

Our Blog

Mitigating Django migration conflicts

AWS Athena Analytics Tool

Django centralised logging using Elasticsearch, Logstash, Kibana (ELK) + Filebeat

We're here to help.