
Django migration conflicts are common when developing a Django application. In this blog, we will leverage AWS Lambda, Bitbucket webhooks, and CircleCI workflows to prevent the introduction of Django migration conflict to the dev/main branch.

## Infrastructure requirements

Before you read any further, be aware that this blog post assumes that you have these resources available at your disposal:
* Webhook service
* VCS
* CI/CD

For this blog, we will use the following infrastructure:
* AWS Lambda as webhook service
* Bitbucket for VCS
* CircleCI as CI/CD.  

You can use other solutions for webhook service, VCS, and CI/CD. The idea is the same, but you will have to figure out how to make this work on your infrastructure. If you use the same infrastructure, be sure that you have sufficient permissions for API Gateway and Lambda AWS services. You will also need administrator rights to your project's Bitbucket repository.

## Motivation

Since you are reading this article, you may as well already know what Django migration conflict is. Still, we need to set up a common ground before we move on. Assuming that you work in a company that develops Django apps, you can easily get Django migration conflict. All it requires is for you and your colleague to edit the same Django model or create a migration file with the same order number at the same time. You both work on some features and each of you creates a separate branch for your implementation. Your colleague managed to get his branch merged into the main branch sooner than you and had his migrations applied. In such a case, you have to pull the newest changes and resolve conflicts, if there are any. However, it is not that hard for humans to forget, right? If you forget to pull your colleague's changes before you merge your branch into the main branch, you will introduce Django migration conflict into the main branch. Someone will have to pull the main branch, fix it and push the changes, or undo your merge to the main branch. Especially in agile and incremental development, where there are several pull requests merged on daily basis, it can become very tedious to fix forgotten Django migration conflicts in the main branch, especially if they can be prevented.   

## Prevention

As the name of this blog post suggests, we can use CI/CD. Let us go back to the scenario with you and your colleague. For this prevention to work, you need to have set up CI/CD workflows for your branches. Let us assume that you have different workflows set up for different types of branches. It is logical to have a different workflow for the feature/bugfix branch and dev/main branch. With CI/CD in place, we can add Django migration conflict check to feature/bugfix workflow. This way, you and your colleagues can be aware of possible conflicts before propagating them further and simply fix them on your branch. Since these conflict checks will be done on feature/bugfix branches, your main/dev branches will not require these checks.

## Prevention flow
![Flow](https://github.com/remastr/django-migration-conflict-check/raw/main/docs/img/flow.png "Flow diagram" 100x20)

At the first stage, a developer does an action (commit, push, merge, ...) in VCS. This VCS action triggers an event, which is sent to a Webhook service. Webhook service provides you with the means to process an event and send a response. In our case, we will respond to CI/CD associated with our VCS and trigger an action (rebuild project, trigger new pipeline, ...).

## Setup
The setup procedure involves several steps. Please follow them strictly in this order:
* Create API Keys.
* Set up your <Webhook-Service> + <CI/CD> pair.
* Create Bitbucket webhook.
* CircleCI config - Django migration check.

### Create API keys
##### **Create Bitbucket App Password:**
Bitbucket App Password is your API key. With this key, you can interact with your Bitbucket. At the time of creation, you have to assign permissions to the key. Permissions cannot be modified afterward. This key does not require 2-step verification, so be careful when sharing this key. Lastly, please note that you can see the content of the key only at the time of creation.

* Sign in to the Bitbucket Cloud.
* Go to your "Personal settings".
* Under the "ACCESS MANAGEMENT" section, go to "App passwords" and click there on the "Create app password" button.
* Select permissions according to your use case and click on "Create".
* Save your App password in a safe place. You will need it later on.  


##### **Create CircleCI Personal API Token:**
As with the Bitbucket App Password, this is also your API key and you can see the context only when creating the key. However, in this case, you cannot assign any permissions. This key has by default full access rights, so you have to be extremely careful with whom you share this key.

* Sign in to the CircleCI.
* Go to your "User settings".
* Click on "Personal API Tokens" and there click on the "Create New Token" button.
* Name accordingly your token and hit "Add API Token".
* Save your API token in a safe place.

### AWS Lambda + CircleCI
We will use AWS Lambda to run our webhook processing script in a container. Running code in a container without any server is called serverless computing. The first sentence from Wiki explains it perfectly - "Serverless computing is a cloud computing execution model in which the cloud provider allocates machine resources on-demand, taking care of the servers on behalf of their customers." This container will be reachable from the Internet thanks to the HTTP trigger that we are going to set up. HTTP trigger is done via API Gateway. API Gateway is a service used for creating API. It will provide us with a web address. We can configure these web address URL paths and what code to execute on each of them. Since our webhook has only one use case, the root web address will do. Lastly, we need to upload the source code of our webhook processing script into the AWS Lambda function we have created. At our Github repository, we already have a script available that you can use. All that's left is to set up AWS Lambda function environment variables. The script will need your Bitbucket username, API keys you have created, and lastly, a slug of the project on which you want to Django migration check.

##### **Create a function:**  
* Sign in to the AWS Management console.
* Go to Lambda service.
* Click on the "Create function" button.
* Select the "Author from scratch" option.
* Name your function.
* Select Python3.8 runtime.
* Click on the "Create function" button.  


##### **Add HTTP trigger:**  
* On the function page, click on the "Add trigger" button.
* Select "API Gateway" from the dropdown menu.
* Next, from the second dropdown menu, choose "Create an API".
* Select "HTTP API" API type.
* Select "Open" security from the third dropdown menu.
* Click on the orange "Add" button.  


##### **Upload code:**  
* Download the latest zip package from [ releases](https://github.com/remastr/django-migration-conflict-check/releases " releases").
* On the function page, under "Code" section, click on "Upload from" button and choose ".zip file" option.
* Upload the "aws_deployment.zip" you have downloaded in the first step.  


##### **Set Lambda Environment variables:**  
* On the function page, under "Configuration" section, click on "Environment variables".
* Click on the "Edit" button and add following variables with your values:
```
BITBUCKET_API_URL	https://api.bitbucket.org
BITBUCKET_APP_PASSWORD	<Your-Bitbucket-App-Password>
BITBUCKET_USERNAME	<Your-Bitbucket-Username>
CIRCLECI_API_TOKEN	<Your-Personal-CircleCI-API-Token>
CIRCLECI_PROJECT_SLUG   bb/<Your-CircleCI-Organization-Name>/<CircleCI-Project-Name>
```
* Click on the "Save" button.

### Create Bitbucket Webhook
The final step is to configure the Bitbucket webhook. In this step, you will set up Bitbucket to send notification (JSON payload) to your AWS Lambda function API gateway URL. Notifications will be sent only if the trigger specified by you will get triggered. The script that we provided you with will read the JSON payload and trigger new pipelines on project branches that have PR to the branch that just triggered the "PR merged" trigger.

* Go to your Bitbucket repository.
* Click on "Repository settings".
* Click on "Webhooks" in the menu.
* Click on "Add webhook", type in the title and URL you got from the HTTP trigger (API Gateway).
* Under Triggers, select "Pull request merged".
* Click on the "Save" button.

After you are done, it should look similar to this, except for the URL and name you chose for the webhook.
![Bitbucket webhook setup](/img/blog/mitigate-django-migration-conflicts/bitbucket-webhook-setup.png)

## CircleCI config - Django migration check
The last step required is to add a migration check to your CircleCI config.yml. An example of how you can implement this:
```
version: 2
jobs:
  # Some jobs before

  migration_check:
    docker:
      - image: cimg/python:3.7
      - image: circleci/postgres:11.5-alpine-ram
    steps:
      - checkout
      - run:
          name: Merge dev branch to PR branch
          command: |
            git config --global user.email "dummy@mail.com"
            git config --global user.name "dummy_user"
            git merge --no-commit dev
      - restore_cache:
          keys:
            - dependencies-{{ checksum "your-lock-file" }}
            - dependencies-
      - run:
          name: Check for conflicting migrations
          command: poetry run python src/manage.py migrate

  # Some jobs after

workflows:
  version: 2
  your_project:
    jobs:
      # Some jobs before

      - migration_check:
          requires:
            - build
          filters:
            branches:
              ignore:
                - master
                - staging
                - dev

      # Some jobs after
```

### Explanation
In the example CircleCI config.yml above, we set up a job called **migration_check**, this job checks not only Django migrations but also git merge conflicts as a bonus. The thing is, if you remember the scenario in the motivation section, your colleague has pushed the changes to the dev branch. Naturally, you have to pull these changes to your branch and resolve the conflicts if any occur. This job will do that for you and in other words, also check if you have not by accident forgot to pull the changes. If you forgot, then this job will fail and such a result is to be expected. Once it has pulled the changes, it will restore your project dependencies from a cache, if you have any (it is recommended by CircleCI to save dependencies from a build job to a cache). After it has the dependencies loaded, it will run the Django command for migration. Django command will fail only in case there are some Django migration conflicts.  

## Conclusion
There you have it. With this infrastructure, you can find and resolve git merge conflicts and Django migration conflicts and push clean into the dev/main branch. This setup might look like a lot of work, but it is worth the time. Imagine you have 7 PRs to the dev/main branch and your lead developer will start to merge them. The first merge might pass, but it could cause the other 6 PRs to introduce conflicts to your branch. The lead developer will have to roll back the merge, give you notice, and order you to resolve the conflicts, push changes again and then try to merge. This infrastructure saves you time since no rollback will be needed and you instantly know if there are conflicts. Also now you have the template on how to create Bitbucket webhooks and Bitbucket offers a lot more triggers than just "pull request merged". You can tweak this to your needs and enjoy the benefits.


Leverage your CI/CD and AWS Lambda to mitigate the risk of propagating hidden Django migration conflicts into main/dev branches.

Mitigating Django migration conflicts


Starting a new project is always exciting because something new and fresh is starting but to maintain this motivation in the long term future, proper tooling has to be set up. The right selection of tools will help developers successfully react to new challenges in software developer life and keep the codebase in proper shape.

When the new project is being configured there are two crucial technologies implemented right at the beginning - Docker, and Poetry. The main reason why Docker is used is to have the same development environment across several developers using the different operating systems. With Docker, project onboarding for new developers is simplified to the bare minimum. For Poetry it is to avoid future dependency hell and have a clear list of used packages.

## Prerequisites

To complete this tutorial, the following items are required:

- Installed Docker on the local machine and some experience writing Dockerfiles + docker-compose
- Django experience with the setting of the new project via **django-admin startproject project_name**
- Basic knowledge of Poetry for dependency management

## Starting with Docker configuration

The main plan is to create the basic service in Docker which will be based on the Python image. In the running container, the Django application is going to be configured. As more and more settings and tools are going to be added, project settings are going to be adjusted (Dockerfile config for Django application).

In the beginning, docker-compose.yml configuration would be needed. Just for quick reference, docker-compose.yml is using to manage multiple Docker containers. At this point, docker compose will be just used to manage the Django application (we can add Postgres later).

```yaml
version: "3"

services:
  app:
    image: python:3
    tty: true
    working_dir: /usr/src/app
    volumes:
      - .:/usr/src/app
```

The configuration above is self-explanatory. It is saying which services are going to be run (app) and specifies service attributes. In this case, only the simple Python container is going to be used without any specific functionality. Once the container is started by using **docker-compose up** command, it is possible to connect inside the container by running **docker-compose exec app bash**.

## Installing dependencies and starting new Django project

When the container is running, the same basic project setup is needed. First of all, Poetry installation and configuration must be done. When initializing the Poetry project via **poetry init** command, some basic questions are asked. After the initialization virtual environment is going to be created. After that, it is possible to start installing dependencies, in this case, it is going to be Django.

```bash
# Install Poetry from Pypi repository
pip3 install poetry

# Set up Poetry project. Some additional questions are asked.
poetry init

# Install new dependecy into Poetry project
poetry add django
```

After running the commands above, the Docker container has properly configured the Poetry project with the installed virtual environment. Now it is time to set up the Django project. As a best practice, it is considered to put project files into **src** directory where all codebase is going to be located. In this directory run **django-admin startproject project** to bootstrap the new project but before this is done, it is needed to load the virtual environment which was created by Poetry. To enable this virtual environment run **poetry shell**.

```bash
# Our working directory
cd /usr/src/app

# Directory where all codebase is going to be located
mkdir src
cd src

# Load virtual environment created by Poetry
poetry shell

# Start new Django project
django-admin startproject project

# Run the project - available on 0.0.0.0:8000
python manage.py runserver
```

## Creating Dockerfile

The steps which were done till this moment manually, need to be put into Dockerfile which is specifying the build process. Developers are not expected to make such a configuration manually each time they are setting up the project but it must be done automatically.

The **Dockerfile** specifies all steps needed to setup the project. To start with Dockerfile only the Dockerfile file must be created in the root directory of the project on the same level as docker-compose.yml. In the general, it describes the same steps as the manual process above but with slight differences. **WORKDIR** is configuring the "home" directory inside the Docker container in which all upcoming actions are going to be executed.

In the build process, the Docker container does not have the access to the directory structure on the local machine via the mount points so some basic configuration files must be copied from the local machine into the container by using **COPY** command. In this case, the Poetry configuration files are copied and in the following actions, they are used by Poetry to set up the virtual environment.

```dockerfile
FROM python:3

ENV PYTHONUNBUFFERED 1

WORKDIR /usr/src/app

COPY poetry.lock pyproject.toml /usr/src/app/

RUN pip3 install poetry

RUN poetry install
```

Once the Dockerfile is created, it is needed to adjust docker-compose.yml to reflect Dockerfile existence in the project. From the previous docker-compose.yml, two lines of code must be changed. By removing **image** parameter and replacing it with **build: .**, it will be communicated to use the new Dockerfile to specify the build process, and **command: "poetry run python src/manage.py runserver 0.0.0.0:8000"** will tell the container which command to run.

```yaml
version: "3"

services:
  app:
    command: "poetry run python src/manage.py runserver 0.0.0.0:8000"
    build: .
    volumes:
      - .:/usr/src/app
    ports:
      - "8000:8000"
```

After saving the new docker-compose.yml, all required changes have been made. Now by running **docker-compose up**, the new container is going to be built with the Django project using Poetry as the dependency management tool. After the successful build, it is possible to access the server on this URL **http://0.0.0.0:8000/**.


In this article, it will be explained how to set up Django with poetry in a docker container.

Setting up Django project with Poetry and Docker


The following article introduces our Django package that is useful for
improving the development experience when working with Django REST framework
and frontend application which is using TypeScript. Continue reading and
learn more about it.

## The 'Full stack' problem

In modern web development, there is a trend to create reactive, fast websites
which offer an amazing experience for the users.
A widely used pattern to achieve this in Django projects is to have a frontend application written in 
Javascript framework or library - there are quite a few to choose from - and connect it to the backend
using [Django REST framework (DRF)](https://www.django-rest-framework.org/ "Django REST framework (DRF)").

To improve the development experience and produce high-quality code developers often opt to use TypeScript
instead of Javascript. It's important to know the format of the data which some endpoint expects or returns -
of course for this you can just look at the API documentation, but there is a **huge** improvement in the
development experience when you have these expected *responses* or *requests* typed in your frontend application.
It improves correct code completion and linting, which eliminates runtime errors in a significant way.

Since TypeScript and Django Serializers cannot share the definition of the format of the API data,
it can get annoying to change it both in backend and frontend every time there is a change made
to the endpoint - especially when you are a full stack developer working on both ends.
Or even type it on the frontend correctly in the first place. Now there is an easy
way to do it with [**drf-typescript-generator**](https://github.com/remastr/drf-typescript-generator "drf-typescript-generator") package.

## How it works

The package provides a management command for generating TypeScript types 
from serializers found in the Django apps you pass to it as arguments. You can also 
choose to generate interfaces instead if it fits more to your coding style.

The process of finding serializers follows general Django practices, but for example names of files
in which serializers should be put is not strictly defined. Someone uses *serializers&#46;py*,
someone *api&#46;py* and someone puts serializers directly into *views&#46;py*. It's personal preference
and the package takes it into account. With that said, the process of finding serializers can be 
divided into the following steps:

1. The process starts with the DRF routers which are found in the project urls&#46;py file as
well as urls&#46;py files of chosen apps.
2. Registered Viewsets are extracted from the routers and used to identify modules that contain them.
3. All serializers found in those modules (serializers are either defined in them or imported from other modules)
are considered as being relevant and are then exported.

This approach prevents generating output for unnecessary apps and serializers that you do not need in your frontend
application and at the same time, you don't need to make changes in your backend code to mark relevant serializers.

## Generating TypeScript

Classes in Python, as well as types in TypeScript, use a naming convention known
as *PascalCase*, so in the generated output the name of the serializer stays also as
the name of the type.
For the properties Python usually uses *snake_case* and TypeScript *camelCase*
therefore the names of serializer fields are transformed - e.g. `house_number` will become `houseNumber`.
Unfortunately, Django REST framework does not accept camel case in request data out of the box so you
would need to map the fields to snake case before sending a request or use a package such as 
[djangorestframework-camel-case](https://github.com/vbabiy/djangorestframework-camel-case "djangorestframework-camel-case") to provide
this functionality at the backend.

### Serializers to types

In addition to the classic DRF `Serializer` which is commonly used, the package also works well with `ModelSerializer`
and with nested serializers. In the case of nested serializers, the TypeScript equivalent is generated even when
the nested serializer referred to in the main serializer is not directly used in views modules and so the output
is without missing type definitions. Example of the nested serializer and generated TS types:

```python
class CompanyInfo(serializers.Serializer):
    name = serializers.CharField(max_length=100)

class BillingInfo(serializers.Serializer):
    company = CompanyInfo()
```

```typescript
export type CompanyInfo = {
  name: string
}

export type BillingInfo = {
  company: CompanyInfo
}
```

## Fields to properties

A lot of various types of fields are already supported. Alongside simple ones like `CharField`,
`IntegerField`, `BooleanField`, etc. there are also more complicated like `ListField`, 
`SerializerMethodField`, `MultipleChoiceField`. Type hinting is taken into account when deriving
property type for `SerializerMethodField` as can be seen in the following example:

```python
class Foo(serializers.Serializer):
    result = serializers.SerializerMethodField()
    result_unknown = serializers.SerializerMethodField()

    # method with type hint
    def get_result(self, obj) -> int:
        return 42

    # method without type hint
    def get_result_unknown(self, obj):
        return None
```

```typescript
export type Foo = {
  result: number
  resultUnknown: any
}
```

Serializer field keyword arguments are also taken into account. Optional fields
are marked accordingly in TypeScript code:

```python
count = serializers.IntegerField(required=False)
```

```typescript
count?: number
```

Sometimes we want the field to stay mandatory (must be included in the payload) but to
allow null values. For that, we can set `allow_null=True` to the field resulting in a composite
TS type similar to the one generated by `ChoiceField`:

```python
choices = serializers.ChoiceField(choices=[1, 2, 3])
count = serializers.IntegerField(allow_null=True)
```

```typescript
choices: 1 | 2 | 3
count: number | null
```

Array-like fields results in `Array` TypeScript type with correct child type derived:

```python
complex_numbers = ComplexNumber(many=True) # this is example nested serializer
limited_numbers = serializers.MultipleChoiceField(choices=[1, 2, 3])
numbers = serializers.ListField(child=serializers.IntegerField())
```

```typescript
complexNumbers: ComplexNumber[]
limitedNumbers: (1 | 2 | 3)[]
numbers: number[]
```

## Output format

Code style differs among companies and individual developers and with that in mind
command allows you to configure the style of generated TS code using different arguments.
You can choose whether you want to have serializers generated as types or as interfaces
if you want to have semicolons in the output and of course... to choose between tabs or 
spaces for indentation.

![Tabs vs Spaces {1200x1200}](/img/blog/tabs_vs_spaces.png "Tabs vs Spaces")

More about the specific arguments can be found in the [package documentation](https://github.com/remastr/drf-typescript-generator "package documentation").


The following article introduces our Django package that is useful for improving the development experience when working with Django REST framework and TypeScript.

Generate TypeScript types from Django REST framework serializers


In this tutorial, we are going to learn how to push application logs from our Django application to Elasticsearch storage and have the ability to display it in a readable way in Kibana web tool. The main aim of this article is to establish a connection between our Django server and ELK stack (Elasticsearch, Kibana, Logstash) using another tool provided by Elastic - Filebeat. We will also briefly cover all preceding steps, such as the reasoning behind logging, configuring logging in Django and installing ELK stack.

## Why and how to log
Having reasonable logging messages in the production helped me discover several non-trivial bugs that would otherwise be undiscoverable. It is also a good practice to use logging messages in the local environment to speed up the development, enabling these messages to stay there for production use. 

In order to start logging, just add the following lines at the top of your file. The variable \_\_name\_\_ will be translated into the name of the module that will also appear in the final log messages.

```python
import logging

logger = logging.getLogger(__name__)
```

### Logging levels

Another part of the log structure is the log level. Log level helps us identify the severity of the message and makes it easier to navigate in the log output. There are five levels that can be used for log messages. My usage of the logging levels is as follows:

- DEBUG - messages that can help us track the flow of the algorithm, but are not important for anything else than troubleshooting. For example, 'Task has started', 'Task has ended in 5.4 seconds', 'Email for user id=55 was sent'
- INFO - messages that carry important information, e.g. 'Payment transaction finished with status={status}', 'Email with invoice id={invoice_id} was sent to user id={user_id}'
- WARNING - messages that could help us identify potential errors like 'Referer {ip} is not in allow list'
- ERROR - 'Got error message={message} while downloading file={file} from external API'
- CRITICAL - higher than error, I do not use it

### Creating log messages
In order to create a logging message, we need to call a method of the logger object. There are corresponding names to the logging levels.

One addition to those is an exception that corresponds to the error level but will also add a traceback, which is very handy, especially with having an integrated error monitoring tool like Sentry.
```python
# DEBUG
logger.debug('Email to user id={} sent'.format(user.id))

# INFO
logger.info('Payment transaction finished with status={}'.format(payment.status))

# WARNING
logger.warning('Referer {} is not in allow list'.format(ip)

# ERROR
logger.error(
    'Got status code={}, error message={} while sending request to external API'.format(
        response.status_code, response.text
    )
)

# EXCEPTION
try:
    print(b)
except:
    logger.exception('Exception happened.')
```

### Setup logging in your Django application
Now, we need to set up the output of our log. The most common use cases are output on the screen (e.g. when you run manage.py runserver) or output to the log file. We want to go one step further and get the data into Elasticsearch storage in order to have a user-friendly web UI (Kibana) that allows easy search and filter options to access the logs.

When I did the first research, I found a couple of examples of integration with python-logstash and python-json-formatter, but it eventually did not work with python3. It also felt like a lot of hassle to get things going when there is an option to use a tool called [Filebeat](https://www.elastic.co/products/beats/filebeat "Filebeat")  by Elastic. Filebeat monitors changes in the log file and sends all new records to the Elasticsearch storage through a parser called Logstash. Unfortunately, there is one condition that requires us to have full control over our instance(s), which is used to serve our Django application. If not, we need to find a way of using a service to stream logs to our ELK server or fall back to python-logstash. 

Since we are going to use Filebeat, all we need to achieve is storing the data in a reasonable format in log rotated files. Besides that, I recommend using [Sentry](https://sentry.io/welcome/ "Sentry") for managing our errors. We will set up our Django logger config to send error and exception log messages to Sentry.

This is a sample setup of logging in settings.py. For more information, you can also refer to [Django documentation](https://docs.djangoproject.com/en/dev/topics/logging/ "Django documentation"). 

```python
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'simple': {
            'format': '[%(asctime)s] %(levelname)s|%(name)s|%(message)s',
            'datefmt': '%Y-%m-%d %H:%M:%S',
        },
    },
    'handlers': {
        'applogfile': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': '/webapps/myproject/logs/django/myproject.log',
            'maxBytes': 1024*1024*15,  # 15MB
            'backupCount': 10,
            'formatter': 'simple',
        },
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
            'formatter': 'simple'
        }
    },
    'loggers': {
        'app1': {
            'handlers': ['applogfile', 'console'],
            'level': 'DEBUG',
        },
        'app2': {
            'handlers': ['applogfile', 'console'],
            'level': 'DEBUG',
        }
    }
}
```

The log format is composed to be easily parsable by Logstash. The following messages will start appearing if everything is set correctly:

```python
[2019-11-01 02:59:02] DEBUG|users.forms|Captcha valid for user_id=443
[2019-11-01 09:53:48] ERROR|jobs.views|Errror while redirecting the job=3232
[2019-11-01 10:20:31] INFO|newsletter.services.newsletter_service|Newsletter 55351 added keyword.
[2019-11-01 12:42:09] INFO|newsletter.services.newsletter_service|Keywords successfully loaded for newsletter 60473.
```

## ELK + Filebeat infrastructure
I have already mentioned these terms a couple of times. What we need is the ELK infrastructure - Elasticsearch for storing data, Logstash for parsing and streaming data from our log files to Elasticsearch, and Kibana for data visualization. We can either choose a cloud-based solution 
(e.g. [Elastic](https://www.elastic.co/products/elasticsearch/service/pricing "Elastic"), [logz.io](https://logz.io/pricing/ "logz.io"), [AWS Elasticsearch](https://aws.amazon.com/elasticsearch-service/ "AWS Elasticsearch") or go with the self-hosted solution. It all depends on the use case, for a hobby project or if you have DevOps capacity, I would go for a self-hosted solution. In the case of a commercial project, a cloud-based solution is a better choice. Elastic is progressing really fast with its releases, prepare yourself for a lot of maintenance work.

For the purpose of this article, I have prepared a self-hosted solution that is more demanding to install but will give you more detail on how things work inside the ELK infrastructure. Furthermore, this ELK server is further used for our micro startup called [Working Nomads](https://www.workingnomads.co/jobs "Working Nomads"), so it does not mean all energy was wasted for nothing.

![ELK schema {666x281}](/img/blog/elk/schema.png)

### Installing ELK server

There is a $20 droplet with 4gb memory used in this example which is a minimum requirement for a server that has all the components of ELK installed. Your requirements may be much more demanding and you will need a whole cluster of several instances running a single component, but in this case, we are going to demonstrate the solution on this bare minimum server.

I am not going to go into details with the installation, I will just share the tutorials I have used and give a couple of hints to overcome the issues with which I struggled.

Install ELK on Ubuntu 18.04 based on this [example](https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elastic-stack-on-ubuntu-18-04 "example"). I have used Java 11, and instead of the ELK 6.x version, I have used ELK 7.4 version.
Make sure you [install certificates for your Kibana instance](https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-18-04 "install certificates for your Kibana instance").
Now, you should be able to access the Kibana dashboard, we will configure it later.

### Setting up Logstash

Now it is time to feed our Elasticsearch with data. To achieve that, we need to configure Filebeat to stream logs to Logstash and Logstash to parse and store processed logs in JSON format in Elasticsearch. At this moment, we will keep the connection between Filebeat and Logstash unsecured to make the troubleshooting easier. Later in this article, we will secure the connection with SSL certificates.

We need to configure the port which will be used by Logstash to listen to Filebeat. Furthermore, we need to configure a grok regex to convert the log message to a structured JSON. A new index file will be created every day.

Sample configuration in /etc/logstash/conf.d/logstash.conf

```bash
input {
  beats {
    port => 5044
    ssl => false
  }
}

filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}] %{LOGLEVEL:loglevel}\|%{GREEDYDATA:module}\|%{GREEDYDATA:content}" }
  }
  date {
    locale => "en"
    match => [ "timestamp", "YYYY-MM-dd HH:mm:ss"]
    target => "@timestamp"
    timezone => "America/New_York"
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" 
  }
}
```

Now, restart Logstash and ensure the Filebeat input is working by checking the Logstash logs.

If you need to match a different pattern with grok regex, I recommend using [Grok debugger](https://grokdebug.herokuapp.com/ "Grok debugger") to find out what you actually need.

### Setting up Filebeat
Now, we need to log in to the instance that serves the Django application to [install](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html "install") and configure Filebeat to monitor the log files and stream them to Logstash. Based on the previous part of this tutorial, there is a log being generated and rotated in /webapps/myproject/logs/django/. You will see files like these in the mentioned directory:

```bash
myproject.log
myproject.log.1
myproject.log.2
```

Filebeat is capable to manage these rotated log files out of the box so no extra work is needed.

Configuration in /etc/filebeat/filebeat.yml

```bash
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /webapps/myproject/logs/django/*

output.logstash:
  hosts: ["ip_or_hostname_of_your_ELK:5044"]
```

You can now run Filebeat in a debug mode with this command:
```bash
filebeat -e -d "*"
```

Ideally, what we are aiming for is ensuring that Filebeat is sending the log messages to Logstash, such as:
```bash
Publish event: {
  "@timestamp": "2019-11-14T12:36:15.340Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.4.2"
  },
  "ecs": {
    "version": "1.1.0"
  },
  "host": {
    "name": "django-host"
  },
  "agent": {
    "hostname": "django-host",
    "id": "e992e850-bedd-48b2-a885-31c5a853e34e",
    "version": "7.4.2",
    "type": "filebeat",
    "ephemeral_id": "bd808730-a3a2-452b-a2ea-4ee6bbd55c7f"
  },
  "message": "[2019-11-14 17:36:11] DEBUG|app1.utils|Sample debug message",
  "log": {
    "offset": 2177175,
    "file": {
      "path": "/webapps/myproject/logs/django/myproject.log"
    }
  },
  "input": {
    "type": "log"
  }
}
```
If the push from Filebeat to Logstash is successful, we can turn off the command and run it as a service. Make sure it runs at startup after the machine is rebooted.

### Data retention
Based on our use case, we should set the time period for which the logs are kept. Elastic comes with another tool called Curator. 
Follow this [tutorial](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/installation.html "tutorial") to install it, for a newer version of Elasticsearch you need to install it via pip, otherwise, 
Curator will not be compatible with Elasticsearch. Here is the [configuration](http://www.elastic.co/guide/en/elasticsearch/client/curator/current/ex_delete_indices.html "configuration") of the action file which deletes all indices older than 45 days. 
It works right away, you only need to add the [configuration file](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/configfile.html "configuration file") to /home/user/.elasticsearch/ and change the disable_action flag to False.

You can test your configuration with a dry run. 

```bash
curator --dry-run curator_action.yml
```

And then run it daily with cron job.

```bash
0 1 * * * user /usr/local/bin/curator /home/user/.curator/curator_action.yml  >> /var/log/curator.log 2>&1
```

### Setting up Kibana
Now it is time to look into Kibana and see if the data is there. At this point, Kibana will probably offer you a way to configure your index pattern, if not, navigate to Settings > Kibana > Index Patterns and add index pattern "filebeat-*".

![Kibana pattern {920x641}](/img/blog/elk/kibana_1.jpeg)

In the second step select @timestamp as Time filter field.

![Kibana filter field {920x616}](/img/blog/elk/kibana_2.jpeg)

Now go to the Discover area of Kibana, select an appropriate time range and finally, you should see the logs coming in. Select the fields you have parsed through the grok regex - loglevel, module, and content and you will see the log feed.

![Kibana log feed {920x842}](/img/blog/elk/kibana_3.jpeg)

This is a bare minimum of the configuration, we are going to add some handy features and encrypt the communication between Kibana and Filebeat.

### Support for multiline logs
When the exception occurs, we will get a multiline log message with the traceback. We are using the square bracket as the beginning of our log messages and Filebeat can be configured to recognize that. See the enhanced configuration file.

```bash
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /webapps/myproject/logs/django/*
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after

output.logstash:
  hosts: ["ip_or_hostname_of_your_ELK:5044"]
```

Multiline messages like these will start to pop up in your logs.

![ELK multiline log messages {920x118}](/img/blog/elk/kibana_9.jpeg)

For more information about multiline examples, you can consult the official [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html "documentation").

### Securing the connection between Filebeat and Logstash
Currently, the connection between Filebeat and Logstash is unsecured which means logs are being sent unencrypted. It is heavily recommended to set up SSL certificates to make the connection secure and also to ensure that Logstash will only accept data from trusted Filebeat instances. 
To set this up, follow these tutorials: [#1](https://benjaminknofe.com/blog/2018/07/08/logstash-authentication-with-ssl-certificates/ "#1") and [#2](https://github.com/Busindre/How-to-configure-SSL-for-FileBeat-and-Logstash-step-by-step/blob/master/README.md "#2"). 
You can also consult the official [article](https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ssl-logstash.html "article"), but it does not say how to generate the certificates.

The only pitfall I experienced following those tutorials was setting the ownership of the server.key to the Logstash user.

These are the final configuration files for Filebeat and Logstash:

```bash
input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate_authorities => ["/etc/logstash/ca.crt"]
    ssl_certificate => "/etc/logstash/server.crt"
    ssl_key => "/etc/logstash/server.key"
    ssl_verify_mode => "force_peer"
  }
}

filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}] %{LOGLEVEL:loglevel}\|%{GREEDYDATA:module}\|%{GREEDYDATA:content}" }
  }
  date {
    locale => "en"
    match => [ "timestamp", "YYYY-MM-dd HH:mm:ss"]
    target => "@timestamp"
    timezone => "America/New_York"
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" 
  }
}
```

```bash
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /webapps/yourproject/logs/django/*

  ### Multiline options
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["ip_or_hostname_of_your_ELK:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["/etc/filebeat/ca.crt"]

  # Certificate for SSL client authentication
  ssl.certificate: "/etc/filebeat/client.crt"

  # Client Certificate Key
  ssl.key: "/etc/filebeat/client.key"
```

## Further steps
### Setting up Kibana searches
Kibana offers features to make the interface simple to use. First of all, we can create searches that can be easily accessed. Imagine we have multiple instances reporting to our ELK server, we can configure the search based on filtering host.name and loglevel and save it. 

![Kibana search {506x401}](/img/blog/elk/kibana_4.jpeg)

![Kibana search {399x313}](/img/blog/elk/kibana_5.jpeg)

We can create a list of searches, it is possible to add more filtering parameters based on your preference. We can then quickly select the search we need.

![Kibana search {288x288}](/img/blog/elk/kibana_6.jpeg)

### Visualizations
We can create visualizations based on our log messages. Since we have set up data retention, the log messages are not going to stay forever, it is thus not suitable for analytics. Though, it can help us quickly see a drop in a certain metric that will indicate that some functionality stopped working.

![Kibana visualization {920x1127}](/img/blog/elk/kibana_7.png)

### Dashboard
We can put multiple visualizations together and get simple monitoring of our systems.

![Kibana dashboard {920x582}](/img/blog/elk/kibana_8.png)


In this tutorial, we are going to learn how to push application logs from our Django application to Elasticsearch storage and display the messages in Kibana.

Django centralised logging using Elasticsearch, Logstash, Kibana (ELK) + Filebeat


When developing a Django application, there are several ways how to manage project dependencies. In this article, it will be explained which approaches can be used (pip, pipenv, poetry) to take care of dependencies. Of course, the article does not aim to explain all of them nor provide detailed instructions on how they can be used.

## Introduction to classic virtual environment approaches
### Pip & requirements.txt

The first one, which is most common and simplest, is to install all packages into the current virtual environment. This virtual environment needs to be created manually, it can be done that by **python -m venv venv** and then **source bin/activate** to obviously activate/switch into the new environment. There are some other ways how to create Python virtual environments but it is out of the scope of articles to describe all of them. By using the command **pip freeze > requirements.txt**, all dependencies installed in the virtual environment can be saved into the main requirements file. Next time when the project is configured this requirements.txt can be used to install these dependencies by running **pip install -r requirements.txt**.

The main issue is that absolutely all packages are saved with exact versions so updating of dependencies must be done manually. At the initial dependencies file creation, this is not a problem at all, but as time goes it is needed to keep up with the most recent packages due to security patches or new features and this is something that is a tedious task to maintain manually.

### Pipenv

Pipenv is Python dependency management tool. It comes with the features which are solving issues described above coming from using requirements.txt. Pipenv creates the virtual environment automatically so there is no need to take care of it anymore. Dependencies can be installed by using the command **pipenv install some_package**. Once the command is run dependency is saved into **Pipfile** configuration file holding and all package dependencies for projects. When the package version is changed all dependencies versions are recalculated. Meanwhile **Pipfile.lock** is created with exact package hash adding security protection into the installation process. So far Pipenv solves plenty of problems as handling virtual environment or dependencies management. On top of that, it adds some security protections. Using Pipenv comparing to regular requirements.txt is the better solution.

Anyway, there are some doubts when it comes to Pipenv. In the last couple of years development of this tool was stagnating and for a long period of time, there were no updates. In the last year 2020, some updates were releases but it is still a sign that regular development can be stopped or the project can be completely abandoned. As a developer, the aim should be to use choose tools that will be maintained for years. It is nothing worse than opening the project in few years and finding out that used tools are not supporting recent features or they don’t work under the new operating system.

Python has officially introduced the project configuration file as the primary source of truth for dependencies management and configurations called pyproject.toml. Unfortunately, this configuration file is not supported by Pipenv (it was mentioned it is using Pipfile). If the project requires some dependencies configurations for tools like Black or pytest, pyproject.toml can be used but with Pipenv it is needed to have another config file called Pipfile so at the end two files have to be maintained when everything should be in one.

## Poetry - the new era

[Poetry](https://python-poetry.org/ "Poetry") is another dependency management tool combining all features altogether. One tool will carry virtual environment configuration, dependency management, and the best practices in terms of plugins configuration. Basically, it merges all the best features of the methods described above.

One of the neatest features coming from using Poetry is mandatory usage of pyproject.toml config file in the projects. Mostly everything that is related to dependencies and project properties can be specified in one file without splitting configurations across several files.

After successful installation of Poetry by running **pip install poetry**, a new configuration file can be created by applying **poetry init** in the project's directory. When the command is run, the user is asked to answer some basic questions about the project.

```bash
$ poetry init

This command will guide you through creating your pyproject.toml config.

Package name [project]:  project_name
Version [0.1.0]:  1.0.0
Description []:  sample project
Author [None, n to skip]:  John Smith <john@example.com>
License []:  MIT
Compatible Python versions [^3.9]:

Would you like to define your main dependencies interactively? (yes/no) [yes] no
Would you like to define your development dependencies interactively? (yes/no) [yes] no
Generated file
```

Below is an example of the pyproject.toml configuration file which is created after **poetry init**. Executing this command is however not mandatory and a custom bootstrap template with more advanced settings can be created, which can be just copy-pasted between projects.

```bash
[tool.poetry]
name = "project_name"
version = "1.0.0"
description = "sample project"
authors = ["John Smith <john@example.com>"]
license = "MIT"

[tool.poetry.dependencies]
python = "^3.9"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
```

Adding a new plugin into dependencies is easy by using **poetry add plugin-name**. This command will update pyproject.toml and it will resolve all necessary dependencies which are going to be saved into poetry.lock file. The purpose of poetry.lock file is similar to the Pipfile.lock which was described in the Pipfile section.

```bash
$ poetry add django
Using version ^3.2.3 for Django

Updating dependencies
Resolving dependencies... (0.6s)

Writing lock file

Package operations: 4 installs, 0 updates, 0 removals

  • Installing asgiref (3.3.4)
  • Installing pytz (2021.1)
  • Installing sqlparse (0.4.1)
  • Installing django (3.2.3)
```

### Uploading packages on Pipy with Poetry

With the Poetry it is possible to upload custom Python plugins into Pypi repository. With one configuration file, it is possible to handle project dependencies and at the same time configure the package properties in the Pypi repository.

Once the package is ready to be uploaded into Pypi just run **poetry build** to create build files which are going to be used in Pypi.

```bash
$ poetry build
Building project_name (1.0.0)
  - Building sdist
  - Built project_name-1.0.0.tar.gz
  - Building wheel
  - Built project_name-1.0.0-py3-none-any.whl
```

After a successful build, there is only one remaining step to do. After running **poetry publish** everything is going to be uploaded into the Pypi package repository.

```bash
$ poetry publish

Publishing poetry (1.0.0) to PyPI
  - Uploading project_name-1.0.0.tar.gz
  - Uploading project_name-1.0.0-py3-none-any.whl
```

With this approach, setup.py as the main configuration file for the plugin is obsolete and it can be removed from the project which leads to a smaller amount and less maintenance of configuration files in the project.

### One config file for all plugins

As all dependencies are installed, some of them can require a specific configuration file. For example pytest, isort or black. When all these files are created, the project directory can be cluttered with plenty of .cfg or .ini files. But when Poetry is used, which forces to use pyproject.toml, all configuration can be stored in one central file.

```bash
tool.isort]
default_section = "THIRDPARTY"
known_first_party = "project"
known_django = "django"
sections = "FUTURE,STDLIB,DJANGO,THIRDPARTY,FIRSTPARTY,LOCALFOLDER"
line_length = 120
multi_line_output = 5
include_trailing_comma = true
ensure_newline_before_comments = true
use_parentheses = true

[tool.pytest.ini_options]
DJANGO_SETTINGS_MODULE = "project.settings"
python_files = ["tests.py", "test_*.py", "*_tests.py"]
```

Some plugins like flake8, do not have an option to store configurations inside pyproject.toml but there are still workarounds. Use flakehell application as a wrapper around flake8 which allows using pyproject.toml.

### Serverless Framework integration

One neat perk coming from using Poetry and the pyproject.toml configuration file is native support in Serverless Framework. When the Django application is deployed in AWS cloud as the serverless function, the main plugin allowing creating python functions called **serverless-python-requirements** will automatically recognize the pyproject.toml file and it will install defined dependencies into the final zip package uploaded into AWS. The final package is cleaned from development dependencies so they will be not uploaded and they will not increase the final package size.

### Benefits of using Poetry

- pyproject.toml as central config file
- one configuration file for all dependencies and their configs
- no need to manually create and manage virtual environments
- automatically resolves dependencies of installed plugins


In this article, it will be explained which approaches can be used (pip, pipenv, poetry) to take care of dependencies.

Written by Remastr, experienced developers creating Django & React applications.

Our Blog

Mitigating Django migration conflicts

Setting up Django project with Poetry and Docker

Generate TypeScript types from Django REST framework serializers

Django centralised logging using Elasticsearch, Logstash, Kibana (ELK) + Filebeat

Comparison of Pip, Pipenv and Poetry dependency management tools

We're here to help.