
There are plenty of ways how to deploy Next.js application. Most of them are already covered in Next.js [nextjs docs](https://nextjs.org/docs/deployment "official documentation"). If you are working with AWS, there are just a few solutions that can be used. The most common solutions are EC2 or using [serverless-nextjs](https://github.com/serverless-nextjs "serverless-nextjs") plugin to deploy on Lambda@Edge.

Considering these two most common solutions in the AWS ecosystem, there are some particular things which have to be taken into account, if you are building a maintainable codebase.

**Managing servers.** When working in a smaller team, there is really little or no time to spend on managing custom servers. Focus should be spend on definition of product requirements and quick iteration.

**Developer experience.** To activate developers, it is always awesome to have access to the most recent features immediately after their release so they can be incorporated as soon as possible. New stable middleware support in Next.js? Great, update Next.hs dependency in packages.json and start implementing some cool authentication-related features in middleware.

**Deployment.** Quick iterations are something crucial in a project. You want to deliver smaller chunks of code/features as soon as possible. Without good and quick pipelines this is not possible. From our experience, to be able to get something on production in less than 6 minutes (build, lint, translations download, deploy) is extremely helpful.

**United tooling.** To increase productivity, it is crucial to use similar if not the same tooling across multiple technologies. We are using Serverless Framework for deploying our backends and we always wanted to have the same or similar configuration options for our front end. Fewer tools to learn means more energy for business logic.

All these requirements don't fit any common solution for Next.js on AWS (EC2 or serverless-nextjs) so we were required to find a new way how to meet our requirements. From our perspective, it is okay to have everything "just" on Lambda (not Lambda@Edge). With the Lambda function, we don't have to take care of server management, we can run the most recent version of Next.js, deployment is going to be fast (invalidating CloudFront with Lambda@Edge function takes quite some time) and we will use a similar configuration as we are using for our backends on Lambda.

## Solution

The output of `next build` will be uploaded on the Lambda function with [custom server](https://github.com/dougmoscrop/serverless-http "custom serverless server") which will be compiled by using [serverless-esbuild](https://github.com/floydspace/serverless-esbuild "serverless-esbuild"). 

### Size of the Lambda function

When deploying something on the Lambda function the max size of the zip file must be less than 50MB and unzipped 250MB. Most of the time just node_modules/ directory is larger than 250MB so we have to find a way how to shrink the size of the package. The solution is using [NFT](https://github.com/vercel/nft "nft (Node File Trace)"). It will take care of removing all unused files from node_modules resulting in a smaller final package. The NFT is already used in Next.js when using output: standalone in next.config.js so there is nothing to do.

```javascript
// next.config.js

/** @type {import('next').NextConfig} */

const nextConfig = {
  reactStrictMode: true,
  swcMinify: true,  
  images: {
      // Nowhere to cache the images in Lambda (read only)
      unoptimized: true, // Next 12.3+, other "experimental -> images -> unoptimized"
  },
  output: "standalone" // THIS IS IMPORTANT
}

module.exports = nextConfig
````

### Custom serverless server

Next.js has its own server included so everything we have to do is to connect this server with the Lambda handler. For this purpose, we can use [custom server](https://github.com/dougmoscrop/serverless-http "custom serverless server") which will build this bridge between Lambda and Next.js application. In the example below you can see an example server.

```typescript
// server.ts

import { NextConfig } from "next";
import NextServer from "next/dist/server/next-server";
import serverless from "serverless-http";
// @ts-ignore
import { config } from "./.next/required-server-files.json";

const nextServer = new NextServer({
  hostname: "localhost",
  port: 3000,
  dir: __dirname,
  dev: false,
  conf: {
    ...(config as NextConfig),
  },
});

export const handler = serverless(nextServer.getRequestHandler(), {
  binary: ["*/*"],
});
```

### Serverless Framework Configration

Configuration for Serverless Framework is more or less standard. In our case, this is good because we don't have to use something that we are not used to. The only exception is serverless-esbuild plugin which is taking care of our custom server in server.ts. This part of the configuration takes server.ts transforms it into JavaScript code (server.js) and adds serverless-http dependency into it.

```yaml
# serverless.yml

service: serverless-next
 
provider:
  name: aws
  runtime: nodejs16.x
  region: eu-central-1
  apiGateway:
    shouldStartNameWithService: true
    binaryMediaTypes:
      - "*/*"

functions:
  api:
    handler: server.handler
    events:
      - http: ANY /
      - http: ANY /{proxy+}
 
plugins:
  - serverless-esbuild
 
package:
  patterns:
    - ".next"
    - "node_modules"
    - "public"
    - "_next"
    - "next.config.js"
    - "next-i18next.config.js"
    - "package.json"

custom:
  esbuild:
    bundle: true
    minify: true
    exclude: "*"
    external:
      - "next"
```

### Making everything to work together

Most of the time when you are doing serverless.yml configuration, you will just run `sls deploy` from the root folder and everything will be deployed. Unfortunately, this is not how it works here, and we must create a new build folder with the proper directory structure. We must ensure that the final folder will contain our most important configuration files and then the results of `next build` process.

After creating the distribution folder, we can deploy the project within the new folder. Of course, it is possible to make the whole structure manually but for the sake of simplicity below is the build script.

```bash
#!/bin/bash

BUILD_FOLDER=.dist

next build
rm -rf $BUILD_FOLDER
mv .next/standalone/ $BUILD_FOLDER/ 
cp -r .next/static $BUILD_FOLDER/.next 
rm $BUILD_FOLDER/server.js
cp -r next.config.js $BUILD_FOLDER/ 
cp serverless.yml $BUILD_FOLDER/
cp server.ts $BUILD_FOLDER/ 
cp -r public $BUILD_FOLDER/ 
cd $BUILD_FOLDER 
sls deploy
```

Guide how to properly deploy Next.js application on AWS Lambda (not Lambda@Edge) with Serverless Framework.

Next.js on AWS Lambda function with Serverless Framework


This is the first part of 3-part series on the Amazon OpenSearch service. This part is introductory and contains the necessary theory. If you wish to go straight to deploying, please continue with the upcoming second part of the series.

In the first part, we will take a look at:
* Introduction to Amazon OpenSearch service.
* What happened to Amazon the ElasticSearch service?
* Deployment architecture.
* Accessibility & Authentication Methods.
* Service pricing.
* Cost estimation.

---

## Introduction

Amazon OpenSearch service is a solution for searching, analyzing, and visualizing your data. The data is indexed into performant data structures that enable fast and easy search and filter capabilities. You can further work with the data to build monitoring systems, anomaly detection, alarms, visualizations and put everything into dashboards. OpenSearch is a community-driven open-source fork of ElasticSearch and Kibana. As such, this service still offers an option to deploy either ElasticSearch or OpenSearch. It is up to you. At the moment of writing this post, these versions are available:  
* OpenSearch: 1.0, 1.1
* ElasticSearch: 1.5, 2.3, 5.1, 5.3, 5.5, 5.6, 6.0, 6.2, 6.3, 6.4, 6.5, 6.7, 6.8, 7.1, 7.4, 7.7, 7.8, 7.9, 7.10  

If you opt-in for ElasticSearch, there is also an option to migrate to OpenSearch. Your choice should also take into consideration these notes:
* Amazon ElasticSearch won't receive any further updates, version 7.10 is the latest you will get.
* OpenSearch alpha was released in April 2021 and the production version (1.0) was released on the 12th of July, 2021. As such, Amazon ElasticSearch might be more stable and mature than Amazon OpenSearch.
* If you are interested in what is to come, check [OpenSearch Roadmap](https://github.com/orgs/opensearch-project/projects/1 " OpenSearch Roadmap").

To have a better idea of what it can look like working with OpenSearch, imagine you need to track the performance metrics of your AWS Lambda function. AWS Lambda includes these metrics in the **REPORT** statement of lambda invocation logs. With the right filter, we can focus on these statements:
![Discover {1900x900}](/img/blog/amazon-opensearch-service/discover.png)

In continuation of our effort, we can create visualizations from these metrics and put them on a dashboard:
![Visualize {1900x900}](/img/blog/amazon-opensearch-service/visualization.png)

These metrics are also a great candidate for anomaly detection. Once you have the anomaly detection in place, you can view the anomaly history:
![Anomaly detection 1 {1900x900}](/img/blog/amazon-opensearch-service/anomaly_detection01.png)

Anomaly detection depends on the availability of data. In this image you can see that we are missing some data points:
![Anomaly detection 2 {1900x900}](/img/blog/amazon-opensearch-service/anomaly_detection02.png)

## What Happened To Amazon ElasticSearch Service?

To those who are completely unaware of the story, the 'Amazon OpenSearch service' was formerly named 'Amazon ElasticSearch service'. The AWS service was named Amazon ElasticSearch from its release in 2015 until this year. Long story short, there are two sides to this story, Amazon and Elastic companies. The Elastic company did not like the way they were treated both from legal and open source POV. Hence, they changed licensing of their products and Amazon responded by creating a fork from ElasticSearch 7.10 - Amazon OpenSearch. This is also a reason why you won't get any further updates on Amazon ElasticSearch. If you would like to read more on the story, check the [Elastic.co blog](https://www.elastic.co/blog/why-license-change-AWS "Elastic.co blog").

## Deployment Architecture
In general, these parameters have the highest impact on the cluster performance:  

**Availability zones** - Deployments using only 1 Availability zone are prone to outages and possibly even to complete loss of data. This can be caused by non-standard events like AWS outages or natural disasters. 1 availability zone also means higher latency for people/resources accessing the OpenSearch cluster from a more remote location. Deployments using 2 or 3 availability zones are using data centers isolated from each other, which makes them immune to AWS outages and natural disasters. The latency is also lower as the deployment covers a wider geographical area.

**Master nodes** - Master nodes help to offload cluster management and maintenance tasks. They don't hold any data. Having master nodes increases the stability of the cluster. You can deploy 1 master node, but in that case, OpenSearch does not have a backup master node and in case of failure, your cluster will be down. Recommended number of master nodes is 3. You should always choose an odd number of master nodes. There is always only one master node activated and the rest are idle and will be activated in case of failure. You are also charged for the idle nodes and their only use case is the backup of the master node.

**Master node instance type** - Master node instance type depends on the number of (Master + Data) nodes, indices, and shards. The more data nodes you have, the higher the master instance type should be. AWS recommends the following:
```
+------------+----------------------------------------------------+
| Node count | Recommended minimum dedicated master instance type |
+------------+----------------------------------------------------+
| 1-10       |         m5.large.search OR m6g.large.search        |
+------------+----------------------------------------------------+
| 10-30      |        c5.xlarge.search OR c6g.xlarge.search       |
+------------+----------------------------------------------------+
| 30-75      |       c5.2xlarge.search OR c6g.2xlarge.search      |
+------------+----------------------------------------------------+
| 75-200     |       r5.4xlarge.search OR r6g.4xlarge.search      |
+------------+----------------------------------------------------+
```

**Data nodes** - Data nodes hold the data and execute operations like uploading and querying the data. You should always have at least 2 nodes for high availability. The number of nodes depends on the amount of total storage you need and the allowed maximum storage per instance type. You can start with a minimal amount of data nodes with a minimal instance type that can hold your data for your data retention period.

**Data node instance type** - Data node instance type dictates vCPU count, amount of RAM, and the maximal amount of storage it can hold. It's more or less trial and error until you find a well-balanced configuration. If you are running out of space, but your CPU load is stable, it's better to just add another data node of the same type. If you are having performance issues and your CPU load is constantly high, it's time to scale up and level up the instance type. Use CloudWatch monitoring to review the CPU/Memory/Storage usage.

**Warm/Cold storage nodes** - Storage nodes are used for read-only data. Warm storage node is used for frequently accessed data. Cold storage node is used for infrequently accessed data. Warm storage is using S3 in the backend with caching solution for reduced latency on top of that. Cold storage is also using S3, but there is no computing involved. Storage nodes can be deployed only to a cluster with dedicated master nodes. When you query data stored in warm storage, the data is moved from Amazon S3 to local storage and processing that requires compute power. If you encounter high latency while querying data from warm storage, you need to scale out or scale in. Good approach is to test the performance of warm storage using example data set from your workload while monitoring warm storage node metrics.

It's an industry standard to develop a project in multiple stages. Your architecture should reflect your use case and requirements for each stage. Each stage comes with different requirements and the closer to the production, the more scaled out/up the resources are, the higher usage/traffic, more data is generated and failure/downtime is less tolerable. In the next few sections, we will describe each stage. Of course, everything depends on the type and scale of the project you are running. Some companies might have a production cluster with requirements not being compliant to even your development cluster. Let's consider these 4 stages:
* Test
* Development
* Staging
* Production

#### Test Stage
OpenSearch usage:
* You are trying out new technology and experimenting
* You are aggregating only dummy/disposable data for the sake of learning/experimenting
* Usage/traffic is very low
* Data throughput is very low
* Downtime is expected regularly

Architecture recommendations:
* 1 Availability zone
* No master nodes
* 1-2 data nodes
* Minimal storage per node (10GB)

#### Development stage
OpenSearch usage:
* OpenSearch is part of the development process and helps with log inspection and debugging
* You are aggregating all the data from the development stage of your project
* Usage/traffic is low
* Data throughput is low
* Downtime is expected, but should not hinder the development process too much

Architecture recommendations:
* 1 Availability zone
* No master nodes
* 2-5 data nodes
* Storage per node according to your data throughput

#### Staging stage
If your staging stage is available to your customers as an open alpha/beta, then you might consider applying production stage recommendations, otherwise, you can use the development stage recommendations. In the end, it depends on the usage/traffic and data throughput.

#### Production stage
OpenSearch usage:
* OpenSearch is a critical part of debugging errors on the production stage
* You are aggregating all the data from the production stage of your project
* Usage/traffic is high
* Data throughput is high
* Downtime is not desired

Architecture recommendations:
* 2-3 Availability zones
* At least 3 master nodes
* At least 5 data nodes per 1 master node
* Storage per node according to your data throughput
* Use warm/cold storage nodes for historical data (good example is moving historical data to storage nodes due to legal obligation of holding the data for certain period of time)

## Accessibility & Authentication Methods
Amazon OpenSearch service incorporates a fork of the Kibana dashboard. Through this dashboard, you can access the data stored in OpenSearch and run queries and visualizations on them. The question is, how do you want to access this dashboard? You have 2 options on accessibility:
* Public access from the Internet
* VPC access

Authentication is also one of the things you have to configure right from the start. This part of the configuration is solely up to you. You have these options:
* IAM user / IP-based access
* HTTP basic auth
* Cognito
* SAML (external SSO providers)

## Pricing
You are charged based on 4 metrics:
* Instance uptime (compute) / per hour
* Type of storage (standard EBS vs. provisioned IOPS) / per month
* Amount of allocated storage / per month
* Data transfer / per GB

Amazon OpenSearch is not a cheap service and if your workloads generate a high amount of logs/data, it will also translate into the requirements of the OpenSearch cluster you will need.

## Cost Estimation
When it comes to cost estimation, you need to have a good understanding of your cluster requirements, data throughput, and data retention.
* Cluster requirements: How many data/master nodes do I need?
* Data throughput: How much data is my project generating? How many logs per minute? How big on average is 1 log?
* Data retention: How long do I need to keep the data?

Data throughput can be tricky. OpenSearch saves the data with an overhead. The best way to estimate the throughput is to deploy a test cluster for a short period, feed the data into it, and calculate the average. The minimal cluster you can deploy is this:
* No master nodes
* 1 data-node t3.small.search (2vCPU, 2GB RAM)
* Standard EBS storage
* Allocated 10GB of storage (per node)
* Total cost: 31.38$/month (Europe/Frankfurt)

Once you have all the numbers, you can use the [AWS cost calculator](https://calculator.aws/#/createCalculator/OpenSearchService "AWS cost calculator").

## Conclusion
This part should give you an idea of what the OpenSearch is, what you can do with it, history with the ElasticSearch, what are the options on deploying the service, how does the pricing work, and how to create a cost estimation.

The second part will be mostly practical as we will guide you through the deployment of publicly accessible OpenSearch.

## Useful Links
* [Amazon OpenSearch service documentation](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/what-is.html "Amazon OpenSearch service documentation")
* [OpenSearch project documentation](https://opensearch.org/docs/latest "OpenSearch project documentation")
* [AWS Blog on OpenSearch best practices](https://aws.amazon.com/blogs/big-data/best-practices-for-configuring-your-amazon-opensearch-service-domain/ "AWS Blog on OpenSearch best practices")
* [AWS Blog on OpenSearch data node storage](https://aws.amazon.com/blogs/database/get-started-with-amazon-elasticsearch-service-how-many-data-instances-do-i-need/ "AWS Blog on OpenSearch storage")
* [AWS Blog on OpenSearch data node instance types](https://aws.amazon.com/blogs/big-data/get-started-with-amazon-opensearch-service-t-shirt-size-your-domain/ "AWS Blog on OpenSearch data node instance types")


This is the first part of 3-part series on the Amazon OpenSearch service. In this part, we will introduce you to the Amazon OpenSearch service and go through the necessary theory.

Amazon OpenSearch Service - Introduction


Django migration conflicts are common when developing a Django application. In this blog, we will leverage AWS Lambda, Bitbucket webhooks, and CircleCI workflows to prevent the introduction of Django migration conflict to the dev/main branch.

## Infrastructure requirements

Before you read any further, be aware that this blog post assumes that you have these resources available at your disposal:
* Webhook service
* VCS
* CI/CD

For this blog, we will use the following infrastructure:
* AWS Lambda as webhook service
* Bitbucket for VCS
* CircleCI as CI/CD.  

You can use other solutions for webhook service, VCS, and CI/CD. The idea is the same, but you will have to figure out how to make this work on your infrastructure. If you use the same infrastructure, be sure that you have sufficient permissions for API Gateway and Lambda AWS services. You will also need administrator rights to your project's Bitbucket repository.

## Motivation

Since you are reading this article, you may as well already know what Django migration conflict is. Still, we need to set up a common ground before we move on. Assuming that you work in a company that develops Django apps, you can easily get Django migration conflict. All it requires is for you and your colleague to edit the same Django model or create a migration file with the same order number at the same time. You both work on some features and each of you creates a separate branch for your implementation. Your colleague managed to get his branch merged into the main branch sooner than you and had his migrations applied. In such a case, you have to pull the newest changes and resolve conflicts, if there are any. However, it is not that hard for humans to forget, right? If you forget to pull your colleague's changes before you merge your branch into the main branch, you will introduce Django migration conflict into the main branch. Someone will have to pull the main branch, fix it and push the changes, or undo your merge to the main branch. Especially in agile and incremental development, where there are several pull requests merged on daily basis, it can become very tedious to fix forgotten Django migration conflicts in the main branch, especially if they can be prevented.   

## Prevention

As the name of this blog post suggests, we can use CI/CD. Let us go back to the scenario with you and your colleague. For this prevention to work, you need to have set up CI/CD workflows for your branches. Let us assume that you have different workflows set up for different types of branches. It is logical to have a different workflow for the feature/bugfix branch and dev/main branch. With CI/CD in place, we can add Django migration conflict check to feature/bugfix workflow. This way, you and your colleagues can be aware of possible conflicts before propagating them further and simply fix them on your branch. Since these conflict checks will be done on feature/bugfix branches, your main/dev branches will not require these checks.

## Prevention flow
![Flow](https://github.com/remastr/django-migration-conflict-check/raw/main/docs/img/flow.png "Flow diagram" 100x20)

At the first stage, a developer does an action (commit, push, merge, ...) in VCS. This VCS action triggers an event, which is sent to a Webhook service. Webhook service provides you with the means to process an event and send a response. In our case, we will respond to CI/CD associated with our VCS and trigger an action (rebuild project, trigger new pipeline, ...).

## Setup
The setup procedure involves several steps. Please follow them strictly in this order:
* Create API Keys.
* Set up your <Webhook-Service> + <CI/CD> pair.
* Create Bitbucket webhook.
* CircleCI config - Django migration check.

### Create API keys
##### **Create Bitbucket App Password:**
Bitbucket App Password is your API key. With this key, you can interact with your Bitbucket. At the time of creation, you have to assign permissions to the key. Permissions cannot be modified afterward. This key does not require 2-step verification, so be careful when sharing this key. Lastly, please note that you can see the content of the key only at the time of creation.

* Sign in to the Bitbucket Cloud.
* Go to your "Personal settings".
* Under the "ACCESS MANAGEMENT" section, go to "App passwords" and click there on the "Create app password" button.
* Select permissions according to your use case and click on "Create".
* Save your App password in a safe place. You will need it later on.  


##### **Create CircleCI Personal API Token:**
As with the Bitbucket App Password, this is also your API key and you can see the context only when creating the key. However, in this case, you cannot assign any permissions. This key has by default full access rights, so you have to be extremely careful with whom you share this key.

* Sign in to the CircleCI.
* Go to your "User settings".
* Click on "Personal API Tokens" and there click on the "Create New Token" button.
* Name accordingly your token and hit "Add API Token".
* Save your API token in a safe place.

### AWS Lambda + CircleCI
We will use AWS Lambda to run our webhook processing script in a container. Running code in a container without any server is called serverless computing. The first sentence from Wiki explains it perfectly - "Serverless computing is a cloud computing execution model in which the cloud provider allocates machine resources on-demand, taking care of the servers on behalf of their customers." This container will be reachable from the Internet thanks to the HTTP trigger that we are going to set up. HTTP trigger is done via API Gateway. API Gateway is a service used for creating API. It will provide us with a web address. We can configure these web address URL paths and what code to execute on each of them. Since our webhook has only one use case, the root web address will do. Lastly, we need to upload the source code of our webhook processing script into the AWS Lambda function we have created. At our Github repository, we already have a script available that you can use. All that's left is to set up AWS Lambda function environment variables. The script will need your Bitbucket username, API keys you have created, and lastly, a slug of the project on which you want to Django migration check.

##### **Create a function:**  
* Sign in to the AWS Management console.
* Go to Lambda service.
* Click on the "Create function" button.
* Select the "Author from scratch" option.
* Name your function.
* Select Python3.8 runtime.
* Click on the "Create function" button.  


##### **Add HTTP trigger:**  
* On the function page, click on the "Add trigger" button.
* Select "API Gateway" from the dropdown menu.
* Next, from the second dropdown menu, choose "Create an API".
* Select "HTTP API" API type.
* Select "Open" security from the third dropdown menu.
* Click on the orange "Add" button.  


##### **Upload code:**  
* Download the latest zip package from [ releases](https://github.com/remastr/django-migration-conflict-check/releases " releases").
* On the function page, under "Code" section, click on "Upload from" button and choose ".zip file" option.
* Upload the "aws_deployment.zip" you have downloaded in the first step.  


##### **Set Lambda Environment variables:**  
* On the function page, under "Configuration" section, click on "Environment variables".
* Click on the "Edit" button and add following variables with your values:
```
BITBUCKET_API_URL	https://api.bitbucket.org
BITBUCKET_APP_PASSWORD	<Your-Bitbucket-App-Password>
BITBUCKET_USERNAME	<Your-Bitbucket-Username>
CIRCLECI_API_TOKEN	<Your-Personal-CircleCI-API-Token>
CIRCLECI_PROJECT_SLUG   bb/<Your-CircleCI-Organization-Name>/<CircleCI-Project-Name>
```
* Click on the "Save" button.

### Create Bitbucket Webhook
The final step is to configure the Bitbucket webhook. In this step, you will set up Bitbucket to send notification (JSON payload) to your AWS Lambda function API gateway URL. Notifications will be sent only if the trigger specified by you will get triggered. The script that we provided you with will read the JSON payload and trigger new pipelines on project branches that have PR to the branch that just triggered the "PR merged" trigger.

* Go to your Bitbucket repository.
* Click on "Repository settings".
* Click on "Webhooks" in the menu.
* Click on "Add webhook", type in the title and URL you got from the HTTP trigger (API Gateway).
* Under Triggers, select "Pull request merged".
* Click on the "Save" button.

After you are done, it should look similar to this, except for the URL and name you chose for the webhook.
![Bitbucket webhook setup](/img/blog/mitigate-django-migration-conflicts/bitbucket-webhook-setup.png)

## CircleCI config - Django migration check
The last step required is to add a migration check to your CircleCI config.yml. An example of how you can implement this:
```
version: 2
jobs:
  # Some jobs before

  migration_check:
    docker:
      - image: cimg/python:3.7
      - image: circleci/postgres:11.5-alpine-ram
    steps:
      - checkout
      - run:
          name: Merge dev branch to PR branch
          command: |
            git config --global user.email "dummy@mail.com"
            git config --global user.name "dummy_user"
            git merge --no-commit dev
      - restore_cache:
          keys:
            - dependencies-{{ checksum "your-lock-file" }}
            - dependencies-
      - run:
          name: Check for conflicting migrations
          command: poetry run python src/manage.py migrate

  # Some jobs after

workflows:
  version: 2
  your_project:
    jobs:
      # Some jobs before

      - migration_check:
          requires:
            - build
          filters:
            branches:
              ignore:
                - master
                - staging
                - dev

      # Some jobs after
```

### Explanation
In the example CircleCI config.yml above, we set up a job called **migration_check**, this job checks not only Django migrations but also git merge conflicts as a bonus. The thing is, if you remember the scenario in the motivation section, your colleague has pushed the changes to the dev branch. Naturally, you have to pull these changes to your branch and resolve the conflicts if any occur. This job will do that for you and in other words, also check if you have not by accident forgot to pull the changes. If you forgot, then this job will fail and such a result is to be expected. Once it has pulled the changes, it will restore your project dependencies from a cache, if you have any (it is recommended by CircleCI to save dependencies from a build job to a cache). After it has the dependencies loaded, it will run the Django command for migration. Django command will fail only in case there are some Django migration conflicts.  

## Conclusion
There you have it. With this infrastructure, you can find and resolve git merge conflicts and Django migration conflicts and push clean into the dev/main branch. This setup might look like a lot of work, but it is worth the time. Imagine you have 7 PRs to the dev/main branch and your lead developer will start to merge them. The first merge might pass, but it could cause the other 6 PRs to introduce conflicts to your branch. The lead developer will have to roll back the merge, give you notice, and order you to resolve the conflicts, push changes again and then try to merge. This infrastructure saves you time since no rollback will be needed and you instantly know if there are conflicts. Also now you have the template on how to create Bitbucket webhooks and Bitbucket offers a lot more triggers than just "pull request merged". You can tweak this to your needs and enjoy the benefits.


Leverage your CI/CD and AWS Lambda to mitigate the risk of propagating hidden Django migration conflicts into main/dev branches.

Mitigating Django migration conflicts


Whenever you are generating a lot of data, a natural question comes up. How to process the data? Do you need to set up a whole infrastructure to create a new full-fledged database? Perhaps, you do not need to actively work with the data, you just need to occasionally access them and get the necessary information. AWS Athena provides a solution for the latter case.

## AWS Athena - Analytics tool

As the headline suggests, AWS Athena is a part of AWS analytic tool-set. This tool can help you create ad hoc databases or tables and run queries on them. All it needs is data in form of CSV, JSON, or some columnar formats such as Apache Parquet or ORC.

## Athena Use-Case
Query services like Amazon Athena make it easy to run interactive queries against data directly in Amazon S3 without worrying about formatting data or managing infrastructure. For example, Athena is great if you just need to run a quick query on logs to help you debug an application. You just define a table for your data and start querying using standard SQL.

It is not advisable to use Athena for anything related to production, business, or enterprise level. For these cases, there are dedicated services such as Amazon Redshift or EMR.

## Getting data to AWS Athena

#### Adding data source
In short, AWS Glue can extract, transform and load data. It can be used to create a crawler that can infer database and table schema from your data in Amazon S3. Inferred schema is then stored in AWS Glue Data catalog. Athena can access this catalog and load databases and tables and run queries on them. AWS Glue is already integrated into AWS Athena. Tutorial on how to add a data source to Athena can be found [here](https://docs.aws.amazon.com/athena/latest/ug/data-sources-glue.html "here").

#### AWS service integration with Athena
Athena can fetch data from other AWS services thanks to the integration. For example, if you have running AWS CloudTrail, you can create a table from CloudTrail -> Event History, with just one click. Afterward, you can run queries on logs from CloudTrail in the Athena query editor. Some other useful Athena integrations are:
* AWS CloudFormation
* Amazon CloudFront
* Identity and Access Management (IAM)
* Amazon S3 Inventory

You can read the full list of integrations [here](https://docs.aws.amazon.com/athena/latest/ug/athena-aws-service-integrations.html "here").

## Using Athena query editor

![Athena-Query-Editor {1905x900}](/img/blog/athena/athena_query_editor.png "Query editor")
The interface is quite self-explanatory. On the left side, you can select Data source and Database and see what tables are present. On the right side, there is a text editor for SQL queries and buttons to run them or save them. After you run a query, query results will show up under the text editor. Some tips:
* Save your query using "Save as". After that, any change will be saved automatically.
* In the History tab (at the top), you can download query results in CSV file format.
* Once you load data to Athena, it will be always updated. Let's say you are logging some data to S3 bucket and you load that data into Athena. Those are logs we are talking about, they keep coming all the time, right? Common sense is that Athena would show the newest data from logs from the date & time you loaded the data in. This common sense is false in this case. The data is always up to date.

#### Example query: CloudTrail user activity
CloudTrail is an AWS service responsible for logging user activity. Logs are stored in S3 bucket. You can view user activity logs via CloudTrail -> Event History. It has one downside though - Event History can view only the last 90 days of logs. One way of viewing all logs is using AWS Athena:
1. Go to CloudTrail -> Event History.
2. Click on "Create Athena table" to create a DB table out of the logs.
3. Select a bucket where CloudTrail logs are stored.
4. Click on "Create table".
5. Go to AWS Athena.
6. Select the table with CloudTrail logs.

Provided that you arrived at point 6 above, we can start writing a query. Let's see what services have been user **intern_user**:
```
SELECT DISTINCT eventsource
FROM cloudtrail_logs_aws_dl_tutorial_bucket
WHERE useridentity.username LIKE '%intern_user%'
```

We can now run the query and see results under the query editor:  
![Athena Query Editor {1905x900}](/img/blog/athena/athena_query_editor.png)

## Athena pricing
 Payment is per query. More precisely you are charged based on the amount of data scanned by each query. Pricing may differ per region. Examples:
 * N. Virginia = $5.00 per TB of data scanned
 * Cape Town = $6.00 per TB of data scanned

#### Savings
It's all about the size of data Athena has to read, therefore if you manage to reduce the size of your file, you can make some savings. If you compress files using GZIP, you might see up to 3:1 compression gains.  

It can be also beneficial to load the data in more efficient formats. Columnar formats like Apache Parquet allow more efficient querying. Using such formats allows Athena to read-only columns specified by a query, therefore reducing the amount of data read.


AWS has many hidden gems in the service catalog. AWS Athena is one of them. We will take a look at what the Athena is and what it is capable of.

AWS Athena Analytics Tool


In this tutorial, we are going to learn how to push application logs from our Django application to Elasticsearch storage and have the ability to display it in a readable way in Kibana web tool. The main aim of this article is to establish a connection between our Django server and ELK stack (Elasticsearch, Kibana, Logstash) using another tool provided by Elastic - Filebeat. We will also briefly cover all preceding steps, such as the reasoning behind logging, configuring logging in Django and installing ELK stack.

## Why and how to log
Having reasonable logging messages in the production helped me discover several non-trivial bugs that would otherwise be undiscoverable. It is also a good practice to use logging messages in the local environment to speed up the development, enabling these messages to stay there for production use.

In order to start logging, just add the following lines at the top of your file. The variable \_\_name\_\_ will be translated into the name of the module that will also appear in the final log messages.

```python
import logging

logger = logging.getLogger(__name__)
```

### Logging levels

Another part of the log structure is the log level. Log level helps us identify the severity of the message and makes it easier to navigate in the log output. There are five levels that can be used for log messages. My usage of the logging levels is as follows:

- DEBUG - messages that can help us track the flow of the algorithm, but are not important for anything else than troubleshooting. For example, 'Task has started', 'Task has ended in 5.4 seconds', 'Email for user id=55 was sent'
- INFO - messages that carry important information, e.g. 'Payment transaction finished with status={status}', 'Email with invoice id={invoice_id} was sent to user id={user_id}'
- WARNING - messages that could help us identify potential errors like 'Referer {ip} is not in allow list'
- ERROR - 'Got error message={message} while downloading file={file} from external API'
- CRITICAL - higher than error, I do not use it

### Creating log messages
In order to create a logging message, we need to call a method of the logger object. There are corresponding names to the logging levels.

One addition to those is an exception that corresponds to the error level but will also add a traceback, which is very handy, especially with having an integrated error monitoring tool like Sentry.
```python
# DEBUG
logger.debug('Email to user id={} sent'.format(user.id))

# INFO
logger.info('Payment transaction finished with status={}'.format(payment.status))

# WARNING
logger.warning('Referer {} is not in allow list'.format(ip)

# ERROR
logger.error(
    'Got status code={}, error message={} while sending request to external API'.format(
        response.status_code, response.text
    )
)

# EXCEPTION
try:
    print(b)
except:
    logger.exception('Exception happened.')
```

### Setup logging in your Django application
Now, we need to set up the output of our log. The most common use cases are output on the screen (e.g. when you run manage.py runserver) or output to the log file. We want to go one step further and get the data into Elasticsearch storage in order to have a user-friendly web UI (Kibana) that allows easy search and filter options to access the logs.

When I did the first research, I found a couple of examples of integration with python-logstash and python-json-formatter, but it eventually did not work with python3. It also felt like a lot of hassle to get things going when there is an option to use a tool called [Filebeat](https://www.elastic.co/products/beats/filebeat "Filebeat")  by Elastic. Filebeat monitors changes in the log file and sends all new records to the Elasticsearch storage through a parser called Logstash. Unfortunately, there is one condition that requires us to have full control over our instance(s), which is used to serve our Django application. If not, we need to find a way of using a service to stream logs to our ELK server or fall back to python-logstash.

Since we are going to use Filebeat, all we need to achieve is storing the data in a reasonable format in log rotated files. Besides that, I recommend using [Sentry](https://sentry.io/welcome/ "Sentry") for managing our errors. We will set up our Django logger config to send error and exception log messages to Sentry.

This is a sample setup of logging in settings.py. For more information, you can also refer to [Django documentation](https://docs.djangoproject.com/en/dev/topics/logging/ "Django documentation").

```python
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'simple': {
            'format': '[%(asctime)s] %(levelname)s|%(name)s|%(message)s',
            'datefmt': '%Y-%m-%d %H:%M:%S',
        },
    },
    'handlers': {
        'applogfile': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': '/webapps/myproject/logs/django/myproject.log',
            'maxBytes': 1024*1024*15,  # 15MB
            'backupCount': 10,
            'formatter': 'simple',
        },
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
            'formatter': 'simple'
        }
    },
    'loggers': {
        'app1': {
            'handlers': ['applogfile', 'console'],
            'level': 'DEBUG',
        },
        'app2': {
            'handlers': ['applogfile', 'console'],
            'level': 'DEBUG',
        }
    }
}
```

The log format is composed to be easily parsable by Logstash. The following messages will start appearing if everything is set correctly:

```python
[2019-11-01 02:59:02] DEBUG|users.forms|Captcha valid for user_id=443
[2019-11-01 09:53:48] ERROR|jobs.views|Errror while redirecting the job=3232
[2019-11-01 10:20:31] INFO|newsletter.services.newsletter_service|Newsletter 55351 added keyword.
[2019-11-01 12:42:09] INFO|newsletter.services.newsletter_service|Keywords successfully loaded for newsletter 60473.
```

## ELK + Filebeat infrastructure
I have already mentioned these terms a couple of times. What we need is the ELK infrastructure - Elasticsearch for storing data, Logstash for parsing and streaming data from our log files to Elasticsearch, and Kibana for data visualization. We can either choose a cloud-based solution
(e.g. [Elastic](https://www.elastic.co/products/elasticsearch/service/pricing "Elastic"), [logz.io](https://logz.io/pricing/ "logz.io"), [AWS Elasticsearch](https://aws.amazon.com/elasticsearch-service/ "AWS Elasticsearch") or go with the self-hosted solution. It all depends on the use case, for a hobby project or if you have DevOps capacity, I would go for a self-hosted solution. In the case of a commercial project, a cloud-based solution is a better choice. Elastic is progressing really fast with its releases, prepare yourself for a lot of maintenance work.

For the purpose of this article, I have prepared a self-hosted solution that is more demanding to install but will give you more detail on how things work inside the ELK infrastructure. Furthermore, this ELK server is further used for our micro startup called [Working Nomads](https://www.workingnomads.co/jobs "Working Nomads"), so it does not mean all energy was wasted for nothing.

![ELK schema {666x281}](/img/blog/elk/schema.png)

### Installing ELK server

There is a $20 droplet with 4gb memory used in this example which is a minimum requirement for a server that has all the components of ELK installed. Your requirements may be much more demanding and you will need a whole cluster of several instances running a single component, but in this case, we are going to demonstrate the solution on this bare minimum server.

I am not going to go into details with the installation, I will just share the tutorials I have used and give a couple of hints to overcome the issues with which I struggled.

Install ELK on Ubuntu 18.04 based on this [example](https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elastic-stack-on-ubuntu-18-04 "example"). I have used Java 11, and instead of the ELK 6.x version, I have used ELK 7.4 version.
Make sure you [install certificates for your Kibana instance](https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-18-04 "install certificates for your Kibana instance").
Now, you should be able to access the Kibana dashboard, we will configure it later.

### Setting up Logstash

Now it is time to feed our Elasticsearch with data. To achieve that, we need to configure Filebeat to stream logs to Logstash and Logstash to parse and store processed logs in JSON format in Elasticsearch. At this moment, we will keep the connection between Filebeat and Logstash unsecured to make the troubleshooting easier. Later in this article, we will secure the connection with SSL certificates.

We need to configure the port which will be used by Logstash to listen to Filebeat. Furthermore, we need to configure a grok regex to convert the log message to a structured JSON. A new index file will be created every day.

Sample configuration in /etc/logstash/conf.d/logstash.conf

```bash
input {
  beats {
    port => 5044
    ssl => false
  }
}

filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}] %{LOGLEVEL:loglevel}\|%{GREEDYDATA:module}\|%{GREEDYDATA:content}" }
  }
  date {
    locale => "en"
    match => [ "timestamp", "YYYY-MM-dd HH:mm:ss"]
    target => "@timestamp"
    timezone => "America/New_York"
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}
```

Now, restart Logstash and ensure the Filebeat input is working by checking the Logstash logs.

If you need to match a different pattern with grok regex, I recommend using [Grok debugger](https://grokdebug.herokuapp.com/ "Grok debugger") to find out what you actually need.

### Setting up Filebeat
Now, we need to log in to the instance that serves the Django application to [install](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html "install") and configure Filebeat to monitor the log files and stream them to Logstash. Based on the previous part of this tutorial, there is a log being generated and rotated in /webapps/myproject/logs/django/. You will see files like these in the mentioned directory:

```bash
myproject.log
myproject.log.1
myproject.log.2
```

Filebeat is capable to manage these rotated log files out of the box so no extra work is needed.

Configuration in /etc/filebeat/filebeat.yml

```bash
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /webapps/myproject/logs/django/*

output.logstash:
  hosts: ["ip_or_hostname_of_your_ELK:5044"]
```

You can now run Filebeat in a debug mode with this command:
```bash
filebeat -e -d "*"
```

Ideally, what we are aiming for is ensuring that Filebeat is sending the log messages to Logstash, such as:
```bash
Publish event: {
  "@timestamp": "2019-11-14T12:36:15.340Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.4.2"
  },
  "ecs": {
    "version": "1.1.0"
  },
  "host": {
    "name": "django-host"
  },
  "agent": {
    "hostname": "django-host",
    "id": "e992e850-bedd-48b2-a885-31c5a853e34e",
    "version": "7.4.2",
    "type": "filebeat",
    "ephemeral_id": "bd808730-a3a2-452b-a2ea-4ee6bbd55c7f"
  },
  "message": "[2019-11-14 17:36:11] DEBUG|app1.utils|Sample debug message",
  "log": {
    "offset": 2177175,
    "file": {
      "path": "/webapps/myproject/logs/django/myproject.log"
    }
  },
  "input": {
    "type": "log"
  }
}
```
If the push from Filebeat to Logstash is successful, we can turn off the command and run it as a service. Make sure it runs at startup after the machine is rebooted.

### Data retention
Based on our use case, we should set the time period for which the logs are kept. Elastic comes with another tool called Curator.
Follow this [tutorial](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/installation.html "tutorial") to install it, for a newer version of Elasticsearch you need to install it via pip, otherwise,
Curator will not be compatible with Elasticsearch. Here is the [configuration](http://www.elastic.co/guide/en/elasticsearch/client/curator/current/ex_delete_indices.html "configuration") of the action file which deletes all indices older than 45 days.
It works right away, you only need to add the [configuration file](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/configfile.html "configuration file") to /home/user/.elasticsearch/ and change the disable_action flag to False.

You can test your configuration with a dry run.

```bash
curator --dry-run curator_action.yml
```

And then run it daily with cron job.

```bash
0 1 * * * user /usr/local/bin/curator /home/user/.curator/curator_action.yml  >> /var/log/curator.log 2>&1
```

### Setting up Kibana
Now it is time to look into Kibana and see if the data is there. At this point, Kibana will probably offer you a way to configure your index pattern, if not, navigate to Settings > Kibana > Index Patterns and add index pattern "filebeat-*".

![Kibana pattern {920x641}](/img/blog/elk/kibana_1.jpeg)

In the second step select @timestamp as Time filter field.

![Kibana filter field {920x616}](/img/blog/elk/kibana_2.jpeg)

Now go to the Discover area of Kibana, select an appropriate time range and finally, you should see the logs coming in. Select the fields you have parsed through the grok regex - loglevel, module, and content and you will see the log feed.

![Kibana log feed {920x842}](/img/blog/elk/kibana_3.jpeg)

This is a bare minimum of the configuration, we are going to add some handy features and encrypt the communication between Kibana and Filebeat.

### Support for multiline logs
When the exception occurs, we will get a multiline log message with the traceback. We are using the square bracket as the beginning of our log messages and Filebeat can be configured to recognize that. See the enhanced configuration file.

```bash
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /webapps/myproject/logs/django/*
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after

output.logstash:
  hosts: ["ip_or_hostname_of_your_ELK:5044"]
```

Multiline messages like these will start to pop up in your logs.

![ELK multiline log messages {920x118}](/img/blog/elk/kibana_9.jpeg)

For more information about multiline examples, you can consult the official [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html "documentation").

### Securing the connection between Filebeat and Logstash
Currently, the connection between Filebeat and Logstash is unsecured which means logs are being sent unencrypted. It is heavily recommended to set up SSL certificates to make the connection secure and also to ensure that Logstash will only accept data from trusted Filebeat instances.
To set this up, follow these tutorials: [#1](https://benjaminknofe.com/blog/2018/07/08/logstash-authentication-with-ssl-certificates/ "#1") and [#2](https://github.com/Busindre/How-to-configure-SSL-for-FileBeat-and-Logstash-step-by-step/blob/master/README.md "#2").
You can also consult the official [article](https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ssl-logstash.html "article"), but it does not say how to generate the certificates.

The only pitfall I experienced following those tutorials was setting the ownership of the server.key to the Logstash user.

These are the final configuration files for Filebeat and Logstash:

```bash
input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate_authorities => ["/etc/logstash/ca.crt"]
    ssl_certificate => "/etc/logstash/server.crt"
    ssl_key => "/etc/logstash/server.key"
    ssl_verify_mode => "force_peer"
  }
}

filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}] %{LOGLEVEL:loglevel}\|%{GREEDYDATA:module}\|%{GREEDYDATA:content}" }
  }
  date {
    locale => "en"
    match => [ "timestamp", "YYYY-MM-dd HH:mm:ss"]
    target => "@timestamp"
    timezone => "America/New_York"
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}
```

```bash
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /webapps/yourproject/logs/django/*

  ### Multiline options
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["ip_or_hostname_of_your_ELK:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["/etc/filebeat/ca.crt"]

  # Certificate for SSL client authentication
  ssl.certificate: "/etc/filebeat/client.crt"

  # Client Certificate Key
  ssl.key: "/etc/filebeat/client.key"
```

## Further steps
### Setting up Kibana searches
Kibana offers features to make the interface simple to use. First of all, we can create searches that can be easily accessed. Imagine we have multiple instances reporting to our ELK server, we can configure the search based on filtering host.name and loglevel and save it.

![Kibana search {506x401}](/img/blog/elk/kibana_4.jpeg)

![Kibana search {399x313}](/img/blog/elk/kibana_5.jpeg)

We can create a list of searches, it is possible to add more filtering parameters based on your preference. We can then quickly select the search we need.

![Kibana search {288x288}](/img/blog/elk/kibana_6.jpeg)

### Visualizations
We can create visualizations based on our log messages. Since we have set up data retention, the log messages are not going to stay forever, it is thus not suitable for analytics. Though, it can help us quickly see a drop in a certain metric that will indicate that some functionality stopped working.

![Kibana visualization {920x1127}](/img/blog/elk/kibana_7.png)

### Dashboard
We can put multiple visualizations together and get simple monitoring of our systems.

![Kibana dashboard {920x582}](/img/blog/elk/kibana_8.png)


In this tutorial, we are going to learn how to push application logs from our Django application to Elasticsearch storage and display the messages in Kibana.

Our Blog

Written by Remastr, experienced developers creating Django & React applications.

Next.js on AWS Lambda function with Serverless Framework

Amazon OpenSearch Service - Introduction

Mitigating Django migration conflicts

AWS Athena Analytics Tool

Django centralised logging using Elasticsearch, Logstash, Kibana (ELK) + Filebeat

We're here to help.