Hypnos

Configuring and Running [django|celery|flower|redis|rabitmq|nginx|uwsgi] in Docker Containers

• Repo with code for this article can be found here:
  https://github.com/LiamBao/Hypnos

After reading this blog post, you will be able to configure Celery with Django, PostgreSQL, Redis, and RabbitMQ, and then run everything in Docker containers.

Today, you'll learn how to set up a distributed task processing system for quick prototyping. You will configure Celery with Django, PostgreSQL, Redis, and RabbitMQ, and then run everything in Docker containers. You'll need some working knowledge of Dockerfor this tutorial, which you can get in one my previous posts here.

Django is a well-known Python web framework, and Celery is a distributed task queue. You'll use PostgreSQL as a regular database to store jobs, RabbitMQ as message broker, and Redis as a task storage backend.

Motivation

When you build a web application, sooner or later you'll have to implement some kind of offline task processing.

Example:

Alice wants to convert her cat photos from .jpg to .png or create a .pdf from her collection of .jpg cat files. Doing either of these tasks in one HTTP request will take too long to execute and will unnecessarily burden the web server - meaning we can't serve other requests at the same time. The common solution is to execute the task in the background - often on another machine - and poll for the result.  

A simple setup for an offline task processing could look like this:

1\. Alice uploads a picture.  
2\. Web server schedules job on worker.  
3\. Worker gets job and converts photo.  
4\. Worker creates some result of the task (in this case, a converted photo).  
5\. Web browser polls for the result.  
6\. Web browser gets the result from the server.  

This setup looks clear, but it has a serious flaw - it doesn't scale well. What if Alice has a lot of cat pictures and one server wouldn't be enough to process them all at once? Or, if there was some other very big job and all other jobs would be blocked by it? Does she care if all of the images are processed at once? What if processing fails at some point?

Frankly, there is a solution that won't kill your machine every time you get a bigger selection of images. You need something between the web server and worker: a broker. The web server would schedule new tasks by communicating with the broker, and the broker would communicate with workers to actually execute these tasks. You probably also want to buffer your tasks, retry if they fail, and monitor how many of them were processed.

You would have to create queues for tasks with different priorities, or for those suitable for different kinds of workers.

All of this can be greatly simplified by using Celery - an open-source, distributed tasks queue. It works like a charm after you configure it - as long as you do so correctly.

How Celery is built

Celery consists of:

• Tasks, as defined in your app
• A broker that routes tasks to workers and queues
• Workers doing the actual work
• A storage backend

Your setup

Start with the standard Django project structure. It can be created with django-admin, by running in shell:

$ django-admin startproject hypnos

At the end of this tutorial, it'll look like this:

.
├── README.md
├── docker
│   ├── Dockerfile_nginx
│   └── Dockerfile_web
├── docker-compose.yml
├── hypnos
│   ├── app.sock
│   ├── celery_test
│   │   ├── __init__.py
│   │   ├── admin.py
│   │   ├── apps.py
│   │   ├── migrations
│   │   ├── models.py
│   │   ├── serializers.py
│   │   ├── tasks.py
│   │   ├── tests.py
│   │   ├── urls.py
│   │   └── views.py
│   ├── hypnos
│   │   ├── __init__.py
│   │   ├── celeryconfig.py
│   │   ├── settings.py
│   │   ├── urls.py
│   │   └── wsgi.py
│   ├── logs
│   │   └── debug.log
│   ├── manage.py
├── nginx
│   ├── hypnos.conf
│   ├── logs
│   ├── nginx.conf
│   ├── nginx.conf.default
│   ├── sample_nginx.conf
│   ├── uwsgi_params
│   └── uwsgi_params.default
├── requirements.txt
├── run_celery.sh
├── run_web.sh
├── uwsgi.ini
└── uwsgi.sample

Creating containers

Since we are working with Docker, we need a proper Dockerfile to specify how our image will be built.

Custom container

Dockerfile_web

FROM python:3

MAINTAINER "Liam Bao"
ENV PYTHONUNBUFFERED 1

RUN mkdir -p /src/app
WORKDIR /src/app
ADD requirements.txt /src/app/
RUN pip install -r requirements.txt
ADD . /src/app/

# create unprivileged user
RUN adduser --disabled-password --gecos '' liam  
RUN chmod 777 /src/app/* \
&& chown liam:liam /src/app/*

Dockerfile_nginx

FROM nginx

MAINTAINER "Liam Bao"

WORKDIR /src/app

ADD nginx/nginx.conf /etc/nginx/nginx.conf
ADD nginx/hypnos.conf /etc/nginx/sites-available/

RUN mkdir -p /etc/nginx/sites-enabled/\
    && ln -s /etc/nginx/sites-available/hypnos.conf /etc/nginx/sites-enabled/

# RUN mkdir -p /etc/nginx/sites-enabled/\
#     && ln -s /etc/nginx/sites-available/hypnos.conf /etc/nginx/sites-enabled/\
#     && rm /etc/nginx/conf.d/default.conf

CMD ["nginx", "-g", "daemon off;"]

Our dependencies are:

requirements.txt

Django
psycopg2
celery
redis
djangorestframework
django-celery-results
django-celery-beat
anyjson
django-choices
django-cacheops
kombu
uwsgi

I've frozen versions of dependencies to make sure that you will have a working setup. If you wish, you can update any of them, but it's not guaranteed to work.

Choosing images for services

Now we only need to set up RabbitMQ, PostgreSQL, and Redis. Since Docker introduced its official library, I use its official images whenever possible. However, even these can be broken sometimes. When that happens, you'll have to use something else.

Here are images I tested and selected for this project:

• Official PostgreSQL image
• Official Redis image
• Official RabbitMQ image

Using docker-compose to set up a multicontainer app

Now you'll use docker-compose to combine your own containers with the ones we chose in the last section.

docker-compose.yml

version: '2'

services:  
  # PostgreSQL database
  db:
    image: postgres
    hostname: db
    environment:
      - POSTGRES_USER=liam
      - POSTGRES_PASSWORD=liambao
      - POSTGRES_DB=hypnos
    ports:
      - "5432:5432"

  # Redis
  redis:
    image: redis
    hostname: redis
    ports:
      - "6379:6379"

  # RabbitMQ
  rabbit:
    hostname: rabbit
    image: rabbitmq
    environment:
      - RABBITMQ_DEFAULT_USER=liam
      - RABBITMQ_DEFAULT_PASS=liambao
    ports:
      - "5672:5672"  # we forward this port because it's useful for debugging
      - "15672:15672"  # here, we can access rabbitmq management plugin

  # Django web server
  web:
    build:
      context: .
      dockerfile: docker/Dockerfile_web
    hostname: web
    command: ./run_web.sh
    volumes:
      - .:/src/app  # mount current directory inside container
    ports:
      - "8801:8801"
    # set up links so that web knows about db, rabbit and redis
    links:
      - db
      - rabbit
      - redis
    depends_on:
      - db

  #Nginx
  nginx:
    restart: always
    build:
      context: .
      dockerfile: docker/Dockerfile_nginx
    ports:
      - "8000:8000"
    volumes:
      - .:/src/app
      # - ~/Desktop/github/Hypnos/nginx/logs:/var/log/nginx
    links:
      - web
    depends_on:
      - web

  # Celery worker
  worker_default:
    restart: always
    build:
      context: .
      dockerfile: docker/Dockerfile_web
    command: ./run_celery.sh
    volumes:
      - .:/src/app
    links:
      - db
      - rabbit
      - redis
    depends_on:
      - rabbit

Configuring the web server and worker

You've probably noticed that both the worker and web server run some starting scripts. Here they are (make sure they're executable):

run_web.sh

#!/bin/sh

# # wait for PSQL server to start
sleep 10
PROJECT_PATH=/src/app

cd /src/app/hypnos
# prepare init migration
su -m liam -c "python manage.py makemigrations"
# migrate db, so we have the latest db schema
su -m liam -c "python manage.py migrate"  
#collection all the static files to /staic
su -m liam -c "python manage.py collectstatic  --noinput"


#server test
#>>>>>>>>>>>>>>>>>>>>>>
#Scaling out a container with docker-compose is extremely easy. Just use the docker-compose scale command with the container name and amount when you dont need uwsgi:
# su -m liam -c "docker-compose scale worker_default=5" 
# start development server on public ip interface, on port 8801
###   FOR DEV
# su -m liam -c "python manage.py runserver 0.0.0.0:8801"
#<<<<<<<<<<<<<<<<<<<<<<



su -m liam -c " rm -rf ${PROJECT_PATH}/uwsgi.ini "
su -m liam -c "rm -rf ${PROJECT_PATH}/hpnos/static"
su -m liam -c "echo '' > ${PROJECT_PATH}/hypnos/logs/debug.log"
# su -m liam -c "rm -rf ${PROJECT_PATH}/nginx/logs && mkdir ${PROJECT_PATH}/nginx/logs"

# generate config file for uwsgi
#https type
cat > /src/app/uwsgi.sample < ${PROJECT_PATH}/uwsgi.ini <

run_celery.sh

#!/bin/sh

# wait for RabbitMQ server to start
sleep 10

cd /src/app/hypnos
# run Celery worker for our project myproject with Celery configuration stored in Celeryconf
su -m liam -c "celery -A hypnos.celeryconfig worker -Q default -n default@%h"  

#for test
# su -m liam -c "celery worker -A hypnos.celeryconfig -Q default -n worker_default@%h -l INFO -f logs/worker_default@%h.log"

The first script - run_web.sh - will migrate the database and start the Django development server on port 8000.
The second one , run_celery.sh, will start a Celery worker listening on a queue default.

At this stage, these scripts won't work as we'd like them to because we haven't yet configured them. Our app still doesn't know that we want to use PostgreSQL as the database, or where to find it (in a container somewhere). We also have to configure Redis and RabbitMQ.

But before we get to that, there are some useful Celery settings that will make your system perform better. Below are the complete settings of this Django app.

hypnos/settings.py

"""
Django settings for hypnos project.

Generated by 'django-admin startproject' using Django 1.11.7.

For more information on this file, see
https://docs.djangoproject.com/en/1.11/topics/settings/

For the full list of settings and their values, see
https://docs.djangoproject.com/en/1.11/ref/settings/
"""

import os
from kombu import Exchange, Queue
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))


# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/1.11/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'bf82%m(v7l0r0@&5f01jxjc&^e%ttn3u_bglrcv6cy4=6)hu*l'

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True

ALLOWED_HOSTS = ['*']


# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    ## by liam
    'hypnos',
    'celery_test',
    'rest_framework'
]

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

ROOT_URLCONF = 'hypnos.urls'

REST_FRAMEWORK = {  
    'DEFAULT_PERMISSION_CLASSES': ('rest_framework.permissions.AllowAny',),
    'PAGINATE_BY': 10
}

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

WSGI_APPLICATION = 'hypnos.wsgi.application'


# Database
# https://docs.djangoproject.com/en/1.11/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'hypnos',
        'USER': 'liam',
        'PASSWORD': 'liambao',
        'HOST': 'db',
        'PORT': 5432,
    }
}


# Password validation
# https://docs.djangoproject.com/en/1.11/ref/settings/#auth-password-validators

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]


# Internationalization
# https://docs.djangoproject.com/en/1.11/topics/i18n/

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'UTC'

USE_I18N = True

USE_L10N = True

USE_TZ = True


# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.11/howto/static-files/

STATIC_URL = '/static/'

# Redis comfiguration
REDIS = {
    'default': {
        'HOST': 'redis',
        'PORT': 6379,
        'DB': 0,
        'PASSWORD': ''
    }
}


RABBIT_HOSTNAME = os.environ.get('RABBIT_PORT_5672_TCP', 'rabbit')

if RABBIT_HOSTNAME.startswith('tcp://'):  
    RABBIT_HOSTNAME = RABBIT_HOSTNAME.split('//')[1]

BROKER_URL = os.environ.get('BROKER_URL','')
if not BROKER_URL:  
    BROKER_URL = 'amqp://{user}:{password}@{hostname}/{vhost}/'.format(
        user=os.environ.get('RABBIT_ENV_USER', 'liam'),
        password=os.environ.get('RABBIT_ENV_RABBITMQ_PASS', 'liambao'),
        hostname=RABBIT_HOSTNAME,
        vhost=os.environ.get('RABBIT_ENV_VHOST', ''))

# We don't want to have dead connections stored on rabbitmq, so we have to negotiate using heartbeats
BROKER_HEARTBEAT = '?heartbeat=30'  
if not BROKER_URL.endswith(BROKER_HEARTBEAT):  
    BROKER_URL += BROKER_HEARTBEAT

BROKER_POOL_LIMIT = 1
BROKER_CONNECTION_TIMEOUT = 10

# Celery configuration
# configure queues, currently we have only one
CELERY_DEFAULT_QUEUE = 'default'  
CELERY_QUEUES = (  
    Queue('default', Exchange('default'), routing_key='default'),
)

# Sensible settings for celery
CELERY_ALWAYS_EAGER = False  
CELERY_ACKS_LATE = True  
CELERY_TASK_PUBLISH_RETRY = True  
CELERY_DISABLE_RATE_LIMITS = False

# By default we will ignore result
# If you want to see results and try out tasks interactively, change it to False
# Or change this setting on tasks level
CELERY_IGNORE_RESULT = True  
CELERY_SEND_TASK_ERROR_EMAILS = False  
CELERY_TASK_RESULT_EXPIRES = 600

# Set redis as celery result backend
CELERY_RESULT_BACKEND = 'redis://%s:%d/%d' % (
                        REDIS['default']['HOST'],
                        REDIS['default']['PORT'],
                        REDIS['default']['DB'])
CELERY_REDIS_MAX_CONNECTIONS = 1

# Don't use pickle as serializer, json is much safer
CELERY_TASK_SERIALIZER = "json"  
CELERY_ACCEPT_CONTENT = ['application/json']

CELERYD_HIJACK_ROOT_LOGGER = False  
CELERYD_PREFETCH_MULTIPLIER = 1  
CELERYD_MAX_TASKS_PER_CHILD = 1000  

# django-cacheops

# CACHEOPS_REDIS = "redis://%s%s:%d/%d" % (
#     (":" + REDIS['default']['PASSWORD'] + "@") if REDIS['default']['PASSWORD'] else '',
#     REDIS['default']['HOST'],
#     REDIS['default']['PORT'],
#     REDIS['default']['DB'])
# # with password
# # CACHEOPS_REDIS = "redis://:password@localhost:6379/1"

# CACHEOPS = {
#     # Automatically cache any User.objects.get() calls for 15 minutes
#     # This includes request.user or post.author access,
#     # where Post.author is a foreign key to auth.User
#     'auth.user': {'ops': 'get', 'timeout': 60*15},

#     # Automatically cache all gets and queryset fetches
#     # to other django.contrib.auth models for an hour
#     'auth.*': {'ops': ('fetch', 'get'), 'timeout': 60*60},

#     # Cache gets, fetches, counts and exists to Permission
#     # 'all' is just an alias for ('get', 'fetch', 'count', 'exists')
#     'auth.permission': {'ops': 'all', 'timeout': 60*60},

#     # Enable manual caching on all other models with default timeout of an hour
#     # Use Post.objects.cache().get(...)
#     #  or Tags.objects.filter(...).order_by(...).cache()
#     # to cache particular ORM request.
#     # Invalidation is still automatic
#     '*.*': {'ops': (), 'timeout': 60*60},
# }


# Log

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'
        },
        'simple': {
            'format': '%(levelname)s %(asctime)s %(message)s'
        },
    },
    'filters': {
        'require_debug_true': {
            '()': 'django.utils.log.RequireDebugTrue',
        },
    },
    'handlers': {
        'console': {
            'level': 'DEBUG',
            'filters': ['require_debug_true'],
            'class': 'logging.StreamHandler',
            'formatter': 'simple'
        },
        'file': {
            'level': 'DEBUG',
            'class': 'logging.FileHandler',
            'filename': 'logs/debug.log',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['console', 'file'],
            'propagate': True,
        },
        'hypnos': {
            'handlers': ['file'],
            'level': 'DEBUG',
            'propagate': True,
        },
        'celery': {
            'handlers': ['console'],
            'level': 'INFO',
            'propagate': True,
        },
    },
}

Those settings will configure the Django app so that it will discover the PostgreSQL database, Redis cache, and Celery.

Now, it's time to connect Celery to the app. Create a file celeryconf.py and paste in this code:

hypnos/celeryconf.py

import os

from celery import Celery  
from django.conf import settings

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "hypnos.settings")

app = Celery('hypnos')

CELERY_TIMEZONE = 'UTC'

app.config_from_object('django.conf:settings')  
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS) 

That should be enough to connect Celery to our app, so the run_X scripts will work. You can read more about first steps with Django and Celery here.

Defining tasks

Celery looks for tasks inside the tasks.py file in each Django app. Usually, tasks are created either with a decorator, or by inheriting the Celery Task Class.

Both are fine and good for slightly different use cases.

hypnos/tasks.py

from functools import wraps
from hypnos.celeryconfig import app
from .models import Job

# decorator to avoid code duplication
def update_job(fn):  
    """Decorator that will update Job with result of the function"""

    # wraps will make the name and docstring of fn available for introspection
    @wraps(fn)
    def wrapper(job_id, *args, **kwargs):
        job = Job.objects.get(id=job_id)
        job.status = 'started'
        job.save()
        try:
            # execute the function fn
            result = fn(*args, **kwargs)
            job.result = result
            job.status = 'finished'
            job.save()
        except:
            job.result = None
            job.status = 'failed'
            job.save()
    return wrapper


# two simple numerical tasks that can be computationally intensive

@app.task
@update_job
def power(n):  
    """Return 2 to the n'th power"""
    return 2 ** n


@app.task
@update_job
def fib(n):  
    """Return the n'th Fibonacci number.
    """
    if n < 0:
        raise ValueError("Fibonacci numbers are only defined for n >= 0.")
    return _fib(n)


def _fib(n):  
    if n == 0 or n == 1:
        return n
    else:
        return _fib(n - 1) + _fib(n - 2)

# mapping from names to tasks

TASK_MAPPING = {  
    'power': power,
    'fibonacci': fib
}

Building an API for scheduling tasks

If you have tasks in your system, how do you run them? In this section, you'll create a user interface for job scheduling. In a backend application, the API will be your user interface. Let's use the Django REST Framework for your API.

To make it as simple as possible, your app will have one model and only one ViewSet (endpoint with many HTTP methods).

Create your model, called Job, in hypnos/models.py.

from django.db import models

class Job(models.Model):  
    """Class describing a computational job"""

    # currently, available types of job are:
    TYPES = (
        ('fibonacci', 'fibonacci'),
        ('power', 'power'),
    )

    # list of statuses that job can have
    STATUSES = (
        ('pending', 'pending'),
        ('started', 'started'),
        ('finished', 'finished'),
        ('failed', 'failed'),
    )

    type = models.CharField(choices=TYPES, max_length=20)
    status = models.CharField(choices=STATUSES, max_length=20)

    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)
    argument = models.PositiveIntegerField()
    result = models.IntegerField(null=True)

    def save(self, *args, **kwargs):
        """Save model and if job is in pending state, schedule it"""
        super(Job, self).save(*args, **kwargs)
        if self.status == 'pending':
            from .tasks import TASK_MAPPING
            task = TASK_MAPPING[self.type]
            task.delay(job_id=self.id, n=self.argument)

Then create a serializer, view, and URL configuration to access it.

hypnos/serializers.py

from rest_framework import serializers

from .models import Job


class JobSerializer(serializers.HyperlinkedModelSerializer):  
    class Meta:
        model = Job
        fields =('type', 'status', 'created_at', 'updated_at' , 'argument')

hypnos/views.py

from rest_framework import mixins, viewsets

from .models import Job  
from .serializers import JobSerializer

class JobViewSet(mixins.CreateModelMixin,  
                 mixins.ListModelMixin,
                 mixins.RetrieveModelMixin,
                 viewsets.GenericViewSet):
    """
    API endpoint that allows jobs to be viewed or created.
    """
    queryset = Job.objects.all()
    serializer_class = JobSerializer

hypnos/urls.py

from django.conf.urls import url, include  
from rest_framework import routers

from hypnos import views

router = routers.DefaultRouter()  
# register job endpoint in the router
router.register(r'jobs', views.JobViewSet)

# Wire up our API using automatic URL routing.
# Additionally, we include login URLs for the browsable API.
urlpatterns = [  
    url(r'^', include(router.urls)),
    url(r'^api-auth/', include('rest_framework.urls', namespace='rest_framework'))
]

For completeness, there is also hypnos/wsgi.py, defining WSGI config for the project:

import os  
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "hypnos.settings")

from django.core.wsgi import get_wsgi_application  
application = get_wsgi_application()  

and manage.py

#!/usr/bin/env python
import os  
import sys

if __name__ == "__main__":  
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "hypnos.settings")

    from django.core.management import execute_from_command_line

    execute_from_command_line(sys.argv)

Running the setup

Since everything is run from Docker Compose, make sure you have both Docker and Docker Compose installed before you try to start the app:

$ cd /path/to/hypnos/where/is/docker-compose.yml
$ docker-compose build
$ docker-compose up

The last command will start five different containers, so just start using your API and have some fun with Celery in the meantime.

Accessing the API

Navigate in your browser to 127.0.0.1:8801 to browse your API and schedule some jobs.

Scale things out

Currently, we have only one instance of each container. We can get information about our group of containers with the docker-compose ps command.

Scaling out a container with docker-compose is extremely easy. Just use the docker-compose scale command with the container name and amount:

$ docker-compose scale worker_default=5

Nginx config file

# the upstream component nginx needs to connect to
upstream uwsgi {
    # server api:8801; # use TCP
    server unix:/src/app/hypnos/app.sock; # for a file socket
    # server 127.0.0.1:8080; # for a web port socket (we'll use this first)
}

# configuration of the server
server {
    # the port your site will be served on
    listen    8000;
    # index  index.html;
    # the domain name it will serve for
    # substitute your machine's IP address or FQDN
    server_name  hypnos.com www.hypnos.com;
    charset     utf-8;

    client_max_body_size 100M;   # adjust to taste

    # Django media
    location /media  {
        alias /src/app/hypnos/media;  # your Django project's media files - amend as required
    }

    location /static {
        alias /src/app/hypnos/static; # your Django project's static files - amend as required
    }
    
    location / {
        uwsgi_pass  uwsgi;        
        include     /etc/nginx/uwsgi_params; # the uwsgi_params file you installed         
    }

}

uwsgi config : uwsgi.ini

[uwsgi]

# Django-related settings
# the base directory (full path)
chdir           = /src/app/hypnos
# Django's wsgi file
module          = hypnos.wsgi:application
# the virtualenv (full path)
# home            = /path/to/virtualenv
# process-related settings
# master
master          = true
# maximum number of worker processes
processes       = 2
threads=2
# the socket (use the full path to be safe
socket          = /src/app/hypnos/app.sock
# ... with appropriate permissions - may be needed
chmod-socket    = 664
# clear environment on exit
vacuum          = true

Summary

Congrats! You just married Django with Celery to build a distributed asynchronous computation system. I think you'll agree it was pretty easy to build an API, and even easier to scale workers for it! However, life isn't always so nice to us, and sometimes we have to troubleshoot.