EzDevInfo.com

Deploying django in a production server

First of all please let me be clear that I am a windows user and very new to the web world. For the past months I have been learning both python and django, and it has been a great experience for me. Now I have somehow created a small project that I would like to deploy in the production server. Since django has its built-in development server there was no problem for me. But now that I have to deploy it to a production server I googled around and found Nginx + uWSGI or Nginx + Gunicorn as the best option for it. And as uWSGI and Gunicord are incompatible with Windows, I think I should adapt Ubuntu or other Unix system.

So my questions are:

1. Just to be clear, as I will have to work with one of the above, please explain to me why do I need two servers?
2. If I have to adapt the Ubuntu environment, do I have to learn Ubuntu shell scripting, SSH and other stuff? Or the hosting provider will help me do that?
3. Please let me be aware of what else do I need for the above concerned.

Thank you so much for your time and please pardon if my question was a lame question. Hoping for positive response answers. Thank you!

Running a flask app with nginx and gunicorn

I'm new at this and have only been using nginx to serve static files. I have now installed flask and gunicorn. If I run gunicorn -b 127.0.0.2:8000 hello:app and then wget it from the server it works well. If I try to access it from a browser, however, it returns a 404 error (I am running this on a server that hosts a wordpress site which is locatet at root).

The flask app:

from flask import Flask
from werkzeug.contrib.fixers import ProxyFix
app = Flask(__name__)

@app.route('/')
def hello():
    return "Hello world!"

app.wsgi_app = ProxyFix(app.wsgi_app)

if __name__ == '__main__':
    app.run()

And the relevant part of my nginx configuration:

location /flask {
                 proxy_set_header       Host            $http_host;
                 proxy_set_header       X-Real-IP       $remote_addr;
                 proxy_set_header       X-Forwarded-For $proxy_add_x_forwarded_\
for;
                 proxy_pass             http://127.0.0.2:8000;
                 proxy_redirect         off;
    }

I hope this is all the relevant info. If not, do tell. Thanks!

Efficient handling of long running HTTP connections in an nginx/gunicorn/django web architecture

I am working on a web service implemented on top of nginx+gunicorn+django. The clients are smartphone applications. The application needs to make some long running calls to external APIs (Facebook, Amazon S3...), so the server simply queues the job to a job server (using Celery over Redis).

Whenever possible, once the server has queued the job, it returns right away, and the HTTP connection is closed. This works fine and allows the server to sustain very high load.

client                   server                 job server
  .                        |                        |
  .                        |                        |
  |------HTTP request----->|                        |
  |                        |--------queue job------>|
  |<--------close----------|                        |
  .                        |                        |
  .                        |                        |

But in some cases, the client needs to get the result as soon as the job is finished. Unfortunately, there's no way the server can contact the client once the HTTP connection is closed. One solution would be to rely on the client application polling the server every few seconds until the job is completed. I would like to avoid this solution, if possible, mostly because it would hinder the reactiveness of the service, and also because it would load the server with many unnecessary poll requests.

In short, I would like to keep the HTTP connection up and running, doing nothing (except perhaps sending a whitespace every once in a while to keep the TCP connection alive, just like Amazon S3 does), until the job is done, and the server returns the result.

client                   server                 job server
  .                        |                        |
  .                        |                        |
  |------HTTP request----->|                        |
  |                        |--------queue job------>|
  |<------keep-alive-------|                        |
  |         [...]          |                        |
  |<------keep-alive-------|                        |
  |                        |<--------result---------|
  |<----result + close-----|                        |
  .                        |                        |
  .                        |                        |

How can I implement long running HTTP connections in an efficient way, assuming the server is under very high load (it is not the case yet, but the goal to be able to sustain the highest possible load, with hundreds or thousands of requests per second)?

Offloading the actual jobs to other servers should ensure a low CPU usage on the server, but how can I avoid processes piling up and using all the server's RAM, or incoming requests being dropped because of too many open connections?

This is probably mostly a matter of configuring nginx and gunicorn properly. I have read a bit about async workers based on greenlets in gunicorn: the documentation says that async workers are used by "Applications making long blocking calls (Ie, external web services)", this sounds perfect. It also says "In general, an application should be able to make use of these worker classes with no changes". This sounds great. Any feedback on this?

Thanks for your advices.

run web app with gevent

I want to try playing around with gevent as a web server and application framework. I don't see any way to "restart" the server or update the application code without killing and starting the whole python application again.

Is this just how it's done? Maybe it's just a matter of me understanding a different paradigm to the apache way.

Also, as a semi-related question, is it even a good idea to run a web server AND the site/service itself through gevent. I've seen other implementations using gunicorn for the server and gevent for the application but from the benchmarks I've seen, gevent far outperforms gunicorn as a server when it comes to scaling.

Gunicorn worker timeout error

I have setup gunicorn with 3 workers 30 worker connections and using eventlet worker class. It is setup behind Nginx. After every few requests, I see this in the logs.

[ERROR] gunicorn.error: WORKER TIMEOUT (pid:23475)
None
[INFO] gunicorn.error: Booting worker with pid: 23514

Why is this happening? How can I figure out whats going wrong?

thanks

How to run gevent in production

I am making use of gevent in my Python application (Django based). However, I am now wondering how to run it in production. What server should I use? During development, I use gevent.pywsgi, but is that production-ready? I have also heard about gunicorn, but I've seen some pretty bad benchmarks about it.

Note: I need SSL.

How to debug gunicorn failure issues? (Worker failed to boot)

I have a Django web application that uses Gunicorn and runs good locally, but when I deploy app on EC2, I see that Gunicorn is failing:

$ gunicorn_django -b 127.0.0.1:8000 --settings=myapp.settings.dev --debug --log-level info
2012-09-16 17:39:24 [28333] [INFO] Starting gunicorn 0.14.6
2012-09-16 17:39:24 [28333] [INFO] Listening at: http://127.0.0.1:8000 (28333)
2012-09-16 17:39:24 [28333] [INFO] Using worker: sync
2012-09-16 17:39:24 [28336] [INFO] Booting worker with pid: 28336
2012-09-16 17:39:24 [28336] [INFO] Worker exiting (pid: 28336)
2012-09-16 17:39:24 [28333] [INFO] Shutting down: Master
2012-09-16 17:39:24 [28333] [INFO] Reason: Worker failed to boot.

I tried logging with --spew but it keeps on running and no error is shown.

How can I debug this issue?

Can't get access log to work for gunicorn

I'm running gunicorn behind ngninx. I want to log errors in gunicorn to gunicorn-error.log and access logs to gunicorn-access.log.

I've got the errorlog working but not the access log, what am I doing wrong?

This is my gunicorn.conf.py:

bind = '127.0.0.1:8888'
backlog = 2048
workers = 3
errorlog = '/home/my/logs/gunicorn-error.log'
accesslog = '/home/my/logs/gunicorn-access.log'
loglevel = 'debug'
proc_name = 'gunicorn-my'
pidfile = '/var/run/my.pid'

This is the script to run gunicorn:

#!/bin/bash
set -e
ENV=/home/my/env/bin/activate
GUNICORN=gunicorn_django
SETTINGS_PATH=/home/my/app/app/settings
PROJECT_PATH=/home/my/app
CONFROOT=/home/my/app/conf/gunicorn.conf.py

cd $SETTINGS_PATH
source $ENV
export PYTHONPATH=$PROJECT_PATH
exec $GUNICORN app.settings.staging -c $CONFROOT

It creates both gunicorn-error.log and gunicorn-access.log but only gunicorn-error.log gets any logs, eg:

2012-11-20 11:49:57 [27817] [INFO] Starting gunicorn 0.14.6
2012-11-20 11:49:57 [27817] [DEBUG] Arbiter booted
2012-11-20 11:49:57 [27817] [INFO] Listening at: http://127.0.0.1:8888 (27817)
2012-11-20 11:49:57 [27817] [INFO] Using worker: sync
2012-11-20 11:49:58 [27825] [INFO] Booting worker with pid: 27825
2012-11-20 11:49:58 [27828] [INFO] Booting worker with pid: 27828
2012-11-20 11:49:58 [27830] [INFO] Booting worker with pid: 27830

What am I doing wrong? Anyone want to share their working gunicorn.conf.py with error logs and access logs?

Django static files under gunicorn

I want to run my django project under gunicorn on localhost. I installed and integrated gunicorn. When I run:

python manage.py run_gunicorn

It works but there are no any static files (css and js)

I disabled debug and template_debug in settings.py (made them false), but it is still same. Am I missing something?

I call statics like:

{{ STATIC_URL }}css/etc....

Thank you.

Running Django with Gunicorn - Best Practice

There are 3 ways to run a django application with gunicorn:

1. Standard gunicorn + wsgi (ref django doc)

   gunicorn project.wsgi:application

2. Using gunicorn django integration (ref gunicorn doc and django doc):

   python manage.py run_gunicorn

3. Using gunicorn_django command (ref gunicorn doc)

   gunicorn_django [OPTIONS] [SETTINGS_PATH]

Django's documentation suggests using 1., which is not even listed as an option on Gunicorn documentation.

Is there any best practice on the best way to run a django app with gunicorn, and what are the foreseable advantages/disadvantages of these different solutions?

Taking a glimpse at gunicorn's code it looks like they pretty much all do the same: 2. seems to be creating a wsgi app using django's internals, and 3. uses 2.

If that's the case, I wouldn't even understand what's the reason for not simply using "1." all the time, especially since a wsgi.py file is autocreated for you since django 1.4; if that's true maybe simply a documentation improvement should be suggested...

Also, best practice for gunicorn settings with django would be great. Using 1., does it make sense to set some defaults in the wsgi file and avoid additional settings?

References:

1. Should I use django-gunicorn integration or wsgi? only concerns choices 1. and 3., there's no hint for the settings and the answer gives no rationale
2. Deploying Django with gunicorn and nginx give some broader information but is not strictly related nor answer this question
3. Django Gunicorn wsgi about version "4", which is launching gunicorn -c configfile and configfile will point to django_settings to django
4. Django WSGI and Gunicorn is just a bit confusing :) mixing up 1. and 3. Of course wsgi.py is used only with 1.

How to use environment variables with supervisor, gunicorn and django (1.6)

I want to configure supervisor to control gunicorn in my django 1.6 project using an environment variable for SECRET_KEY.

I set my secret key in .bashrc as

export SECRET_KEY=[my_secret_key] 

And I have a shell script to start gunicorn:

NAME="myproject"
LOGFILE=/home/django/myproject/log/gunicorn.log
LOGDIR=$(dirname $LOGFILE)
NUM_WORKERS=3
DJANGO_WSGI_MODULE=myproject.wsgi

USER=django
GROUP=django
IP=0.0.0.0
PORT=8001

echo "Starting $NAME"

cd /home/django/myproject/myproject
source /home/django/.virtualenvs/myproject/bin/activate

test -d $LOGDIR || mkdir -p $LOGDIR

exec gunicorn ${DJANGO_WSGI_MODULE} \
 --name $NAME \
 --workers $NUM_WORKERS \
 --user=$USER --group=$GROUP \
 --log-level=debug \
 --bind=$IP:$PORT
 --log-file=$LOGFILE 2>>$LOGFILE

Then to configure my project's gunicorn server in supervisor:

[program:my_django_project]
directory=/home/django/my_django_project/my_django_project
command=/home/django/my_django_project/my_django_project/gunicorn.sh
user=django
autostart=true
autorestart=true
stdout_logfile=/home/django/my_django_project/log/supervisord.log
stderr_logfile=/home/django/my_django_project/log/supervisor_error.log

If I start gunicorn using my shell script it doesn't throw any error but when I start it with supervisor it fails and I see in the logs that it doesn't "find" my SECRET_KEY.

What's the correct way to configure supervisor to read my shell variables (I wan't to keep them in my .bashrc unless there's a more appropriate way)?

How can I modify Procfile to run Gunicorn process in a non-standard folder on Heroku?

I'm new to heroku and gunicorn so I'm not sure how this works. But I've done some searching and I think I'm close to deploying my Django app (1.5.1). So I know I need a Procfile which has

web: gunicorn app.wsgi

Because my directories are a bit different. I can't run gunicorn in the root directory

app_project
    requirements/
    contributors/
    app/
        app/
            settings/
            wsgi.py
        # Normally Procfile goes here
    Procfile

Normally app/ would be the root directory, but I decided to structure my folders this way to separate my django app from some other things. Since I have to put the Procfile in the root directory for heroku to recognize it, what should I put in the Procfile and/or what parameters should I place in the gunicorn command?

Note:

web: gunicorn app.wsgi # won't work because Procfile is in a directory above
                       # I also want to keep the directories as is
                       # I also don't want to create a secondary git inside the app folder just for heroku
web: gunicorn app.app.wsgi # won't work because I don't want to convert the folder into a python module

gunicorn.errors.HaltServer: django

I have a django app and trying to set it up with gunicorn first and later with supervisor and nginx.

The app is running with the normal django command perfectly like python manage.py runserver

I installed the gunicorn using pip like pip install gunicorn and django version is 1.5.3

when i run the below command inside the virtual env like below

gunicorn hello.wsgi:application -b xx.xxx.xxx.xx:8000 and faced the error

Traceback (most recent call last):
  File "/root/Envs/proj/bin/gunicorn", line 9, in <module>
    load_entry_point('gunicorn==19.0.0', 'console_scripts', 'gunicorn')()
  File "/root/Envs/proj/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 74, in run
    WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
  File "/root/Envs/proj/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 166, in run
    super(Application, self).run()
  File "/root/Envs/proj/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 71, in run
    Arbiter(self).run()
  File "/root/Envs/proj/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 169, in run
    self.manage_workers()
  File "/root/Envs/proj/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 477, in manage_workers
    self.spawn_workers()
  File "/root/Envs/proj/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 537, in spawn_workers
    time.sleep(0.1 * random.random())
  File "/root/Envs/proj/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 209, in handle_chld
    self.reap_workers()
  File "/root/Envs/proj/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 459, in reap_workers
    raise HaltServer(reason, self.WORKER_BOOT_ERROR)
gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>

So why actually the above error is encountered and whats the fix ?

Gunicorn Nginx timeout problem

I'm running django on gunicorn+nginx. I'm facing a problem with file uploads. Actually uploads are working fine but gunicorn times out thus causing this in nginx:

2011/07/25 12:13:47 [error] 15169#0: *2317 upstream timed out (110: Connection timed out) while reading response header from upstream, client: IP-ADDRESS, server: SERVER, request: "GET /photos/events/event/25 HTTP/1.1", upstream: "http://127.0.0.1:29000/photos/events/event/25", host: "HOST", referrer: "REFERER_ADDRESS"

If I refresh the page, I can see all the photos are uploaded just fine. The problem is that it causes a timeout thus giving the impression that upload did not work.

here is my gunicorn conf:

bind = "127.0.0.1:29000"
logfile = "/path/to/logs/gunicorn.log"
workers = 3

I tried changing timeout but it didn't work.

A better way to restart/reload Gunicorn (via Upstart) after 'git pull'ing my Django projects

Im looking for something better than sudo restart projectname every time I issue a git pull origin master, which pulls down my latest changes to a Django project. This restart command, I believe, is related to Upstart, which I use to start/top my Gunicorn server process.

This restart causes a brief outage. Users hitting the web server (nginx) will get a 500, because Gunicorn is still restarting. In fact, it seems to restart instantly, but it takes a few seconds for pages to load.

Any ideas on how to make this seamless? Ideally, I'd like to issue my git pull and Gunicorn reloads automatically.