Setting up Continuous Integration with Django and Jenkins

After my experiement with ASP.NET MVC, I went back to using Django for my current side project. I found that I click better with Django better and you have to appreciate the magical admin interface that Django provides. But more on that later.

I've tasked myself to create a Jenkins job that will run all the tests and create coverage report. Eventually, I plan to use it to deploy it to my own Continuous Integration (CI) server. So, let me guide you through what I did today, step-by-step.

Step 1 - Separate your settings

I've separated my settings.py file into a Python module itself. Currently, I have the following structure:

settings/
    base.py
    local.py
    ci.py
    production.py

The base.py contains the most of my Django settings, including all my apps and other configuration. But there are some apps & settings that do not belong in production environment - such as django-debug-toolbar. My ci.py contains the following:

# Import the main (base) settings file
from .base import *

DATABASES = {
    # your database settings for your CI server
}

# The following two are settings related to django_jenkins package
PROJECT_APPS = (
    # List of all of your apps that you want the coverage report generated for, such as Polls (from Django tutorial)
)

JENKINS_TASKS = (
    # List of all tasks you want django_jenkins to run (linters) - see documentation
)

# Additional apps you want running on your CI server
INSTALLED_APPS += ("debug_toolbar", "django_jenkins")

From this file, you can tell that we are going to need the django_jenkins package. You should read its documentation - its very useful.

Step 2 - Separate your pip requirements

I did a very similar thing for my requirements.txt files - I created a requirements folder and added base.txt, local.txt, ci.txt, and production.txt. Each one of those has their corresponding requirements (e.g. only the ci.txt would have django_jenkins package listed).

Step 3 - Install Jenkins plugins

I am going to assume you have Jenkins up & running on a server somewhere. For all this to work, you're going to need the following Jenkins plugins:

  • Bitbucket/GitHub plugin - whatever you use to pull the sources from
  • ShiningPanda - for managing Python's executables (and virtualenv)
  • Cobertura - for creating coverage reports
Step 4 - Create a new Jenkins job

Before creating a Jenkins job, we need to do some prep work:

  • Login to your server via SSH. Create a directory somewhere, e.g. /home/<username>/webapps/djangoproject. Replace the <username> part with your username if you choose to put it there.
  • Run the following: sudo chown -R jenkins:jenkins /home/<username>/webapps/djangoproject

Now, back in the Jenkins' "Create a New Item" interface, select a "Freestyle Project" and then:

  • Under "General" tab, click on "Advanced". Then, check "Use custom workspace". In the "Directory" input, put the path to the folder you created earlier using SSH (e.g. /home/<username>/webapps/djangoproject

  • For "Source Code Management" I really cannot help you much - the idea is to configure it so that it pulls the changes from your repository. Note that for Bitbucket (when using its plugin), you will have to configure webhooks inside your Bitbucket repository - more inforomation here.

  • For "Build Triggers", check whatever you want. Personally, I left it at "Build when a change is pushed to BitBucket".

  • Build environment - I left this as it is.

  • Build - the beast. You actually need only one build step. Under "Add Build Step", select "Virtualenv Builder". This is a part of the ShiningPanda plugin. You do not need to create a virtualenv manually - the plugin will do it for you. Make sure to select the correct Python version (e.g. Python 2 or Python 3) and set the "Nature" to "Shell". Finally, in the "Command box", put the following:

    cd /home/<username>/webapps/djangoproject/<your_django_root>
    pip install -r requirements/ci.txt
    ./manage.py migrate --settings=settings.ci
    ./manage.py jenkins --enable-coverage --settings=settings.ci
    

Let's see what is happening: first, we cd into the directory we created earlier. We also enter the Django project root. Then, we use pip to install the requirements from the ci.txt file. Then, we run the migrate command and point the settings to our ci.py file we created earlier. Finally, we run the jenkins command provided by the django_jenkins package, again pointing to our ci.py settings file.

The build should be working fine now! It would be a good idea to run the build manually just to make sure that what we did so far works. Remember to check the Console Log if the build fails.

Step 5 - Generating reports

In the "Post-build Actions", first add the "Publish Cobertura Coverage Report". Configure the "Cobertura xml report pattern" to point to <your_django_root>/reports/coverage.xml.

Then, add the "Publish JUnit test results report" and under "Test Report XMLs", add <your_django_root>/reports/junit.xml.

Step 6 - Run it!

That's it! You should be able to run the build which will now run the tests and create coverage reports (all thanks to django_jenkins package). If you have any problems, let me know and I'll try to help!