Hacks by Ruddra

Django 1.7 + Scrapy

This post is depricated. Use it at your own risk.

Today I am going to share how to use Scrapy and Django together to crawl to a website and store scraped data to Database using Django.

Project Setup


First, let us build a Django application using the following commands.

pip install django==1.7
django-admin.py startproject example_project
cd example_project

Inside the example_project directory, we will create a django app named app:

python manage.py startapp app

Then, we will update the models.py like this:

from django.db import models

class ExampleDotCom(models.Model):
    title = models.CharField(max_length=255)
    description = models.CharField(max_length=255)

    def __str__(self):
        return self.title

Now we shall update the admin.py inside the app directory:

from django.contrib import admin
from app.models import ExampleDotCom


Update INSTALLED_APPS of settings.py like:

INSTALLED_APPS += ('app',)

Now, we will run the following commands in project directory:

python manage.py makemigrations
python manage.py migrate
python manage.py createsuperuser

The last command will prompt to create a super user for the application. Now we will run the following command:

python manage.py runserver

It will start the django application.

Django part is complete for now. Lets start the scrapy project.


In separate directory, we will create a scrapy project using the following commands:

pip install Scrapy==1.0.3
scrapy startproject example_bot

To use with Django application from scrapy application, we shall update its settings.py inside example_bot project directory:

import os
import sys

DJANGO_SETTINGS_MODULE = 'example_project.settings'

sys.path.insert(0, DJANGO_PROJECT_PATH)
BOT_NAME = 'example_bot'

SPIDER_MODULES = ['example_bot.spiders']

To connect with django model, we need to install DjangoItem like this:

pip install scrapy-djangoitem==1.0.0

Inside example_bot directory, we will update the items.py file like this:

from scrapy_djangoitem import DjangoItem
from app.models import ExampleDotCom

class ExampleDotComItem(DjangoItem):
    django_model = ExampleDotCom

Now we will create a crawl spider named example.py inside spiders directory:

from scrapy.spiders import BaseSpider
from example_bot.items import ExampleDotComItem

class ExampleSpider(BaseSpider):
    name = "example"
    allowed_domains = ["example.com"]
    start_urls = ['http://www.example.com/']

    def parse(self, response):
            title = response.xpath('//title/text()').extract()[0]
            description = response.xpath('//body/div/p/text()').extract()[0]
            return ExampleDotComItem(title=title, description=description)

Now we shall create an pipeline class like this inside pipelines.py:

class ExPipeline(objec```t):
    def process_item(self, item, spider):
        return item

Now we need to update the settings.py with this:

    'example_bot.pipelines.ExPipeline': 1000,

Project structure will be like this:

├── django1.7+scrapy
│   ├── example_bot
│   │   ├── __init__.py
│   │   ├── items.py
│   │   ├── pipelines.py
│   │   ├── settings.py
│   │   └── spiders
│   │       ├── __init__.py
│   │       └── example.py
│   └── scrapy.cfg
└── example_project
    ├── manage.py
    ├── app
    │   ├── __init__.py
    │   ├── models.py
    │   ├── admin.py
    │   └── views.py
    └── example_project
        ├── __init__.py
        ├── settings.py
        └── urls.py

Now we shall run the application using the following command:

scrapy crawl example

Now let us return to the running django application. If everything above is done correctly, then we shall see an object of ExampleDotCom class has been created like the below screenshot in this url http://localhost:8000/admin/app/exampledotcom/:


Thats all. Up and running django 1.7 + Scrapy project.

Drawbacks: Only implemented using django 1.7

Source: https://github.com/ruddra/django1.7-scrapy1.0.3

Got help and clues from this SO link.