Hacks by Ruddra

Django 1.7 and Scrapy

This post is depricated. Use it at your own risk.

Today I am going to share how to use Scrapy and Django together to crawl to a website and store scraped data to Database using Django.

Project setup


First, let us build a Django application using the following commands.

pip install django==1.7
django-admin.py startproject example_project
cd example_project

Inside the example_project directory, we will create a django app named app:

python manage.py startapp app

Then, we will update the models.py like this:

from django.db import models

class ExampleDotCom(models.Model):
    title = models.CharField(max_length=255)
    description = models.CharField(max_length=255)

    def __str__(self):
        return self.title

Now we shall update the admin.py inside the app directory:

from django.contrib import admin
from app.models import ExampleDotCom


Update INSTALLED_APPS of settings.py like:

INSTALLED_APPS += ('app',)

Now, we will run the following commands in project directory:

python manage.py makemigrations
python manage.py migrate
python manage.py createsuperuser

The last command will prompt to create a super user for the application. Now we will run the following command:

python manage.py runserver

It will start the django application.

Django part is complete for now. Lets start the scrapy project.


In separate directory, we will create a scrapy project using the following commands:

pip install Scrapy==1.0.3
scrapy startproject example_bot

To use with Django application from scrapy application, we shall update its settings.py inside example_bot project directory:

import os
import sys

DJANGO_SETTINGS_MODULE = 'example_project.settings'

sys.path.insert(0, DJANGO_PROJECT_PATH)
BOT_NAME = 'example_bot'

SPIDER_MODULES = ['example_bot.spiders']

To connect with django model, we need to install DjangoItem like this:

pip install scrapy-djangoitem==1.0.0

Inside example_bot directory, we will update the items.py file like this:

from scrapy_djangoitem import DjangoItem
from app.models import ExampleDotCom

class ExampleDotComItem(DjangoItem):
    django_model = ExampleDotCom

Now we will create a crawl spider named example.py inside spiders directory:

from scrapy.spiders import BaseSpider
from example_bot.items import ExampleDotComItem

class ExampleSpider(BaseSpider):
    name = "example"
    allowed_domains = ["example.com"]
    start_urls = ['http://www.example.com/']

    def parse(self, response):
            title = response.xpath('//title/text()').extract()[0]
            description = response.xpath('//body/div/p/text()').extract()[0]
            return ExampleDotComItem(title=title, description=description)

Now we shall create an pipeline class like this inside pipelines.py:

class ExPipeline(objec```t):
    def process_item(self, item, spider):
        return item

Now we need to update the settings.py with this:

    'example_bot.pipelines.ExPipeline': 1000,

Project structure will be like this:

├── django1.7+scrapy
│   ├── example_bot
│   │   ├── __init__.py
│   │   ├── items.py
│   │   ├── pipelines.py
│   │   ├── settings.py
│   │   └── spiders
│   │       ├── __init__.py
│   │       └── example.py
│   └── scrapy.cfg
└── example_project
    ├── manage.py
    ├── app
    │   ├── __init__.py
    │   ├── models.py
    │   ├── admin.py
    │   └── views.py
    └── example_project
        ├── __init__.py
        ├── settings.py
        └── urls.py

Now we shall run the application using the following command:

scrapy crawl example

Now let us return to the running django application. If everything above is done correctly, then we shall see an object of ExampleDotCom class has been created like the below screenshot in this url http://localhost:8000/admin/app/exampledotcom/:


Thats all. Up and running django 1.7 + Scrapy project.

Drawbacks: Only implemented using django 1.7

Source: https://github.com/ruddra/django1.7-scrapy1.0.3

Got help and clues from this SO link.

Topics: Scrapy , Django

Share Your Thoughts
M ↓   Markdown

Dário Marcelino
Friday, Jul 22, 2016

I’m using Django 1.9 and I had to add import django django.setup() to scrappy settings.py for it run. Thanks

Monday, Sep 4, 2017

Just in case if you are get blocked, you can use a free proxy with thousands of IP addresses for Scrapy at https://proxy.webshare.io/

Haris Jafri
Wednesday, Apr 25, 2018

where should i put it?

Haris Jafri
Wednesday, Apr 25, 2018

i still get error,,using django 2.0

Friday, Apr 27, 2018

with django 2.x worked code is sys.path.append(os.path.dirname(os.path.abspath('.'))) os.environ['DJANGO_SETTINGS_MODULE'] = 'my_project.settings' import django django.setup()

Haris Jafri
Friday, Apr 27, 2018

There is error that no module app ..where should i add the app module?