Django 1.7 + Scrapy

Today I am going to share how to use Scrapy and Django together to crawl to a website and store scrapped data to Database using Django.

First, let us build a Django application using the following commands.

pip install django==1.7
django-admin.py startproject example_project
cd example_project

Inside the example_project directory, we will create a django app named app:

python manage.py startapp app

Then, we will update the models.py like this:

from django.db import models

class ExampleDotCom(models.Model):
    title = models.CharField(max_length=255)
    description = models.CharField(max_length=255)

    def __str__(self):
        return self.title

Now we shall update the admin.py inside the app directory:

from django.contrib import admin
from app.models import ExampleDotCom

admin.site.register(ExampleDotCom)

Update INSTALLED_APPS of settings.py like:

INSTALLED_APPS += ('app',)

Now, we will run the following commands in project directory:

python manage.py makemigrations
python manage.py migrate
python manage.py createsuperuser

The last command will prompt to create a super user for the application. Now we will run the following command:

python manage.py runserver

It will start the django application.

Django part is complete for now. Lets start the scrapy project.

In separate directory, we will create a scrapy project using the following commands:

pip install Scrapy==1.0.3
scrapy startproject example_bot

To use with Django application from scrapy application, we shall update its settings.py inside example_bot project directory:

import os
import sys

DJANGO_PROJECT_PATH = 'YOUR/PATH/TO/DJANGO/PROJECT'
DJANGO_SETTINGS_MODULE = 'example_project.settings'

sys.path.insert(0, DJANGO_PROJECT_PATH)
os.environ['DJANGO_SETTINGS_MODULE'] = DJANGO_SETTINGS_MODULE
BOT_NAME = 'example_bot'

SPIDER_MODULES = ['example_bot.spiders']

To connect with django model, we need to install DjangoItem like this:

pip install scrapy-djangoitem==1.0.0

Inside example_bot directory, we will update the items.py file like this:

from scrapy_djangoitem import DjangoItem
from app.models import ExampleDotCom

class ExampleDotComItem(DjangoItem):
    django_model = ExampleDotCom

Now we will create a crawl spider named example.py inside spiders directory:

from scrapy.spiders import BaseSpider
from example_bot.items import ExampleDotComItem

class ExampleSpider(BaseSpider):
    name = "example"
    allowed_domains = ["example.com"]
    start_urls = ['http://www.example.com/']

    def parse(self, response):
         title = response.xpath('//title/text()').extract()[0]
         description = response.xpath('//body/div/p/text()').extract()[0]
         return ExampleDotComItem(title=title, description=description)

Now we shall create an pipeline class like this inside pipelines.py:

class ExPipeline(object):
    def process_item(self, item, spider):
        item.save()
        return item

Now we need to update the settings.py with this:

ITEM_PIPELINES = {
    'example_bot.pipelines.ExPipeline': 1000,
}

Project structure will be like this:

├── django1.7+scrapy
│   ├── example_bot
│   │   ├── __init__.py
│   │   ├── items.py
│   │   ├── pipelines.py
│   │   ├── settings.py
│   │   └── spiders
│   │       ├── __init__.py
│   │       └── example.py 
│   └── scrapy.cfg
└── example_project
    ├── manage.py
    ├── app
    │   ├── __init__.py
    │   ├── models.py
    │   ├── admin.py
    │   └── views.py
    └── example_project
        ├── __init__.py
        ├── settings.py
        └── urls.py

Now we shall run the application using the following command:

scrapy crawl example

Now let us return to the running django application. If everything above is done correctly, then we shall see an object of ExampleDotCom class has been created like the below screenshot in this url http://localhost:8000/admin/app/exampledotcom/:

All Objects

Single Object

Thats all. Up and running django 1.7 + Scrapy project.

Drawbacks: Only implemented using django 1.7

Source: https://github.com/ruddra/django1.7-scrapy1.0.3

Got help and clues from this stackoverflow link.

comments powered by Disqus