Integrating Django with Scrapy

To make paths easy to call, we remove the external scrapy_spider folder so that inside the movie_reviews_analyzer_app, the webmining_server folder is at the same level as the scrapy_spider folder:

├── db.sqlite3
├── scrapy.cfg
├── scrapy_spider
│   ├── ...
│   ├── spiders
│   │   ...
└── webmining_server

We set the Django path into the Scrapy settings.py file:

# Setting up django's project full path.
import sys
sys.path.insert(0, BASE_DIR+'/webmining_server')
# Setting up django's settings module name.
os.environ['DJANGO_SETTINGS_MODULE'] = 'webmining_server.settings'
#import django to load models(otherwise AppRegistryNotReady: Models aren't loaded yet):
import django
django.setup()

Now we can install the library that will allow managing Django models from Scrapy:

sudo pip install scrapy-djangoitem

In the items.py file, we write the links between Django models and Scrapy items as follows:

from scrapy_djangoitem import DjangoItem
from pages.models import Page...