Deploying Spiders

Popular choices for deploying Scrapy spiders are:

Scrapyd is an open source application to run Scrapy spiders. It providesa server with HTTP API, capable of running and monitoring Scrapy spiders.

Scrapyd is maintained by some of the Scrapy developers.

Deploying to Scrapy Cloud

Scrapy Cloud is a hosted, cloud-based service by ,the company behind Scrapy.

To deploy spiders to Scrapy Cloud you can use the shub command line tool.Please refer to the for more information.

Scrapy Cloud is compatible with Scrapyd and one can switch betweenthem as needed - the configuration is read from the filejust like scrapyd-deploy.