Deploying Spiders

Popular choices for deploying Scrapy spiders are:

Scrapyd is an open source application to run Scrapy spiders. It provides a server with HTTP API, capable of running and monitoring Scrapy spiders.

To deploy spiders to Scrapyd, you can use the scrapyd-deploy tool provided by the package. Please refer to the scrapyd-deploy documentation for more information.

Deploying to Zyte Scrapy Cloud

Zyte Scrapy Cloud is a hosted, cloud-based service by , the company behind Scrapy.

Zyte Scrapy Cloud removes the need to setup and monitor servers and provides a nice UI to manage spiders and review scraped items, logs and stats.

Zyte Scrapy Cloud is compatible with Scrapyd and one can switch between them as needed - the configuration is read from the file just like scrapyd-deploy.