Benchmarking

    Scrapy提供了一个简单的性能测试工具。其创建了一个本地HTTP服务器，并以最大可能的速度进行爬取。该测试性能工具目的是测试Scrapy在您的硬件上的效率，来获得一个基本的底线用于对比。其使用了一个简单的spider，仅跟进链接，不做任何处理。

    您能看到类似的输出:

    1. 2013-05-16 13:08:46-0300 [scrapy] INFO: Scrapy 0.17.0 started (bot: scrapybot)
    2. 2013-05-16 13:08:47-0300 [scrapy] INFO: Spider opened
    3. 2013-05-16 13:08:47-0300 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
    4. 2013-05-16 13:08:48-0300 [scrapy] INFO: Crawled 74 pages (at 4440 pages/min), scraped 0 items (at 0 items/min)
    5. 2013-05-16 13:08:49-0300 [scrapy] INFO: Crawled 143 pages (at 4140 pages/min), scraped 0 items (at 0 items/min)
    6. 2013-05-16 13:08:50-0300 [scrapy] INFO: Crawled 210 pages (at 4020 pages/min), scraped 0 items (at 0 items/min)
    7. 2013-05-16 13:08:51-0300 [scrapy] INFO: Crawled 274 pages (at 3840 pages/min), scraped 0 items (at 0 items/min)
    8. 2013-05-16 13:08:52-0300 [scrapy] INFO: Crawled 343 pages (at 4140 pages/min), scraped 0 items (at 0 items/min)
    9. 2013-05-16 13:08:53-0300 [scrapy] INFO: Crawled 410 pages (at 4020 pages/min), scraped 0 items (at 0 items/min)
    10. 2013-05-16 13:08:54-0300 [scrapy] INFO: Crawled 474 pages (at 3840 pages/min), scraped 0 items (at 0 items/min)
    11. 2013-05-16 13:08:56-0300 [scrapy] INFO: Crawled 602 pages (at 3840 pages/min), scraped 0 items (at 0 items/min)
    12. 2013-05-16 13:08:57-0300 [scrapy] INFO: Closing spider (closespider_timeout)
    13. 2013-05-16 13:08:57-0300 [scrapy] INFO: Crawled 666 pages (at 3840 pages/min), scraped 0 items (at 0 items/min)
    14. 2013-05-16 13:08:57-0300 [scrapy] INFO: Dumping Scrapy stats:
    15.     {'downloader/request_bytes': 231508,
    16.      'downloader/request_count': 682,
    17.      'downloader/request_method_count/GET': 682,
    18.      'downloader/response_bytes': 1172802,
    19.      'downloader/response_count': 682,
    20.      'downloader/response_status_count/200': 682,
    21.      'finish_time': datetime.datetime(2013, 5, 16, 16, 8, 57, 985539),
    22.      'log_count/INFO': 14,
    23.      'request_depth_max': 34,
    24.      'response_received_count': 682,
    25.      'scheduler/dequeued': 682,
    26.      'scheduler/dequeued/memory': 682,
    27.      'scheduler/enqueued': 12767,
    28.      'scheduler/enqueued/memory': 12767,
    29.      'start_time': datetime.datetime(2013, 5, 16, 16, 8, 47, 676539)}
    30. 2013-05-16 13:08:57-0300 [scrapy] INFO: Spider closed (closespider_timeout)

    未来会有更多的用例会被加入到性能测试套装中，以覆盖更多常见的情景。