Using FormRequest to send data via HTTP POST

If you want to simulate a HTML Form POST in your spider and send a couple ofkey-value fields, you can return a object (from yourspider) like this:

return [FormRequest(url="http://www.example.com/post/action",
                    formdata={'name': 'John Doe', 'age': '27'},
                    callback=self.after_post)]

使用FormRequest.from_response()方法模拟用户登录

通常网站通过 <input type="hidden"> 实现对某些表单字段（如数据或是登录界面中的认证令牌等）的预填充。使用Scrapy抓取网页时，如果想要预填充或重写像用户名、用户密码这些表单字段，可以使用 方法实现。下面是使用这种方法的爬虫例子:

import scrapy
 
    name = 'example.com'
    start_urls = ['http://www.example.com/users/login.php']
 
    def parse(self, response):
        return scrapy.FormRequest.from_response(
            response,
            formdata={'username': 'john', 'password': 'secret'},
            callback=self.after_login
        )
 
    def after_login(self, response):
        # check login succeed before going on
        if "authentication failed" in response.body:
            self.logger.error("Login failed")
            return
 
        # continue scraping with authenticated session...

class scrapy.http.Response(url[, status=200, headers, body, flags])

A object represents an HTTP response, which is usuallydownloaded (by the Downloader) and fed to the Spiders for processing.

url

A string containing the URL of the response.

This attribute is read-only. To change the URL of a Response use.

status

An integer representing the HTTP status of the response. Example: 200,404.

headers

A dictionary-like object which contains the response headers.

A str containing the body of this Response. Keep in mind that Response.bodyis always a str. If you want the unicode version use (only available inTextResponse and subclasses).

This attribute is read-only. To change the body of a Response use.

request

The object that generated this response. This attribute isassigned in the Scrapy engine, after the response and the request have passedthrough all Downloader Middlewares.In particular, this means that:

- HTTP redirections will cause the original request (to the URL beforeredirection) to be assigned to the redirected response (with the finalURL after redirection).
- Response.request.url doesn’t always equal Response.url
- This attribute is only available in the spider code, and in the, but not inDownloader Middlewares (although you have the Request available there byother means) and handlers of the response_downloaded signal.

meta

A shortcut to the Request.meta attribute of the object (ie. self.request.meta).

Unlike the Response.request attribute, the attribute is propagated along redirects and retries, so you will getthe original Request.meta sent from your spider.


参见

attribute

flags

A list that contains flags for this response. Flags are labels used fortagging Responses. For example: ‘cached’, ‘redirected‘, etc. Andthey’re shown on the string representation of the Response (strmethod) which is used by the engine for logging.

copy()

Returns a new Response which is a copy of this Response.

replace([url, status, headers, body, request, flags, cls])

Returns a Response object with the same members, except for those membersgiven new values by whichever keyword arguments are specified. Theattribute is copied by default.

urljoin(url)

Constructs an absolute url by combining the Response’s witha possible relative url.

This is a wrapper over urlparse.urljoin, it’s merely an alias formaking this call:

urlparse.urljoin(response.url, url)

TextResponse objects

class scrapy.http.TextResponse(url[, encoding[, …]])

objects adds encoding capabilities to the baseResponse class, which is meant to be used only for binary data,such as images, sounds or any media file.

objects support a new constructor argument, inaddition to the base Response objects. The remaining functionalityis the same as for the class and is not documented here.



TextResponse objects support the following attributes in additionto the standard ones:

encoding

A string with the encoding of this response. The encoding is resolved bytrying the following mechanisms, in order:

- the encoding passed in the constructor encoding argument
- the encoding declared in the Content-Type HTTP header. If thisencoding is not valid (ie. unknown), it is ignored and the nextresolution mechanism is tried.
- the encoding declared in the response body. The TextResponse classdoesn’t provide any special functionality for this. However, the and XmlResponse classes do.
- the encoding inferred by looking at the response body. This is the morefragile method but also the last one tried.

selector

A Selector instance using the response astarget. The selector is lazily instantiated on first access.

objects support the following methods in addition tothe standard Response ones:

bodyas_unicode()

Returns the body of the response as unicode. This is equivalent to:








But not equivalent to:

unicode(response.body)

Since, in the latter case, you would be using you system default encoding(typically _ascii) to convert the body to unicode, instead of the responseencoding.

xpath(query)

A shortcut to TextResponse.selector.xpath(query):

response.xpath('//p')

css(query)

A shortcut to TextResponse.selector.css(query):

class scrapy.http.HtmlResponse(url[, …])

The class is a subclass of TextResponsewhich adds encoding auto-discovering support by looking into the HTML attribute. See TextResponse.encoding.

XmlResponse objects

class scrapy.http.XmlResponse(url[, …])

The class is a subclass of whichadds encoding auto-discovering support by looking into the XML declarationline. See .