Item Exporters

    For this purpose Scrapy provides a collection of Item Exporters for different output formats, such as XML, CSV or JSON.

    If you are in a hurry, and just want to use an Item Exporter to output scraped data see the Feed exports. Otherwise, if you want to know how Item Exporters work or need more custom functionality (not covered by the default exports), continue reading below.

    In order to use an Item Exporter, you must instantiate it with its required args. Each Item Exporter requires different arguments, so check each exporter documentation to be sure, in . After you have instantiated your exporter, you have to:

    1. call the method start_exporting() in order to signal the beginning of the exporting process

    2. call the method for each item you want to export

    3. and finally call the finish_exporting() to signal the end of the exporting process

    Here you can see an which uses multiple Item Exporters to group scraped items to different files according to the value of one of their fields:

    By default, the field values are passed unmodified to the underlying serialization library, and the decision of how to serialize them is delegated to each particular serialization library.

    However, you can customize how each field value is serialized before it is passed to the serialization library.

    There are two ways to customize how a field will be serialized, which are described next.

    If you use you can declare a serializer in the field metadata. The serializer must be a callable which receives a value and returns its serialized form.

    Example:

    1. import scrapy
    3. def serialize_price(value):
    4.     return f'$ {str(value)}'
    6. class Product(scrapy.Item):
    7.     name = scrapy.Field()
    8.     price = scrapy.Field(serializer=serialize_price)

    2. Overriding the serialize_field() method

    You can also override the serialize_field() method to customize how your field value will be exported.

    Make sure you call the base class method after your custom code.

    Example:

    1. from scrapy.exporter import XmlItemExporter
    3. class ProductXmlExporter(XmlItemExporter):
    5.         if name == 'price':
    6.             return f'$ {str(value)}'
    7.         return super().serialize_field(field, name, value)

    Here is a list of the Item Exporters bundled with Scrapy. Some of them contain output examples, which assume you’re exporting these two items:

    BaseItemExporter

    class scrapy.exporters.BaseItemExporter(fields_to_export=None, export_empty_fields=False, encoding=’utf-8’, indent=0, dont_fail=False)

    This is the (abstract) base class for all Item Exporters. It provides support for common features used by all (concrete) Item Exporters, such as defining what fields to export, whether to export empty fields, or which encoding to use.

    These features can be configured through the __init__ method arguments which populate their respective instance attributes: fields_to_export, , encoding, .

    New in version 2.0: The dont_fail parameter.

    • export_item(item)[source]

      Exports the given item. This method must be implemented in subclasses.

    • serialize_field(field, name, value)

      Return the serialized value for the given field. You can override this method (in your custom Item Exporters) if you want to control how a particular field or value will be serialized/exported.

      By default, this method looks for a serializer declared in the item field and returns the result of applying that serializer to the value. If no serializer is found, it returns the value unchanged.

      • Parameters

        • field (Field object or a instance) – the field being serialized. If the source item object does not define field metadata, field is an empty .

        • name (str) – the name of the field being serialized

        • value – the value being serialized

    • start_exporting()

      Signal the beginning of the exporting process. Some exporters may use this to generate some required header (for example, the XmlItemExporter). You must call this method before exporting any items.

    • finish_exporting()

      Signal the end of the exporting process. Some exporters may use this to generate some required footer (for example, the XmlItemExporter). You must always call this method after you have no more items to export.

    • A list with the name of the fields that will be exported, or None if you want to export all fields. Defaults to None.

      Some exporters (like ) respect the order of the fields defined in this attribute.

      When using item objects that do not expose all their possible fields, exporters that do not support exporting a different subset of fields per item will only export the fields found in the first item exported. Use fields_to_export to define all the fields to be exported.

    • export_empty_fields

      Whether to include empty/unpopulated item fields in the exported data. Defaults to False. Some exporters (like ) ignore this attribute and always export all empty fields.

      This option is ignored for dict items.

    • encoding

      The output character encoding.

    • indent

      Amount of spaces used to indent the output on each level. Defaults to 0.

      • indent=None selects the most compact representation, all items in the same line with no indentation

      • indent<=0 each item on its own line, no indentation

      • indent>0 each item on its own line, indented with the provided numeric value

    class scrapy.exporters.PythonItemExporter(**, dont_fail=False, **kwargs*)[source]

    This is a base class for item exporters that extends with support for nested items.

    It serializes items to built-in Python types, so that any serialization library (e.g. json or ) can be used on top of it.

    XmlItemExporter

    class scrapy.exporters.XmlItemExporter(file, item_element=’item’, root_element=’items’, \*kwargs*)

    Exports items in XML format to the specified file object.

    • Parameters

      • file – the file-like object to use for exporting the data. Its write method should accept (a disk file opened in binary mode, a io.BytesIO object, etc)

      • root_element (str) – The name of root element in the exported XML.

      • item_element () – The name of each item element in the exported XML.

    The additional keyword arguments of this __init__ method are passed to the BaseItemExporter __init__ method.

    A typical output of this exporter would be:

    1. <?xml version="1.0" encoding="utf-8"?>
    2. <items>
    3.   <item>
    4.     <name>Color TV</name>
    5.  </item>
    6.   <item>
    7.     <name>DVD player</name>
    8.     <price>200</price>
    9.  </item>
    10. </items>

    Unless overridden in the serialize_field() method, multi-valued fields are exported by serializing each value inside a <value> element. This is for convenience, as multi-valued fields are very common.

    For example, the item:

    1. Item(name=['John', 'Doe'], age='23')

    Would be serialized as:

    CsvItemExporter

    class scrapy.exporters.CsvItemExporter(file, include_headers_line=True, join_multivalued=’,’, errors=None, \*kwargs*)[source]

    Exports items in CSV format to the given file-like object. If the fields_to_export attribute is set, it will be used to define the CSV columns and their order. The export_empty_fields attribute has no effect on this exporter.

    • Parameters

      • file – the file-like object to use for exporting the data. Its write method should accept bytes (a disk file opened in binary mode, a io.BytesIO object, etc)

      • include_headers_line () – If enabled, makes the exporter output a header line with the field names taken from BaseItemExporter.fields_to_export or the first exported item fields.

      • join_multivalued – The char (or chars) that will be used for joining multi-valued fields, if found.

    The additional keyword arguments of this __init__ method are passed to the __init__ method, and the leftover arguments to the csv.writer() function, so you can use any function argument to customize this exporter.

    A typical output of this exporter would be:

    1. product,price
    2. Color TV,1200
    3. DVD player,200

    class scrapy.exporters.PickleItemExporter(file, protocol=0, \*kwargs*)[source]

    Exports items in pickle format to the given file-like object.

    • Parameters

      • file – the file-like object to use for exporting the data. Its method should accept bytes (a disk file opened in binary mode, a io.BytesIO object, etc)

      • protocol () – The pickle protocol to use.

    For more information, see pickle.

    The additional keyword arguments of this __init__ method are passed to the __init__ method.

    Pickle isn’t a human readable format, so no output examples are provided.

    PprintItemExporter

    class scrapy.exporters.PprintItemExporter(file, \*kwargs*)

    Exports items in pretty print format to the specified file object.

    • Parameters

      file – the file-like object to use for exporting the data. Its write method should accept bytes (a disk file opened in binary mode, a io.BytesIO object, etc)

    The additional keyword arguments of this __init__ method are passed to the BaseItemExporter __init__ method.

    A typical output of this exporter would be:

    1. {'name': 'Color TV', 'price': '1200'}
    2. {'name': 'DVD player', 'price': '200'}

    Longer lines (when present) are pretty-formatted.

    JsonItemExporter

    class scrapy.exporters.JsonItemExporter(file, \*kwargs*)[source]

    Exports items in JSON format to the specified file-like object, writing all objects as a list of objects. The additional __init__ method arguments are passed to the __init__ method, and the leftover arguments to the JSONEncoder __init__ method, so you can use any __init__ method argument to customize this exporter.

    • Parameters

      file – the file-like object to use for exporting the data. Its write method should accept bytes (a disk file opened in binary mode, a io.BytesIO object, etc)

    A typical output of this exporter would be:

    Warning

    JSON is very simple and flexible serialization format, but it doesn’t scale well for large amounts of data since incremental (aka. stream-mode) parsing is not well supported (if at all) among JSON parsers (on any language), and most of them just parse the entire object in memory. If you want the power and simplicity of JSON with a more stream-friendly format, consider using JsonLinesItemExporter instead, or splitting the output in multiple chunks.

    class scrapy.exporters.JsonLinesItemExporter(file, \*kwargs*)

    Exports items in JSON format to the specified file-like object, writing one JSON-encoded item per line. The additional __init__ method arguments are passed to the BaseItemExporter __init__ method, and the leftover arguments to the __init__ method, so you can use any JSONEncoder __init__ method argument to customize this exporter.

    • Parameters

      file – the file-like object to use for exporting the data. Its write method should accept bytes (a disk file opened in binary mode, a io.BytesIO object, etc)

    A typical output of this exporter would be:

    1. {"name": "Color TV", "price": "1200"}
    2. {"name": "DVD player", "price": "200"}

    Unlike the one produced by , the format produced by this exporter is well suited for serializing large amounts of data.

    MarshalItemExporter

    class scrapy.exporters.MarshalItemExporter(file, \*kwargs*)

    Exports items in a Python-specific binary format (see marshal).

    • Parameters