Custom Type Example

    We’ll start by getting a clean database to use for the example:

    Since the purpose of the example is to demonstrate working with custom types, we’ll need a custom data type to use. For this example, we will be working with the Decimal type from Python’s standard library. Since the BSON library’s type (that implements the IEEE 754 decimal128 decimal-based floating-point numbering format) is distinct from Python’s built-in Decimal type, attempting to save an instance of Decimal with PyMongo, results in an exception.

    1. >>> from decimal import Decimal
    2. >>> num = Decimal("45.321")
    3. >>> db.test.insert_one({'num': num})
    4. Traceback (most recent call last):
    5. ...
    6. bson.errors.InvalidDocument: cannot encode object: Decimal('45.321'), of type: <class 'decimal.Decimal'>

    New in version 3.8.

    In order to encode a custom type, we must first define a type codec for that type. A type codec describes how an instance of a custom type can be transformed to and/or from one of the types bson already understands. Depending on the desired functionality, users must choose from the following base classes when defining type codecs:

    • : subclass this to define a codec that encodes a custom Python type to a known BSON type. Users must implement the python_type property/attribute and the transform_python method.
    • TypeDecoder: subclass this to define a codec that decodes a specified BSON type into a custom Python type. Users must implement the bson_type property/attribute and the transform_bson method.
    • : subclass this to define a codec that can both encode and decode a custom type. Users must implement the python_type and bson_type properties/attributes, as well as the transform_python and transform_bson methods.

    The type codec for our custom type simply needs to define how a Decimal instance can be converted into a instance and vice-versa. Since we are interested in both encoding and decoding our custom type, we use the TypeCodec base class to define our codec:

    1. >>> from bson.decimal128 import Decimal128
    2. >>> from bson.codec_options import TypeCodec
    3. >>> class DecimalCodec(TypeCodec):
    4. ...     python_type = Decimal    # the Python type acted upon by this type codec
    5. ...     bson_type = Decimal128   # the BSON type acted upon by this type codec
    6. ...     def transform_python(self, value):
    7. ...         """Function that transforms a custom type value into a type
    8. ...         that BSON can encode."""
    9. ...         return Decimal128(value)
    10. ...     def transform_bson(self, value):
    11. ...         """Function that transforms a vanilla BSON type value into our
    12. ...         custom type."""
    13. ...         return value.to_decimal()
    14. >>> decimal_codec = DecimalCodec()

    New in version 3.8.

    Before we can begin encoding and decoding our custom type objects, we must first inform PyMongo about the corresponding codec. This is done by creating a TypeRegistry instance:

    1. >>> from bson.codec_options import TypeRegistry
    2. >>> type_registry = TypeRegistry([decimal_codec])

    Note that type registries can be instantiated with any number of type codecs. Once instantiated, registries are immutable and the only way to add codecs to a registry is to create a new one.

    Finally, we can define a instance with our type_registry and use it to get a Collection object that understands the data type:

    1. >>> from bson.codec_options import CodecOptions
    2. >>> codec_options = CodecOptions(type_registry=type_registry)
    3. >>> collection = db.get_collection('test', codec_options=codec_options)

    Now, we can seamlessly encode and decode instances of Decimal:

    1. >>> collection.insert_one({'num': Decimal("45.321")})
    2. >>> mydoc = collection.find_one()
    3. >>> import pprint
    4. >>> pprint.pprint(mydoc)
    5. {u'_id': ObjectId('...'), u'num': Decimal('45.321')}

    We can see what’s actually being saved to the database by creating a fresh collection object without the customized codec options and using that to query MongoDB:

    Consider the situation where, in addition to encoding , we also need to encode a type that subclasses Decimal. PyMongo does this automatically for types that inherit from Python types that are BSON-encodable by default, but the type codec system described above does not offer the same flexibility.

    1. >>> class DecimalInt(Decimal):
    2. ...     def my_method(self):
    4. ...         return int(self)

    If we try to save an instance of this type without first registering a type codec for it, we get an error:

    1. >>> collection.insert_one({'num': DecimalInt("45.321")})
    2. Traceback (most recent call last):
    3. ...
    4. bson.errors.InvalidDocument: cannot encode object: Decimal('45.321'), of type: <class 'decimal.Decimal'>

    In order to proceed further, we must define a type codec for DecimalInt. This is trivial to do since the same transformation as the one used for Decimal is adequate for encoding DecimalInt as well:

    1. >>> class DecimalIntCodec(DecimalCodec):
    2. ...     @property
    3. ...     def python_type(self):
    4. ...         """The Python type acted upon by this type codec."""
    5. ...         return DecimalInt
    6. >>> decimalint_codec = DecimalIntCodec()

    Note

    No attempt is made to modify decoding behavior because without additional information, it is impossible to discern which incoming Decimal128 value needs to be decoded as Decimal and which needs to be decoded as DecimalInt. This example only considers the situation where a user wants to encode documents containing either of these types.

    After creating a new codec options object and using it to get a collection object, we can seamlessly encode instances of DecimalInt:

    1. >>> type_registry = TypeRegistry([decimal_codec, decimalint_codec])
    2. >>> codec_options = CodecOptions(type_registry=type_registry)
    3. >>> collection = db.get_collection('test', codec_options=codec_options)
    4. >>> collection.drop()
    5. >>> collection.insert_one({'num': DecimalInt("45.321")})
    6. <pymongo.results.InsertOneResult object at ...>
    7. >>> mydoc = collection.find_one()
    8. >>> pprint.pprint(mydoc)
    9. {u'_id': ObjectId('...'), u'num': Decimal('45.321')}

    Note that the transform_bson method of the base codec class results in these values being decoded as Decimal (and not DecimalInt).

    The decoding treatment of types having subtype = 0 by the bson module varies slightly depending on the version of the Python runtime in use. This must be taken into account while writing a TypeDecoder that modifies how this datatype is decoded.

    On Python 3.x, data (subtype = 0) is decoded as a bytes instance:

    1. >>> # On Python 3.x.
    2. >>> from bson.binary import Binary
    3. >>> newcoll = db.get_collection('new')
    4. >>> newcoll.insert_one({'_id': 1, 'data': Binary(b"123", subtype=0)})
    5. >>> doc = newcoll.find_one()
    6. >>> type(doc['data'])
    7. bytes

    On Python 2.7.x, the same data is decoded as a Binary instance:

    As a consequence of this disparity, users must set the bson_type attribute on their classes differently, depending on the python version in use.

    Note

    For codebases requiring compatibility with both Python 2 and 3, type decoders will have to be registered for both possible bson_type values.

    In addition to type codecs, users can also register a callable to encode types that BSON doesn’t recognize and for which no type codec has been registered. This callable is the fallback encoder and like the transform_python method, it accepts an unencodable value as a parameter and returns a BSON-encodable value. The following fallback encoder encodes python’s Decimal type to a :

    1. >>> def fallback_encoder(value):
    2. ...     if isinstance(value, Decimal):
    4. ...     return value

    After declaring the callback, we must create a type registry and codec options with this fallback encoder before it can be used for initializing a collection:

    1. >>> codec_options = CodecOptions(type_registry=type_registry)
    2. >>> collection = db.get_collection('test', codec_options=codec_options)
    3. >>> collection.drop()

    We can now seamlessly encode instances of Decimal:

    1. >>> collection.insert_one({'num': Decimal("45.321")})
    2. <pymongo.results.InsertOneResult object at ...>
    3. >>> mydoc = collection.find_one()
    4. >>> pprint.pprint(mydoc)
    5. {u'_id': ObjectId('...'), u'num': Decimal128('45.321')}

    Note

    Fallback encoders are invoked after attempts to encode the given value with standard BSON encoders and any configured type encoders have failed. Therefore, in a type registry configured with a type encoder and fallback encoder that both target the same custom type, the behavior specified in the type encoder will prevail.

    Because fallback encoders don’t need to declare the types that they encode beforehand, they can be used to support interesting use-cases that cannot be serviced by TypeEncoder. One such use-case is described in the next section.

    In this example, we demonstrate how a fallback encoder can be used to save arbitrary objects to the database. We will use the the standard library’s module to serialize the unknown types and so naturally, this approach only works for types that are picklable.

    We start by defining some arbitrary custom types:

    1. class MyStringType(object):
    2.     def __init__(self, value):
    3.         self.__value = value
    4.     def __repr__(self):
    5.         return "MyStringType('%s')" % (self.__value,)
    7. class MyNumberType(object):
    8.     def __init__(self, value):
    9.         self.__value = value
    10.     def __repr__(self):
    11.         return "MyNumberType(%s)" % (self.__value,)

    We also define a fallback encoder that pickles whatever objects it receives and returns them as Binary instances with a custom subtype. The custom subtype, in turn, allows us to write a TypeDecoder that identifies pickled artifacts upon retrieval and transparently decodes them back into Python objects:

    1. import pickle
    2. from bson.binary import Binary, USER_DEFINED_SUBTYPE
    3. def fallback_pickle_encoder(value):
    4.     return Binary(pickle.dumps(value), USER_DEFINED_SUBTYPE)
    6. class PickledBinaryDecoder(TypeDecoder):
    7.     bson_type = Binary
    8.     def transform_bson(self, value):
    9.         if value.subtype == USER_DEFINED_SUBTYPE:
    10.             return pickle.loads(value)
    11.         return value

    Note

    The above example is written assuming the use of Python 3. If you are using Python 2, bson_type must be set to Binary. See the section for a detailed explanation.

    Finally, we create a CodecOptions instance:

    1. collection = db.get_collection('test_fe', codec_options=codec_options)
    2. collection.insert_one({'_id': 1, 'str': MyStringType("hello world"),
    3.                        'num': MyNumberType(2)})
    4. mydoc = collection.find_one()
    5. assert isinstance(mydoc['str'], MyStringType)
    6. assert isinstance(mydoc['num'], MyNumberType)

    PyMongo’s type codec and fallback encoder features have the following limitations:

    1. Users cannot customize the encoding behavior of Python types that PyMongo already understands like int and str (the ‘built-in types’). Attempting to instantiate a type registry with one or more codecs that act upon a built-in type results in a TypeError. This limitation extends to all subtypes of the standard types.
    2. Chaining type encoders is not supported. A custom type value, once transformed by a codec’s transform_python method, must result in a type that is either BSON-encodable by default, or can be transformed by the fallback encoder into something BSON-encodable–it cannot be transformed a second time by a different type codec.
    3. gridfs does not apply custom type encoding or decoding to any documents received from or to returned to the user.