MLeap Spark Integration

• Serialization/Deserialization of Transformers and Pipelines to and from Bundle.ML
• Support of additional feature transformers and models (ex: SVM, OneVsRest, MapTransform)
• Support for custom transformers

To use MLeap you do not have to change how you construct your existing pipelines, so the rest of the documentation is going to focus on how to serialize and deserialize your pipeline to and from bundle.ml.To see how to execute your pipeline outside of Spark, refer to the MLeap Runtime section.

Serializing with Spark

In order to serialize to a zip file, make sure the URI begins with and ends with a .zip.

implicit val context = SparkBundleContext().withDataset(sparkTransformed)
for(bundle <- managed(BundleFile("jar:file:/tmp/mleap-examples/simple-json.zip"))) {
}

Protobuf Format

In order to serialize to a directory, make sure the URI begins withfile.

  pipeline.writeBundle.format(SerializationFormat.Json).save(bundle)(context)
}

Protobuf Format

Deserializing is just as easy as serializing. You don’t need to know theformat the MLeap Bundle was serialized as beforehand, you just need toknow where the bundle is.

// Deserialize a zip bundle
// Use Scala ARM to make sure resources are managed properly
val zipBundle = (for(bundle <- managed(BundleFile("jar:file:/tmp/mleap-examples/simple-json.zip"))) yield {
}).opt.get

Directory Bundle