With this connector, you can

• load a single TsFile, from either the local file system or hdfs, into Hadoop
• load all files in a specific directory, from either the local file system or hdfs, into hadoop
• write data from Hadoop into TsFile

TSFInputFormat extract data from tsfile and format them into records of MapWritable.

Suppose that we want to extract data of the device named d1 which has three sensors named s1, s2, s3.

The MapWritable struct will be like:

In the Map job of Hadoop, you can get any value you want by key as following:

mapwritable.get(new Text("s1"))

Read Example: calculate the sum

    // configure reading time enable
    TSFInputFormat.setReadTime(job, true); 
    // configure reading deviceId enable
    TSFInputFormat.setReadDeviceId(job, true); 
    // configure reading which deltaObjectIds
    String[] deviceIds = {"device_1"};
    TSFInputFormat.setReadDeviceIds(job, deltaObjectIds);
    // configure reading which measurementIds
    String[] measurementIds = {"sensor_1", "sensor_2", "sensor_3"};
    TSFInputFormat.setReadMeasurementIds(job, measurementIds);

And then,the output key and value of mapper and reducer should be specified

Then, the mapper and reducer class is how you deal with the MapWritable produced by TSFInputFormat class.

  public static class TSMapper extends Mapper<NullWritable, MapWritable, Text, DoubleWritable> {
    @Override
        Mapper<NullWritable, MapWritable, Text, DoubleWritable>.Context context)
        throws IOException, InterruptedException {
      Text deltaObjectId = (Text) value.get(new Text("device_id"));
    }
  }
  public static class TSReducer extends Reducer<Text, DoubleWritable, Text, DoubleWritable> {
    @Override
    protected void reduce(Text key, Iterable<DoubleWritable> values,
        Reducer<Text, DoubleWritable, Text, DoubleWritable>.Context context)
        throws IOException, InterruptedException {
      double sum = 0;
      for (DoubleWritable value : values) {
        sum = sum + value.get();
      }
      context.write(key, new DoubleWritable(sum));
    }
  }

Write Example: write the average into Tsfile

Except for the OutputFormatClass, the rest of configuration code for hadoop map-reduce job is almost same as above.

    public static class TSMapper extends Mapper<NullWritable, MapWritable, Text, MapWritable> {
        @Override
        protected void map(NullWritable key, MapWritable value,
                           Mapper<NullWritable, MapWritable, Text, MapWritable>.Context context)
                throws IOException, InterruptedException {
            Text deltaObjectId = (Text) value.get(new Text("device_id"));
            if (timestamp % 100000 == 0) {
                context.write(deltaObjectId, new MapWritable(value));
            }
    }
    /**
     * This reducer calculate the average value.
     */
    public static class TSReducer extends Reducer<Text, MapWritable, NullWritable, HDFSTSRecord> {
        @Override
        protected void reduce(Text key, Iterable<MapWritable> values,
                              Reducer<Text, MapWritable, NullWritable, HDFSTSRecord>.Context context) throws IOException, InterruptedException {
            long sensor1_value_sum = 0;
            long sensor2_value_sum = 0;
            double sensor3_value_sum = 0;
            long num = 0;
            for (MapWritable value : values) {
                num++;
                sensor1_value_sum += ((LongWritable)value.get(new Text("sensor_1"))).get();
                sensor2_value_sum += ((LongWritable)value.get(new Text("sensor_2"))).get();
                sensor3_value_sum += ((DoubleWritable)value.get(new Text("sensor_3"))).get();
            }
            HDFSTSRecord tsRecord = new HDFSTSRecord(1L, key.toString());
            DataPoint dPoint1 = new LongDataPoint("sensor_1", sensor1_value_sum / num);
            DataPoint dPoint2 = new LongDataPoint("sensor_2", sensor2_value_sum / num);
            DataPoint dPoint3 = new DoubleDataPoint("sensor_3", sensor3_value_sum / num);
            tsRecord.addTuple(dPoint1);
            tsRecord.addTuple(dPoint2);
            tsRecord.addTuple(dPoint3);
            context.write(NullWritable.get(), tsRecord);
    }