System architecture

    When you config to store TSFile on HDFS, your data files will be in distributed storage. The system architecture is as below:

    Config and usage

    To store TSFile and related data files in HDFS, here are the steps:

    Build server and Hadoop module by:

    Then, copy the target jar of Hadoop module hadoop-tsfile-X.X.X-jar-with-dependencies.jar into server target lib folder .../server/target/iotdb-server-X.X.X/lib.

    Edit user config in . Related configurations are:

    • tsfile_storage_fs
    • core_site_path
    Namecore_site_path
    DescriptionAbsolute file path of core-site.xml if Tsfile and related data files are stored in HDFS.
    TypeString
    Default/etc/hadoop/conf/core-site.xml
    EffectiveAfter restart system
    Namehdfs_site_path
    DescriptionAbsolute file path of hdfs-site.xml if Tsfile and related data files are stored in HDFS.
    TypeString
    Default/etc/hadoop/conf/hdfs-site.xml
    EffectiveAfter restart system
    • hdfs_ip
    Namehdfs_ip
    DescriptionIP of HDFS if Tsfile and related data files are stored in HDFS. If there are more than one hdfs_ip in configuration, Hadoop HA is used.
    TypeString
    Defaultlocalhost
    EffectiveAfter restart system
    • hdfs_port
    • dfs_nameservices
    Namehdfs_nameservices
    DescriptionNameservices of HDFS HA if using Hadoop HA
    TypeString
    Defaulthdfsnamespace
    EffectiveAfter restart system
    • dfs_ha_namenodes
    Namehdfs_ha_namenodes
    DescriptionNamenodes under DFS nameservices of HDFS HA if using Hadoop HA
    TypeString
    Defaultnn1,nn2
    EffectiveAfter restart system
    • dfs_ha_automatic_failover_enabled
    Namedfs_ha_automatic_failover_enabled
    DescriptionWhether using automatic failover if using Hadoop HA
    TypeBoolean
    Defaulttrue
    EffectiveAfter restart system
    • dfs_client_failover_proxy_provider
    Namehdfs_use_kerberos
    DescriptionWhether use kerberos to authenticate hdfs
    TypeString
    Defaultfalse
    EffectiveAfter restart system
    • kerberos_keytab_file_path
    Namekerberos_keytab_file_path
    DescriptionFull path of kerberos keytab file
    TypeString
    Default/path
    EffectiveAfter restart system
    • kerberos_principal
    Namekerberos_principal
    DescriptionKerberos pricipal
    TypeString
    Defaultyour principal
    EffectiveAfter restart system

    To reset storage file system to local, just edit configuration tsfile_storage_fs to LOCAL. In this situation, if data files are already on HDFS, you should either download them to local and move them to your config data file folder ( by default), or restart your process and import data to IoTDB.

    Frequent questions

    1. What Hadoop version does it support?

    A: Both Hadoop 2.x and Hadoop 3.x can be supported.

    1. When starting the server or trying to create timeseries, I encounter the error below:

    A: It indicates that you forget to put Hadoop module dependency in IoTDB server. You can solve it by:

    • Build Hadoop module: mvn clean package -pl hadoop -am -Dmaven.test.skip=true -P get-jar-with-dependencies
    • Copy the target jar of Hadoop module hadoop-tsfile-X.X.X-jar-with-dependencies.jar into server target lib folder .