Installation and deployment

Doris, as an open source MPP architecture OLAP database, can run on most mainstream commercial servers. In order to make full use of the concurrency advantages of MPP architecture and the high availability features of Doris, we recommend that the deployment of Doris follow the following requirements:

Linux Operating System Version Requirements

Software requirements

OS Installation Requirements

Set the maximum number of open file handles in the system

Clock synchronization

The metadata of Doris requires the time precision to be less than 5000ms, so all machines in the cluster need to synchronize the clocks to avoid service exceptions caused by inconsistencies in metadata caused by clock problems.

Close the swap partition (swap)

The Linux swap partition will cause serious performance problems for Doris, you need to disable the swap partition before installation

Linux file system

Here we recommend using the ext4 file system. When installing the operating system, please select the ext4 file system.

Development Test Environment

Production environment

• Usually we recommend about 10 to 100 machines to give full play to Doris’s performance (3 of them deploy FE (HA) and the rest deploy BE)
• Of course, Doris performance is positively correlated with the number and configuration of nodes. With a minimum of four machines (one FE, three BEs, one BE mixed with one Observer FE to provide metadata backup) and a lower configuration, Doris can still run smoothly.
• If FE and BE are mixed, we should pay attention to resource competition and ensure that metadata catalogue and data catalogue belong to different disks.

Broker deployment

Broker is a process for accessing external data sources, such as hdfs. Usually, a broker instance is deployed on each machine.

Network Requirements

Doris instances communicate directly over the network. The following table shows all required ports

IP binding

Because of the existence of multiple network cards, or the existence of virtual network cards caused by the installation of docker and other environments, the same host may have multiple different ips. Currently Doris does not automatically identify available IP. So when you encounter multiple IP on the deployment host, you must force the correct IP to be specified through the priority_networks configuration item.

Priority_networks is a configuration that both FE and BE have, and the configuration items need to be written in fe.conf and be.conf. This configuration item is used to tell the process which IP should be bound when FE or BE starts. Examples are as follows:

This is a representation of CIDR. FE or BE will find the matching IP based on this configuration item as their own local IP.

Note: When priority networks is configured and FE or BE is started, only the correct IP binding of FE or BE is ensured. In ADD BACKEND or ADD FRONTEND statements, you also need to specify IP matching priority networks configuration, otherwise the cluster cannot be established. Give an example:

BE is configured as priority_networks = 10.1.3.0/24'..

When you want to ADD BACKEND use: ALTER SYSTEM ADD BACKEND "192.168.0.1:9050";

Then FE and BE will not be able to communicate properly.

At this point, DROP must remove the BE that added errors and re-use the correct IP to perform ADD BACKEND.

FE is the same.

BROKER does not currently have, nor does it need, priority_networks. Broker’s services are bound to 0.0.0 by default. Simply execute the correct accessible BROKER IP when ADD BROKER is used.

Table Name Case Sensitivity Setting

By default, doris is case-sensitive. If there is a need for case-insensitive table names, you need to set it before cluster initialization. The table name case sensitivity cannot be changed after cluster initialization is completed.

See the section on lower_case_table_names variables in Variables for details.

Deploy FE

• Copy the FE deployment file to the specified node

  Copy the Fe folder under output generated by source code compilation to the node specified deployment path of FE and enter this directory.

• Configure FE

  1. The configuration file is conf/fe.conf. Note: meta_dir indicates the Metadata storage location. The default value is ${DORIS_HOME}/doris-meta. The directory needs to be created manually.

• Start FE

  bin/start_fe.sh --daemon

  The FE process starts and enters the background execution. Logs are stored in the log/ directory by default. If startup fails, you can view error messages by looking at log/fe.log or log/fe.out.

• For deployment of multiple FEs, see the section “FE scaling and downsizing”

Deploy BE

• Copy BE deployment files to all nodes to deploy BE

  Copy the be folder under output generated by source code compilation to the specified deployment path of the BE node.

• Modify all BE configurations

  Modify be/conf/be.conf. Mainly configure storage_root_path: data storage directory. The default is be/storage, this directory needs to be created manually by. In multi directories case, using ; separation (do not add ; after the last directory).
  If the user does not use a mix of SSD and HDD disks, they do not need to configure the configuration methods in Example 1 and Example 2 below, but only need to specify the storage directory; they also do not need to modify the default storage media configuration of FE.

  eg.1:

  Note: For SSD disks, ‘.SSD ‘is followed by the directory, and for HDD disks,’.HDD ‘is followed by the directory

  storage_root_path=/home/disk1/doris.HDD;/home/disk2/doris.SSD;/home/disk2/doris

  instructions

  • 1./home/disk1/doris.HDD : The storage medium is HDD;

  • 3./home/disk2/doris : The storage medium is HDD(default)

    eg.2:

    Note: you do not need to add the suffix to either HDD or SSD disk directories. You only need to set the medium parameter

    storage_root_path=/home/disk1/doris,medium:hdd;/home/disk2/doris,medium:ssd

    instructions

• BE webserver_port configuration

  If the Be componet is installed in hadoop cluster , need to change configuration webserver_port=8040 to avoid port used.

• Add all BE nodes to FE

  BE nodes need to be added in FE before they can join the cluster. You can use mysql-client() to connect to FE:

  ./mysql-client -h fe_host -P query_port -uroot

  The fe_host is the node IP where FE is located; the query_port in fe/conf/fe.conf; the root account is used by default and no password is used to login.

  After login, execute the following commands to add each BE:

  ALTER SYSTEM ADD BACKEND "be_host:heartbeat_service_port";

  The be_host is the node IP where BE is located; the heartbeat_service_port in be/conf/be.conf.

• Start BE

  The BE process will start and go into the background for execution. Logs are stored in be/log/directory by default. If startup fails, you can view error messages by looking at be/log/be.log or be/log/be.out.

• View BE status

  Connect to FE using mysql-client and execute SHOW PROC '/backends'; View BE operation. If everything is normal, the isAlivecolumn should be true.

(Optional) FS_Broker deployment

Broker is deployed as a plug-in, independent of Doris. If you need to import data from a third-party storage system, you need to deploy the corresponding Broker. By default, it provides fs_broker to read HDFS ,Object storage (supporting S3 protocol). Fs_broker is stateless and it is recommended that each FE and BE node deploy a Broker.

• Copy the corresponding Broker directory in the output directory of the source fs_broker to all the nodes that need to be deployed. It is recommended to maintain the same level as the BE or FE directories.

• Modify the corresponding Broker configuration

  In the corresponding broker/conf/directory configuration file, you can modify the corresponding configuration.

• Start Broker

  bin/start_broker.sh --daemon

• Add Broker

  To let Doris FE and BE know which nodes Broker is on, add a list of Broker nodes by SQL command.

  Use mysql-client to connect the FE started, and execute the following commands:

  ALTER SYSTEM ADD BROKER broker_name "broker_host1:broker_ipc_port1","broker_host2:broker_ipc_port2",...;

  The broker_host is Broker’s node ip; the broker_ipc_port is in the Broker configuration file.

• View Broker status

  Connect any booted FE using mysql-client and execute the following command to view Broker status: SHOW PROC '/brokers';

Note: In production environments, daemons should be used to start all instances to ensure that processes are automatically pulled up after they exit, such as . For daemon startup, in 0.9.0 and previous versions, you need to modify the start_xx.sh scripts to remove the last & symbol. Starting with version 0.10.0, call sh start_xx.sh directly to start. Also refer to here

1. How to determine the success of FE process startup

   After the FE process starts, metadata is loaded first. According to the different roles of FE, you can see transfer from UNKNOWN to MASTER/FOLLOWER/OBSERVERin the log. Eventually, you will see the thrift server started log and connect to FE through MySQL client, which indicates that FE started successfully.

   You can also check whether the startup was successful by connecting as follows:

   http://fe_host:fe_http_port/api/bootstrap

   If returned:

   The startup is successful, there may be problems in other cases.

2. How to determine the success of BE process startup

   After the BE process starts, if there is data before, there may be several minutes of data index loading time.

   If BE is started for the first time or the BE has not joined any cluster, the BE log will periodically scroll the words waiting to receive first heartbeat from frontend. BE has not received Master’s address through FE’s heartbeat and is waiting passively. This error log will disappear after ADD BACKEND in FE sends the heartbeat. If the word master client', get client from cache failed. host:, port: 0, code: 7 master client appears again after receiving heartbeat, it indicates that FE has successfully connected BE, but BE cannot actively connect FE. It may be necessary to check the connectivity of rpc_port from BE to FE.

   If BE has been added to the cluster, the heartbeat log from FE should be scrolled every five seconds: get heartbeat, host:xx. xx.xx.xx, port:9020, cluster id:xxxxxxx, indicating that the heartbeat is normal.

   Secondly, the word should be scrolled every 10 seconds in the log to indicate that BE’s communication to FE is normal.

   At the same time, if there is a data query, you should see the rolling logs, and have execute time is xxx logs, indicating that BE started successfully, and the query is normal.

   You can also check whether the startup was successful by connecting as follows:

   http://be_host:be_http_port/api/health

   If returned:

   {"status": "OK","msg": "To Be Added"}

   If the startup is successful, there may be problems in other cases.

3. How to determine the normal connectivity of FE and BE after building the system

   Firstly, confirm that FE and BE processes have been started separately and normally, and confirm that all nodes have been added through ADD BACKEND or ADD FOLLOWER/OBSERVER statements.

   If the heartbeat is normal, BE logs will show get heartbeat, host:xx.xx.xx.xx, port:9020, cluster id:xxxxx If the heartbeat fails, the words backend [10001] get Exception: org.apache.thrift.transport.TTransportException will appear in FE’s log, or other thrift communication abnormal log, indicating that the heartbeat fails from FE to 10001 BE. Here you need to check the connectivity of FE to BE host’s heart-beating port.

   If BE’s communication to FE is normal, the BE log will display the words finish report task success. return code: 0. Otherwise, the words master client, get client from cache failed` will appear. In this case, the connectivity of BE to the rpc_port of FE needs to be checked.

4. File descriptor number of BE process

   The number of file descriptor of BE process is controlled by the two parameters min_file_descriptor_number/max_file_descriptor_number.

   If it is not in the [min_file_descriptor_number, max_file_descriptor_number] interval, error will occurs when starting BE process.

   Please using ulimit command to set file descriptor under this circumstance.

   The default value of min_file_descriptor_number is 65536.

   The default value of max_file_descriptor_number is 131072.

   For Example: ulimit -n 65536; this command set file descriptor to 65536.

   After starting BE process, you can use cat /proc/$pid/limits to see the actual limit of process.

   if use , try to modify minfds in supervisord.conf