Kubernetes 安装

    Please follow in order to deploy a Kubernetes cluster.If you want to run Kubernetes locally, we recommend using MiniKube.

    Note: If using MiniKube please make sure to execute before deploying a Flink cluster. Otherwise Flink components are not able to self reference themselves through a Kubernetes service.

    A Flink session cluster is executed as a long-running Kubernetes Deployment. Note that you can run multiple Flink jobs on a session cluster.Each job needs to be submitted to the cluster after the cluster has been deployed.

    A basic Flink session cluster deployment in Kubernetes has three components:

    • a Deployment/Job which runs the JobManager
    • a Deployment for a pool of TaskManagers
    • a Service exposing the JobManager’s REST and UI ports

    Using the resource definitions for a , launch the cluster with the kubectl command:

    Note that you could define your own customized options of flink-conf.yaml within flink-configuration-configmap.yaml.

    1. ./bin/flink run -m localhost:8081 ./examples/streaming/WordCount.jar
    • Create a NodePort service on the rest service of jobmanager:
      • Run kubectl create -f jobmanager-rest-service.yaml to create the NodePort service on jobmanager. The example of jobmanager-rest-service.yaml can be found in appendix.
      • Run kubectl get svc flink-jobmanager-rest to know the node-port of this service and navigate to in your browser.
      • Similarly to port-forward solution, you could also use the following command below to submit jobs to the cluster:
    1. ./bin/flink run -m <public-node-ip>:<node-port> ./examples/streaming/WordCount.jar

    In order to terminate the Flink session cluster, use kubectl:

    A Flink job cluster is a dedicated cluster which runs a single job. The job is part of the image and, thus, there is no extra job submission needed.

    The Flink job cluster image needs to contain the user code jars of the job for which the cluster is started.Therefore, one needs to build a dedicated container image for every job.Please follow these instructions to build the Docker image.

    In order to deploy the a job cluster on Kubernetes please follow these .

    The Deployment definitions use the pre-built image flink:latest which can be found on Docker Hub.The image is built from this .

    flink-configuration-configmap.yaml

    1. apiVersion: v1
    2. kind: ConfigMap
    3. metadata:
    4.   name: flink-config
    5.   labels:
    6.     app: flink
    7. data:
    8.   flink-conf.yaml: |+
    9.     jobmanager.rpc.address: flink-jobmanager
    10.     taskmanager.numberOfTaskSlots: 1
    11.     jobmanager.rpc.port: 6123
    12.     taskmanager.rpc.port: 6122
    13.     jobmanager.heap.size: 1024m
    14.     taskmanager.heap.size: 1024m
    15.   log4j.properties: |+
    16.     log4j.rootLogger=INFO, file
    17.     log4j.logger.akka=INFO
    18.     log4j.logger.org.apache.kafka=INFO
    19.     log4j.logger.org.apache.hadoop=INFO
    20.     log4j.logger.org.apache.zookeeper=INFO
    21.     log4j.appender.file=org.apache.log4j.FileAppender
    23.     log4j.appender.file.layout=org.apache.log4j.PatternLayout
    24.     log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n
    25.     log4j.logger.org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, file

    jobmanager-deployment.yaml

    1. apiVersion: extensions/v1beta1
    2. kind: Deployment
    3. metadata:
    4.   name: flink-jobmanager
    5. spec:
    6.   replicas: 1
    7.   template:
    8.     metadata:
    9.       labels:
    10.         app: flink
    11.         component: jobmanager
    12.     spec:
    13.       containers:
    14.       - name: jobmanager
    15.         image: flink:latest
    16.         workingDir: /opt/flink
    17.         command: ["/bin/bash", "-c", "$FLINK_HOME/bin/jobmanager.sh start;\
    18.           while :;
    19.           do
    20.             if [[ -f $(find log -name '*jobmanager*.log' -print -quit) ]];
    21.               then tail -f -n +1 log/*jobmanager*.log;
    22.             fi;
    23.           done"]
    24.         ports:
    25.         - containerPort: 6123
    26.           name: rpc
    27.         - containerPort: 6124
    28.           name: blob
    29.         - containerPort: 8081
    30.         livenessProbe:
    31.           tcpSocket:
    32.             port: 6123
    33.           initialDelaySeconds: 30
    34.           periodSeconds: 60
    35.         volumeMounts:
    36.         - name: flink-config-volume
    38.       volumes:
    39.       - name: flink-config-volume
    40.         configMap:
    41.           name: flink-config
    42.           items:
    43.           - key: flink-conf.yaml
    44.             path: flink-conf.yaml
    45.           - key: log4j.properties
    46.             path: log4j.properties

    taskmanager-deployment.yaml

    jobmanager-service.yaml

    1. apiVersion: v1
    2. kind: Service
    3. metadata:
    4.   name: flink-jobmanager
    5. spec:
    6.   type: ClusterIP
    7.   ports:
    8.   - name: rpc
    9.     port: 6123
    10.   - name: blob
    11.     port: 6124
    12.   - name: ui
    13.     port: 8081
    14.   selector:
    15.     app: flink
    16.     component: jobmanager

    jobmanager-rest-service.yaml. Optional service, that exposes the jobmanager rest port as public Kubernetes node’s port.

    1. apiVersion: v1
    2. kind: Service
    3. metadata:
    4.   name: flink-jobmanager-rest
    5. spec:
    6.   type: NodePort
    7.   ports:
    8.   - name: rest
    9.     port: 8081
    10.     targetPort: 8081
    11.   selector: