Deploying NATS with Helm

    2.   image: nats:2.4.0
    3.   pullPolicy: IfNotPresent

    Limits

    1. nats:
    2.   # The number of connect attempts against discovered routes.
    3.   connectRetries: 30
    5.   # How many seconds should pass before sending a PING
    6.   # to a client that has no activity.
    7.   pingInterval: 
    9.   # Server settings.
    10.   limits:
    11.     maxConnections: 
    12.     maxSubscriptions: 
    13.     maxControlLine: 
    14.     maxPayload: 
    16.     writeDeadline: 
    17.     maxPending: 
    18.     maxPings: 
    19.     lameDuckDuration: 
    21.   # Number of seconds to wait for client connections to end after the pod termination is requested
    22.   terminationGracePeriodSeconds: 60

    Logging

    Note: It is not recommended to enable trace or debug in production since enabling it will significantly degrade performance.

    1. nats:
    2.   logging:
    3.     debug: 
    4.     trace: 
    5.     logtime: 
    6.     connectErrorReports: 
    7.     reconnectErrorReports:

    TLS setup for client connections

    You can find more on how to set up and troubleshoot TLS connections at: 

    1. nats:
    2.   tls:
    3.     secret:
    4.       name: nats-client-tls
    5.     ca: "ca.crt"
    6.     cert: "tls.crt"
    7.     key: "tls.key"

    Clustering

    If clustering is enabled, then a 3-node cluster will be set up. More info at: 

    1. cluster:
    2.   enabled: true
    3.   replicas: 3
    5.   tls:
    6.     secret:
    7.       name: nats-server-tls
    8.     ca: "ca.crt"
    9.     cert: "tls.crt"
    10.     key: "tls.key"

    Example:

    1. $ helm install nats nats/nats --set cluster.enabled=true

    Leafnodes

    Leafnode connections to extend a cluster. More info at: 

    1. leafnodes:
    2.   enabled: true
    3.   remotes:
    4.     - url: "tls://connect.ngs.global:7422"
    5.       # credentials:
    6.       #   secret:
    7.       #     name: leafnode-creds
    8.       #     key: TA.creds
    9.       # tls:
    10.       #   secret:
    11.       #     name: nats-leafnode-tls
    12.       #   ca: "ca.crt"
    13.       #   cert: "tls.crt"
    14.       #   key: "tls.key"
    16.   #######################
    17.   #                     #
    18.   #  TLS Configuration  #
    19.   #                     #
    20.   #######################
    21.   # 
    22.   #  # You can find more on how to setup and trouble shoot TLS connnections at:
    23.   # 
    24.   #  # https://docs.nats.io/nats-server/configuration/securing_nats/tls
    25.   # 
    26.   tls:
    27.     secret:
    28.       name: nats-client-tls
    29.     ca: "ca.crt"
    30.     cert: "tls.crt"
    31.     key: "tls.key"
    1. websocket:
    2.   enabled: true
    3.   port: 443
    5.   tls:
    6.     secret:
    7.       name: nats-tls
    8.     cert: "fullchain.pem"
    9.     key: "privkey.pem"

    Setting up External Access

    Using HostPorts

    In case of both external access and advertisements being enabled, an initializer container will be used to gather the public IPs. This container will be required to have enough RBAC policy to be able to make a look up of the public IP of the node where it is running.

    For example, to set up external access for a cluster and advertise the public IP to clients:

    1. nats:
    2.   # Toggle whether to enable external access.
    3.   # This binds a host port for clients, gateways and leafnodes.
    4.   externalAccess: true
    6.   # Toggle to disable client advertisements (connect_urls),
    7.   # in case of running behind a load balancer (which is not recommended)
    8.   # it might be required to disable advertisements.
    9.   advertise: true
    11.   # In case both external access and advertise are enabled
    12.   # gather the public IP from a node.
    13.   serviceAccount: "nats-server"

    Where the service account named nats-server has the following RBAC policy for example:

    1. ---
    2. apiVersion: v1
    3. kind: ServiceAccount
    4. metadata:
    5.   name: nats-server
    6.   namespace: default
    7. ---
    8. apiVersion: rbac.authorization.k8s.io/v1
    9. kind: ClusterRole
    10. metadata:
    11.   name: nats-server
    12. rules:
    13. - apiGroups: [""]
    14.   resources:
    15.   - nodes
    16.   verbs: ["get"]
    17. ---
    18. apiVersion: rbac.authorization.k8s.io/v1
    19. kind: ClusterRoleBinding
    21.   name: nats-server-binding
    22. roleRef:
    23.   apiGroup: rbac.authorization.k8s.io
    24.   kind: ClusterRole
    25.   name: nats-server
    26. subjects:
    27. - kind: ServiceAccount
    28.   name: nats-server
    29.   namespace: default

    Using LoadBalancers

    When using a load balancer for external access, it is recommended to disable no advertise so that internal IPs from the NATS Servers are not advertised to the clients connecting through the load balancer.

    1. nats:
    2.   image: nats:alpine
    4. cluster:
    5.   enabled: true
    6.   noAdvertise: true
    8. leafnodes:
    9.   enabled: true
    10.   noAdvertise: true
    12. natsbox:
    13.   enabled: true

    You could then use an L4 enabled load balancer to connect to NATS, for example:

    1. apiVersion: v1
    2. kind: Service
    3. metadata:
    4.   name: nats-lb
    5. spec:
    6.   type: LoadBalancer
    7.   selector:
    8.     app.kubernetes.io/name: nats
    9.   ports:
    10.     - protocol: TCP
    11.       port: 4222
    12.       targetPort: 4222
    13.       name: nats
    14.     - protocol: TCP
    15.       port: 7422
    16.       targetPort: 7422
    17.       name: leafnodes
    18.     - protocol: TCP
    19.       port: 7522
    20.       targetPort: 7522
    21.       name: gateways

    Gateways

    A supercluster can be formed by pointing to remote gateways. You can find more about gateways in the NATS documentation: nats-server/configuration/gateways.

    1. gateway:
    2.   enabled: false
    3.   name: 'default'
    5.   #############################
    6.   #                           #
    7.   #  List of remote gateways  #
    8.   #                           #
    9.   #############################
    10.   # gateways:
    11.   #   - name: other
    12.   #     url: nats://my-gateway-url:7522
    14.   #######################
    15.   #                     #
    16.   #  TLS Configuration  #
    17.   #                     #
    18.   #######################
    19.   # 
    20.   #  # You can find more on how to setup and trouble shoot TLS connnections at:
    21.   # 
    22.   #  # https://docs.nats.io/nats-server/configuration/securing_nats/tls
    23.   #
    24.   # tls:
    25.   #   secret:
    26.   #     name: nats-client-tls
    27.   #   ca: "ca.crt"
    28.   #   cert: "tls.crt"
    29.   #   key: "tls.key"
    1. auth:
    2.   enabled: true
    4.   # Reference to the Operator JWT.
    5.   operatorjwt:
    6.     configMap:
    7.       name: operator-jwt
    8.       key: KO.jwt
    10.   # Public key of the System Account
    11.   systemAccount:
    13.   resolver:
    14.     ############################
    15.     #                          #
    16.     # Memory resolver settings #
    17.     #                          #
    18.     ##############################
    19.     type: memory
    21.     # 
    22.     # Use a configmap reference which will be mounted
    23.     # into the container.
    24.     # 
    25.     configMap:
    26.       name: nats-accounts
    27.       key: resolver.conf

    Auth using an Account Server Resolver

    1. auth:
    2.   enabled: true
    4.   # Reference to the Operator JWT.
    5.   operatorjwt:
    6.     configMap:
    7.       name: operator-jwt
    8.       key: KO.jwt
    10.   # Public key of the System Account
    11.   systemAccount:
    13.     ##########################
    14.     #                        #
    15.     #  URL resolver settings #
    16.     #                        #
    17.     ##########################
    18.     type: URL
    19.     url: "http://nats-account-server:9090/jwt/v1/accounts/"

    JetStream

    Setting up Memory and File Storage

    1. nats:
    2.   image: nats:alpine
    4.   jetstream:
    7.     memStorage:
    8.       enabled: true
    9.       size: 2Gi
    11.     fileStorage:
    12.       enabled: true
    13.       size: 1Gi
    14.       storageDirectory: /data/
    15.       storageClassName: default

    Using with an existing PersistentVolumeClaim

    For example, given the following PersistentVolumeClaim:

    1. ---
    2. kind: PersistentVolumeClaim
    3. apiVersion: v1
    4. metadata:
    5.   name: nats-js-disk
    6.   annotations:
    7.     volume.beta.kubernetes.io/storage-class: "default"
    8. spec:
    9.   accessModes:
    10.     - ReadWriteOnce
    11.   resources:
    12.     requests:
    13.       storage: 3Gi

    You can start JetStream so that one pod is bound to it:

    1. nats:
    2.   image: nats:alpine
    4.   jetstream:
    5.     enabled: true
    7.     fileStorage:
    8.       enabled: true
    9.       storageDirectory: /data/
    10.       existingClaim: nats-js-disk
    11.       claimStorageSize: 3Gi

    Clustering example

    1. nats:
    2.   image: nats:alpine
    4.   jetstream:
    5.     enabled: true
    7.     memStorage:
    8.       enabled: true
    9.       size: "2Gi"
    11.     fileStorage:
    12.       enabled: true
    13.       size: "1Gi"
    14.       storageDirectory: /data/
    15.       storageClassName: default
    17. cluster:
    18.   enabled: true
    19.   # Cluster name is required, by default will be release name.
    20.   # name: "nats"
    21.   replicas: 3

    Misc

    NATS Box

    A lightweight container with NATS and NATS Streaming utilities deployed along the cluster to confirm the setup. You can find the image at: https://github.com/nats-io/nats-box

    1. natsbox:
    2.   enabled: true
    3.   image: nats:alpine
    4.   pullPolicy: IfNotPresent
    6.   # credentials:
    7.   #   secret:
    8.   #     name: nats-sys-creds
    9.   #     key: sys.creds

    The NATS config reloader image to use:

    Prometheus Exporter sidecar

    You can toggle whether to start the sidecar to be used to feed metrics to Prometheus:

    1. exporter:
    2.   enabled: true
    3.   image: natsio/prometheus-nats-exporter:latest
    4.   pullPolicy: IfNotPresent

    Prometheus operator ServiceMonitor support

    1. exporter:
    2.   # You have to enable exporter first
    3.   enabled: true
    4.   serviceMonitor:
    5.     enabled: true
    6.     ## Specify the namespace where Prometheus Operator is running
    7.     # namespace: monitoring
    8.     # ...

    Pod Customizations

    Security Context

    1.  # Toggle whether to use setup a Pod Security Context
    2.  # ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
    3. securityContext:
    4.   fsGroup: 1000
    5.   runAsUser: 1000
    6.   runAsNonRoot: true

    Affinity

    https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity

    matchExpressions must be configured according to your setup

    1. affinity:
    2.   nodeAffinity:
    3.     requiredDuringSchedulingIgnoredDuringExecution:
    4.       nodeSelectorTerms:
    5.         - matchExpressions:
    6.             - key: node.kubernetes.io/purpose
    7.               operator: In
    8.               values:
    9.                 - nats
    10.   podAntiAffinity:
    11.     requiredDuringSchedulingIgnoredDuringExecution:
    12.       - labelSelector:
    13.           matchExpressions:
    14.             - key: app
    15.               operator: In
    16.               values:
    17.                 - nats
    18.                 - stan
    19.         topologyKey: "kubernetes.io/hostname"

    Service topology

    Service topology is disabled by default but can be enabled by setting topologyKeys. For example:

    1. topologyKeys:
    2.   - "kubernetes.io/hostname"
    3.   - "topology.kubernetes.io/zone"
    4.   - "topology.kubernetes.io/region"

    CPU/Memory Resource Requests/Limits

    Sets the pods CPU/memory requests/limits

    1. nats:
    2.   resources:
    3.     requests:
    4.       cpu: 2
    5.       memory: 4Gi
    6.     limits:
    7.       cpu: 4
    8.       memory: 6Gi

    No resources are set by default.

    Annotations

    1. podAnnotations:
    2.   key1 : "value1",
    3.   key2 : "value2"

    Name Overrides

    Can change the name of the resources as needed with:

    1. nameOverride: "my-nats"

    Image Pull Secrets

    1. imagePullSecrets:
    2. - name: myRegistry

    Adds this to the StatefulSet:

    1. spec: