Troubleshooting guide

    See also:

    Chaining / workflows

    When chaning or invoking another function, do this by calling it via the gateway either using the external IP address of the cluster / gateway, or by the internal DNS name.

    A common user-error is to try to invoke the gateway from within a function using or localhost - this address simply points back to the container the function is running in and is invalid.

    The default address for the gateway on Kubernetes is http://gateway.openfaas:8080.

    On Docker Swarm

    The default address for the gateway is

    Timeouts

    Default timeouts are configured at the HTTP level and must be set both on the gateway and the function.

    Timeouts - Your function

    Default read_timeout and write_timeout value for functions are 5 seconds (subject to change) and can be overridden using read_timeout and write_timeout environment variable for each function.

    You can also enforce a hard-timeout for your function with the hard_timeout environmental variable.

    For watchdog configuration see the README.

    The best way to set the timeout is in the YAML file generated by the faas-cli.

    Example Go app that sleeps for (10 seconds):

    handler.go

    1. package function
    2.  
    3. ...
    4.  
    5. func Handle(req []byte) string {
    6.     time.Sleep(time.Second * 10)
    7.     return fmt.Sprintf("Hello, Go. You said: %s", string(req))
    8. }

    Timeouts - Gateway

    For the gateway set the following environmental variables:

    1.             read_timeout:  "25s"        # Maximum time to read HTTP request
    2.             upstream_timeout: "20s"     # Maximum duration of upstream function call

    Timeouts - Function provider

    When using a gateway version older than 0.7.8 a timeout matching the gateway should be set for the or faas-netes controller.

    1. read_timeout: 25s
    2. write_timeout: 25s

    For asynchronous invocations of functions a separate timeout can be configured at the queue-worker level in the ack_wait environmental variable.

    Timeouts - Cloud Service Providers

    There are situations where timeout values external to OpenFaaS may impact successful function execution. A typical scenario is where a cloud platform's load balancer product is fronting the cluster in which OpenFaaS is running. A common example is when using the .

    By default the functions will not log out the result, but just show how long the process took to run and the length of the result in bytes.

    1. $ echo test this | faas invoke json-hook -g 127.0.0.1:31112
    2. Received JSON webook. Elements: 10
    3.  
    4. $ kubectl logs deploy/json-hook -n openfaas-fn
    5. 2018/01/28 20:47:21 Writing lock-file to: /tmp/.lock
    6. 2018/01/28 20:47:27 Forking fprocess.
    7. 2018/01/28 20:47:27 Wrote 35 Bytes - Duration: 0.001844 seconds

    If you want to see the result of a function in the function's logs then deploy it with the write_debug environmental variable set to true.

    For example:

    1. provider:
    2.   name: faas
    3.   gateway: http://127.0.0.1:8080
    4.  
    5. functions:
    6.   json-hook:
    7.     lang: go
    8.     handler: ./json-hook
    9.     image: json-hook
    10.     environment:
    11.       write_debug: true

    Now you'll see logs like this:

    1. $ echo test this | faas invoke json-hook -g 127.0.0.1:31112
    2. Received JSON webook. Elements: 10
    3.  
    4. $ kubectl logs deploy/json-hook -n openfaas-fn
    5. 2018/01/28 20:50:27 Writing lock-file to: /tmp/.lock
    6. 2018/01/28 20:50:35 Forking fprocess.
    7. 2018/01/28 20:50:35 Query
    8. 2018/01/28 20:50:35 Path  /function/json-hook
    9. Received JSON webook. Elements: 10
    10. 2018/01/28 20:50:35 Duration: 0.001857 seconds

    You can then find the logs of the function using Docker Swarm or Kubernetes as listed in the section below.

    Healthcheck

    Most problems reported via GitHub or Slack stem from a configuration problem or issue with a function. Here is a checklist of things you can try before digging deeper:

    Checklist:

    • All core services are deployed: i.e. gateway
    • Check functions are deployed and started
    • Check request isn't timing out at the gateway or the function level

    CLI unresponsive - 127.0.0.1 vs. localhost

    On certain Linux distributions the name localhost maps to an IPv6 alias meaning that the CLI may hang. In these circumstances you have two options:

    • Use the -g or —gateway argument with 127.0.0.1:8080 or similar

    • Set the OPENFAAS_URL environmental variable to 127.0.0.1:8080 or similar

    • Edit the /etc/hosts file on your machine and remove the IPv6 alias for localhost (this forces the use of IPv4)

    If you'd like to uninstall or remove OpenFaaS from a host follow the steps below.

    CLI

    If you'd like to remove the CLI and you installed it with brew, then use to remove it.

    If you installed via the curl/sh utility script:

    • Run rm -rf /usr/local/bin/faas-cli
    • Delete saved gateway login details: rm -rf ~/.openfaas

    Swarm

    Remove any functions you deployed:

    1. $ docker service ls --filter="label=function" -q | xargs docker service rm

    Remove the whole stack

    1. $ docker stack rm func && \
    2.    docker secret rm basic-auth-user && \
    3.    docker secret rm basic-auth-password

    Kubernetes

    If deployed via Helm:

    1. helm delete --purge openfaas

    If installed via YAML files:

    1. kubectl delete namespace openfaas openfaas-fn

    Troubleshooting Swarm or Kubernetes

    List all functions

    Find a function's logs

    1. $ docker service logs --tail 100 FUNCTION

    Find out if a function failed to start

      I forgot my gateway password

      If you've logged into the OpenFaaS CLI then you can retrieve the credentials from config.yaml in ~/.openfaas/. Use the value from the token field such as: echo -n HASHED_VALUE | base64 -D/-d to view the contents in plain-text. If you don't have access to bash or the base64 utility then type in docker run -ti alpine:3.9 to run a shell in Docker.

      If you never logged in via the CLI then you can retrieve the contents from the cluster secret store:

      Swarm

      Use the jaas task-runner for Swarm (easiest option):

      1. $ docker run -ti -v /var/run/docker.sock:/var/run/docker.sock \
      2.   alexellis2/jaas:1.1.0 \
      3.   run --secret basic-auth-password \
      4.   --image alpine:3.10 \
      5.   --command "cat /run/secrets/basic-auth-password"
      6.  
      7. Printing service logs
      8. 2018-08-28T07:50:46.431268693Z  21f596c9cd75a0fe5e335fb743995d18399e83418a37a79e719576a724efbbb6
      • Or use a one-shot Docker Service:
      1. $ docker service rm print-password \
      2.  ; docker service create --detach --secret basic-auth-password \
      3.    --restart-condition=none --name=print-password \
      4.    alpine:3.9 cat /run/secrets/basic-auth-password
      5.  
      6. $ docker service logs print-password
      7. print-password.1.59bwe0bb4d99@nuc    | 21f596c9cd75a0fe5e335fb743995d18399e83418a37a79e719576a724efbbb6

      Troubleshoot Kubernetes

      If you have deployed OpenFaaS to the recommended namespaces then functions are in the openfaas-fn namespace and the core services are in the openfaas namespace. The -n flag to kubectl sets the namespace to look at.

      List OpenFaaS services

      1. $ kubectl get deploy -n openfaas

      List all functions

      1. $ kubectl get deploy -n openfaas-fn

      Find a function's logs

      1. $ kubectl logs -n openfaas-fn deploy/FUNCTION_NAME

      If you have more than one replica of a function, you may want to use a log-tailing tool like .

      View logs of all functions:

      1. kail -n openfaas-fn

      View logs of all replicas of a single function:

      1. kail -n openfaas-fn --deploy=FUNCTION_NAME

      Find out if a function failed to start

      1. $ kubectl describe -n openfaas-fn deploy/FUNCTION_NAME

      Check logs of the core services

      Check for any relevant events:

      1. $ kubectl get events --sort-by=.metadata.creationTimestamp -n openfaas

      These instructions may differ depending on whether you are using faas-netes (default) or the OpenFaaS Operator

      Get logs using faas-netes
      1. $ kubectl logs -n openfaas deploy/gateway -c faas-netes
      2. $ kubectl logs -n openfaas deploy/gateway -c gateway
      Check the queue-worker
      1. $ kubectl logs -n openfaas-fn deploy/queue-worker
      Get logs using OpenFaaS Operator
      1. $ kubectl logs -n openfaas deploy/gateway -c operator
      2. $ kubectl logs -n openfaas deploy/gateway -c gateway

      I forgot my gateway password

      Use the following to print the secret on the terminal:

      1. PASSWORD=$(kubectl get secret -n openfaas basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode; echo)
      2.  
      3. echo $PASSWORD

      Set OPENFAAS_URL then log into the gateway:

      1. echo -n $PASSWORD | faas-cli login --username admin --password-stdin

      If you installed OpenFaaS into a custom namespace then change the value to -n custom-ns.

      Watchdog

      Debug your function without deploying it

      Here's an example of how you can deploy a function without using an orchestrator and the API gateway. It is especially useful for testing:

      1. $ docker run --name debug-alpine \
      2.   -p 8081:8080 -ti functions/alpine:3.9 sh
      3. # fprocess=date fwatchdog &

      Now you can access the function with one of the supported HTTP methods such as GET/POST etc:

      1. $ curl -4 127.0.0.1:8081

      Edit your function without rebuilding it

      You can bind-mount code straight into your function and work with it locally, until you are ready to re-build. This is a common flow with containers, but should be used sparingly.

      Within the CLI directory for instance:

      Build the samples:

      1. $ git clone https://github.com/openfaas/faas-cli && \
      2.   cd faas-cli
      3. $ faas-cli -action build -f ./samples.yml

      Now work with the Python-hello sample, with the code mounted live:

      1. $ docker run -v `pwd`/sample/url-ping/:/root/function/ \
      2.   --name debug-alpine -p 8081:8080 -ti alexellis/faas-url-ping sh
      3. $ touch ./function/__init__.py
      4. # fwatchdog

      Now you can start editing the code in the sample/url-ping folder and it will reload live for every request.

      1. $ echo "def handle(req):" > sample/url-ping/handler.py
      2. $ echo '    print("Nothing to see here")' >> sample/url-ping/handler.py
      3. $ curl 127.0.0.1:8081 -d "https://www.google.com"