Getting Started Securing Memcached

    NOTE: memcached-aware policy support is still in beta. It is not yet ready for production use. Additionally, the memcached-specific policy language is highly likely to change in a future Cilium version.

    Memcached is a high performance, distributed memory object caching system. It’s simple yet powerful, and used by dynamic web applications to alleviate database load. Memcached is designed to work efficiently for a very large number of open connections. Thus, clients are encouraged to cache their connections rather than the overhead of reopening TCP connections every time they need to store or retrieve data. Multiple clients can benefit from this distributed cache’s performance benefits.

    There are two kinds of data sent in the memcache protocol: text lines and unstructured (binary) data. We will demonstrate clients using both types of protocols to communicate with a memcached server.

    If you haven’t read the yet, we’d encourage you to do that first.

    The best way to get help if you get stuck is to ask a question on the Cilium Slack channel. With Cilium contributors across the globe, there is almost always someone available to help.

    If you have not set up Cilium yet, pick any installation method as described in section to set up Cilium for your Kubernetes environment. If in doubt, pick Getting Started Using Minikube as the simplest way to set up a Kubernetes cluster with Cilium:

    Step 2: Deploy the Demo Application

    Now that we have Cilium deployed and operating correctly we can deploy our demo memcached application. Since our first HTTP-aware Cilium demo was based on Star Wars, we continue with the theme for the memcached demo as well.

    Ever wonder how the Alliance Fleet manages the changing positions of their ships? The Alliance Fleet uses memcached to store the coordinates of their ships. The Alliance Fleet leverages the memcached-svc service implemented as a memcached server. Each ship in the fleet constantly updates its coordinates and has the ability to get the coordinates of other ships in the Alliance Fleet.

    In this simple example, the Alliance Fleet uses a memcached server for their starfighters to store their own supergalatic coordinates and get those of other starfighters.

    In order to avoid collisions and protect against compromised starfighters, memcached commands are limited to gets for any starfighter coordinates and sets only to a key specific to the starfighter. Thus the following operations are allowed:

    • A-wing: can set coordinates to key “awing-coord” and get the key coordinates.
    • X-wing: can set coordinates to key “xwing-coord” and get the key coordinates.
    • Alliance-Tracker: can get any coordinates but not set any.

    To keep the setup small, we will launch a small number of pods to represent a larger environment:

    • memcached-server : A Kubernetes service represented by a single pod running a memcached server (label app=memcd-server).
    • a-wing memcached binary client : A pod representing an A-wing starfighter, which can update its coordinates and read it via the binary memcached protocol (label app=a-wing).
    • x-wing memcached text-based client : A pod representing an X-wing starfighter, which can update its coordinates and read it via the text-based memcached protocol (label app=x-wing).
    • alliance-tracker memcached binary client : A pod representing the Alliance Fleet Tracker, able to read the coordinates of all starfighters (label name=fleet-tracker).

    Memcached clients access the memcached-server on TCP port 11211 and send memcached protocol messages to it.

    The file memcd-sw-app.yaml contains a Kubernetes Deployment for each of the pods described above, as well as a Kubernetes Service memcached-server for the Memcached server.

    1. $ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.8/examples/kubernetes-memcached/memcd-sw-app.yaml
    2. deployment.extensions/memcached-server created
    3. service/memcached-server created
    4. deployment.extensions/a-wing created
    5. deployment.extensions/x-wing created
    6. deployment.extensions/alliance-tracker created

    Kubernetes will deploy the pods and service in the background. Running kubectl get svc,pods will inform you about the progress of the operation. Each pod will go through several states until it reaches Running at which point the setup is ready.

    1. $ kubectl get svc,pods
    2. NAME                       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)     AGE
    3. service/kubernetes         ClusterIP   10.96.0.1    <none>        443/TCP     31m
    4. service/memcached-server   ClusterIP   None         <none>        11211/TCP   14m
    6. NAME                                    READY   STATUS    RESTARTS   AGE
    7. pod/a-wing-67db8d5fcc-dpwl4             1/1     Running   0          14m
    8. pod/alliance-tracker-6b6447bd69-sz5hz   1/1     Running   0          14m
    9. pod/memcached-server-bdbfb87cd-8tdh7    1/1     Running   0          14m
    10. pod/x-wing-fd5dfb9d9-wrtwn              1/1     Running   0          14m

    In all three terminal windows, set some handy environment variables for the demo with the following script:

    1. $ curl -s https://raw.githubusercontent.com/cilium/cilium/v1.8/examples/kubernetes-memcached/memcd-env.sh > memcd-env.sh
    2. $ source memcd-env.sh

    In the terminal window dedicated for the A-wing pod, exec in, use python to import the binary memcached library and set the client connection to the memcached server:

    1. $ kubectl exec -ti $AWING_POD sh
    2. # python
    3. Python 3.7.0 (default, Sep  5 2018, 03:25:31)
    4. [GCC 6.3.0 20170516] on linux
    5. Type "help", "copyright", "credits" or "license" for more information.
    6. >>> import bmemcached
    7. >>> client = bmemcached.Client(("memcached-server:11211", ))

    In the terminal window dedicated for the Alliance-Tracker, exec in, use python to import the binary memcached library and set the client connection to the memcached server:

    1. $ kubectl exec -ti $TRACKER_POD sh
    2. # python
    3. Python 3.7.0 (default, Sep  5 2018, 03:25:31)
    4. [GCC 6.3.0 20170516] on linux
    5. Type "help", "copyright", "credits" or "license" for more information.
    6. >>> import bmemcached
    7. >>> client = bmemcached.Client(("memcached-server:11211", ))

    Let’s show that each client is able to access the memcached server. Execute the following to have the A-wing and X-wing starfighters update the Alliance Fleet memcached-server with their respective supergalatic coordinates:

    A-wing will access the memcached-server using the binary protocol. In your terminal window for A-Wing, set A-wing’s coordinates:

    1. >>> client.set("awing-coord","4309.432,918.980",time=2400)
    2. True
    3. >>> client.get("awing-coord")
    4. '4309.432,918.980'

    In your main terminal window, have X-wing starfighter set their coordinates using the text-based protocol to the memcached server.

    Check that the Alliance Fleet Tracker is able to get all starfighters’ coordinates in your terminal window for the Alliance-Tracker:

    1. >>> client.get("awing-coord")
    2. '4309.432,918.980'
    3. >>> client.get("xwing-coord")
    4. '8893.34,234.3290'

    Step 4: The Danger of a Compromised Memcached Client

    Imagine if a starfighter ship is captured. Should the starfighter be able to set the coordinates of other ships, or get the coordinates of all other ships? Or if the Alliance-Tracker is compromised, can it modify the coordinates of any starfighter ship? If every client has access to the Memcached API on port 11211, all have over-privileged access until further locked down.

    With L4 port access to the memcached server, all starfighters could write to any key/ship and read all ship coordinates. In your main terminal, execute:

    1. $ kubectl exec $XWING_POD sh -- -c "echo -en \"$GETAC\" | nc memcached-server 11211"
    2. 4309.432,918.980
    3. END

    In your A-Wing terminal window, confirm the over-privileged access:

    1. >>> client.get("xwing-coord")
    2. '8893.34,234.3290'
    3. >>> client.set("xwing-coord","0.0,0.0",time=2400)
    4. True
    5. >>> client.get("xwing-coord")

    From A-Wing, set the X-Wing coordinates back to their proper position:

    1. >>> client.set("xwing-coord","8893.34,234.3290",time=2400)
    2. True

    Thus, the Alliance Fleet Tracking System could be made weak if a single starfighter ship is compromised.

    Cilium helps lock down Memcached servers to ensure clients have secure access to it. Beyond just providing access to port 11211, Cilium can enforce specific key value access by understanding both the text-based and the unstructured (binary) memcached protocol.

    We’ll create a policy that limits the scope of what a starfighter can access and write. Thus, only the intended memcached protocol calls to the memcached-server can be made.

    In this example, we’ll only allow A-Wing to get and set the key “awing-coord”, only allow X-Wing to get and set key “xwing-coord”, and allow Alliance-Tracker to only get coordinates.

    Here is the CiliumNetworkPolicy rule that limits the access of starfighters to their own key and allows Alliance Tracker to get any coordinate:

    1. apiVersion: "cilium.io/v2"
    2. kind: CiliumNetworkPolicy
    3. metadata:
    4.   name: "secure-fleet"
    5. spec:
    6.   endpointSelector:
    7.     matchLabels:
    8.       app: memcd-server
    9.   ingress:
    10.   - fromEndpoints:
    11.     - matchLabels:
    12.         app: a-wing
    13.     toPorts:
    14.     - ports:
    15.       - port: '11211'
    16.         protocol: TCP
    17.       rules:
    18.         l7proto: memcache
    19.         l7:
    20.         - command: get
    21.           keyExact: awing-coord
    22.         - command: set
    23.           keyExact: awing-coord
    24.   - fromEndpoints:
    25.     - matchLabels:
    26.         app: x-wing
    27.     toPorts:
    28.     - ports:
    29.       - port: '11211'
    30.         protocol: TCP
    31.       rules:
    32.         l7proto: memcache
    33.         l7:
    34.         - command: get
    35.           keyExact: xwing-coord
    36.         - command: set
    37.           keyExact: xwing-coord
    38.   - fromEndpoints:
    39.     - matchLabels:
    40.         name: fleet-tracker
    41.     toPorts:
    42.     - ports:
    43.       - port: '11211'
    44.         protocol: TCP
    45.         l7proto: memcache
    46.         l7:
    47.         - command: get
    48.           keyExact: awing-coord
    49.         - command: get

    A CiliumNetworkPolicy contains a list of rules that define allowed memcached commands, and requests that do not match any rules are denied. The rules explicitly match connections destined to the Memcached Service on TCP 11211.

    The rules apply to inbound (i.e., “ingress”) connections bound for memcached-server pods (as indicated by app:memcached-server in the “endpointSelector” section). The rules apply differently depending on the client pod: app:a-wing, app:x-wing, or name:fleet-tracker as indicated by the “fromEndpoints” section.

    With the policy in place, A-wings can only get and set the key “awing-coord”; similarly the X-Wing can only get and set “xwing-coord”. The Alliance Tracker can only get coordinates - not set.

    Apply this Memcached-aware network security policy using kubectl in your main terminal window:

    1. $ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.8/examples/kubernetes-memcached/memcd-sw-security-policy.yaml

    If we then try to perform the attacks from the X-wing pod from the main terminal window, we’ll see that they are denied:

    From the A-Wing terminal window, we can confirm that if A-wing goes outside of the bounds of its allowed calls. You may need to run the client.get command twice for the python call:

    1. >>> client.get("awing-coord")
    2. '4309.432,918.980'
    3. >>> client.get("xwing-coord")
    4. Traceback (most recent call last):
    5.   File "<stdin>", line 1, in <module>
    6.   File "/usr/local/lib/python3.7/site-packages/bmemcached/client/replicating.py", line 42, in get
    7.     value, cas = server.get(key)
    8.   File "/usr/local/lib/python3.7/site-packages/bmemcached/protocol.py", line 440, in get
    9.     raise MemcachedException('Code: %d Message: %s' % (status, extra_content), status)
    10. bmemcached.exceptions.MemcachedException: ("Code: 8 Message: b'access denied'", 8)

    Similarly, the Alliance-Tracker cannot set any coordinates, which you can attempt from the Alliance-Tracker terminal window:

    1. >>> client.get("xwing-coord")
    2. '8893.34,234.3290'
    3. >>> client.set("awing-coord","0.0,0.0",time=1200)
    4. Traceback (most recent call last):
    5.   File "<stdin>", line 1, in <module>
    6.   File "/usr/local/lib/python3.7/site-packages/bmemcached/client/replicating.py", line 112, in set
    7.     returns.append(server.set(key, value, time, compress_level=compress_level))
    8.   File "/usr/local/lib/python3.7/site-packages/bmemcached/protocol.py", line 604, in set
    9.     return self._set_add_replace('set', key, value, time, compress_level=compress_level)
    10.   File "/usr/local/lib/python3.7/site-packages/bmemcached/protocol.py", line 583, in _set_add_replace
    11.     raise MemcachedException('Code: %d Message: %s' % (status, extra_content), status)
    12. bmemcached.exceptions.MemcachedException: ("Code: 8 Message: b'access denied'", 8)

    The policy is working as expected.

    With the CiliumNetworkPolicy in place, the allowed Memcached calls are still allowed from the respective pods.

    In the main terminal window, execute:

    1. $ kubectl exec $XWING_POD sh -- -c "echo -en \"$GETXC\" | nc memcached-server 11211"
    2. VALUE xwing-coord 0 16
    3. 8893.34,234.3290
    4. END
    5. $ SETXC="set xwing-coord 0 1200 16\r\n9854.34,926.9187\r\nquit\r\n"
    6. $ kubectl exec $XWING_POD sh -- -c "echo -en \"$SETXC\" | nc memcached-server 11211"
    7. STORED
    8. $ kubectl exec $XWING_POD sh -- -c "echo -en \"$GETXC\" | nc memcached-server 11211"
    9. VALUE xwing-coord 0 16
    10. 9854.34,926.9187
    11. END

    In the A-Wing terminal window, execute:

    1. >>> client.set("awing-coord","9852.542,892.1318",time=1200)
    2. True
    3. >>> client.get("awing-coord")
    4. '9852.542,892.1318'
    5. >>> exit()
    6. # exit

    In the Alliance-Tracker terminal window, execute:

    1. >>> client.get("awing-coord")
    2. >>> client.get("xwing-coord")
    3. >>> exit()
    4. # exit

    Step 7: Clean Up

    You have now installed Cilium, deployed a demo app, and tested L7 memcached-aware network security policies. To clean up, in your main terminal window, run:

    1. $ kubectl delete -f https://raw.githubusercontent.com/cilium/cilium/v1.8/examples/kubernetes-memcached/memcd-sw-app.yaml
    2. $ kubectl delete cnp secure-fleet

    For some handy memcached references, see below: