How to TCPdump effectively in Kubernetes (part 2)

📅 February 5, 2022 ✍️ Philippe Bogaerts ⏱️ 4 min read 📁 Kubernetes

tcpdump Kubernetes Container

In previous blog posts, we focused on how to use TCPdump in a specific container:

While researching some other things recently, I came across a comment suggesting a quick fix for another issue like this:

```bash
$ kubectl run -it --rm debug --restart=Never --image=ubuntu --overrides='{"kind":"Pod", "apiVersion":"v1", "spec": {"hostNetwork":true}}'
```

This is pretty similar as described in a previous blog posts:

```bash
docker run -it --net=host ubuntu
```

...but in the kubectl case, we do not need SSH access to a node or access to the docker client, nor do we need to re-deploy the deployment (aka restart of the pods).

So let's try this...

Create a small K8S cluster and deploy a simple nginx service (I used the managed K8S service from Digitalocean and tested as well on an Azure environment):

```bash
$ kubectl get no
NAME STATUS ROLES AGE VERSION
demo-pool1-lyg2 Ready 2m3s v1.16.2
demo-pool1-lygl Ready 2m12s v1.16.2
demo-pool1-lygt Ready 2m14s v1.16.2

$ kubectl get svc -n radarhack
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-radarhack-clusterip ClusterIP 10.245.94.156 80/TCP 100s

$ kubectl get po -n radarhack -o wide
NAME READY STATUS RESTARTS AGE IP NODE
radarhack-deployment-655b776bd-6v6dl 1/1 Running 0 22m 10.244.0.232 demo-pool1-lygt
radarhack-deployment-655b776bd-wmqv9 1/1 Running 0 22m 10.244.1.87 demo-pool1-lygl
radarhack-deployment-655b776bd-zrxdd 1/1 Running 0 22m 10.244.2.223 demo-pool1-lyg2
```

So lets deploy an ubuntu pod like described before:

```bash
$ kubectl run -it --rm debug --restart=Never --image=ubuntu --overrides='{"kind":"Pod", "apiVersion":"v1", "spec": {"hostNetwork":true}}'
If you don't see a command prompt, try pressing enter.
```

Inside the pod, install following packages:

```bash
root@demo-pool1-lygt:/# apt-get update && apt-get install -y net-tools && apt-get install -y tcpdump
```

And now we can list all the interfaces of the host inside the pod!

```bash
root@demo-pool1-lygt:/# ifconfig
cilium_host: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1500
inet 10.244.0.191 netmask 255.255.255.255 broadcast 0.0.0.0
...

docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
...

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 188.166.35.94 netmask 255.255.192.0 broadcast 188.166.63.255
...
```

We can now attach TCPdump to any of the interfaces and record the traffic:

```bash
root@demo-pool1-lygt:/# tcpdump -i eth0 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:29:40.285086 IP 178.62.231.175.443 > 188.166.35.94.56492: Flags [P.], ...

root@demo-pool1-lygt:/# tcpdump -i any -n port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
12:44:22.727590 IP 10.244.1.42.36658 > 10.244.0.232.80: Flags [S], ...
```

Remark: The traffic being displayed in the second example in address space `10.244.0.0/16` is inter-pod traffic.

It is even possible to intercept the traffic in/out the kube-system pods ;-)

I hope this article provides an easy way to capture traffic and learn more on the inner workings of our K8S cluster. Thanks for reading!