How to TCPdump effectively in Kubernetes (part 2)
- Philippe Bogaerts
- Feb 5, 2022
- 4 min read
In previous blog posts, we focused on how to use TCPdump in a specific container
While researching some other things recently, I came across a comment suggesting a quick fix for another issue like this …
$ kubectl run -it --rm debug --restart=Never --image=ubuntu --overrides='{"kind":"Pod", "apiVersion":"v1", "spec": {"hostNetwork":true}}'This is pretty similar as described in a previous blog posts …
docker run -it --net=host ubuntu … but in the kubectl case, we do not need SSH access to a node or access to the docker client, nor do we need to re-deploy the deployment (aka restart of the pods)
So let’s try this ... Create a small K8S cluster and deploy a simple nginx service (I used the managed K8S service from Digitalocean and tested as well on an Azure environment)
$ kubectl get no
NAME STATUS ROLES AGE VERSION
demo-pool1-lyg2 Ready <none> 2m3s v1.16.2
demo-pool1-lygl Ready <none> 2m12s v1.16.2
demo-pool1-lygt Ready <none> 2m14s v1.16.2$ kubectl get svc -n radarhack
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-radarhack-clusterip ClusterIP 10.245.94.156 <none> 80/TCP 100s$ kubectl get po -n radarhack -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
radarhack-deployment-655b776bd-6v6dl 1/1 Running 0 22m 10.244.0.232 demo-pool1-lygt <none> <none>
radarhack-deployment-655b776bd-wmqv9 1/1 Running 0 22m 10.244.1.87 demo-pool1-lygl <none> <none>
radarhack-deployment-655b776bd-zrxdd 1/1 Running 0 22m 10.244.2.223 demo-pool1-lyg2 <none> <none>
So lets deploy an ubuntu pod like described before …
$ kubectl run -it --rm debug --restart=Never --image=ubuntu --overrides='{"kind":"Pod", "apiVersion":"v1", "spec": {"hostNetwork":true}}'
If you don't see a command prompt, try pressing enter.… inside the pod, install following packages …
root@demo-pool1-lygt:/# apt-get update && apt-get install -y net-tools && apt-get install -y tcpdump
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
...
Setting up tcpdump (4.9.2-3) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...… and now we can list all the interfaces of the host inside the pod !
root@demo-pool1-lygt:/# ifconfig
cilium_host: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1500
inet 10.244.0.191 netmask 255.255.255.255 broadcast 0.0.0.0
inet6 fe80::44ca:4aff:fee7:ddba prefixlen 64 scopeid 0x20<link>
ether 46:ca:4a:e7:dd:ba txqueuelen 1000 (Ethernet)
RX packets 445 bytes 46829 (46.8 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 133 bytes 6790 (6.7 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
cilium_net: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1500
inet6 fe80::144b:40ff:fee3:892d prefixlen 64 scopeid 0x20<link>
ether 16:4b:40:e3:89:2d txqueuelen 1000 (Ethernet)
RX packets 133 bytes 6790 (6.7 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 445 bytes 46829 (46.8 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
cilium_vxlan: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::4e9:f7ff:fe9e:8a5a prefixlen 64 scopeid 0x20<link>
ether 06:e9:f7:9e:8a:5a txqueuelen 1000 (Ethernet)
RX packets 248 bytes 23105 (23.1 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 250 bytes 29122 (29.1 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:b9:15:7e:44 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 188.166.35.94 netmask 255.255.192.0 broadcast 188.166.63.255
inet6 fe80::7421:6bff:fed4:7a97 prefixlen 64 scopeid 0x20<link>
ether 76:21:6b:d4:7a:97 txqueuelen 1000 (Ethernet)
RX packets 35107 bytes 184254131 (184.2 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8237 bytes 637679 (637.6 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.18.0.9 netmask 255.255.0.0 broadcast 10.18.255.255
ether 76:21:6b:d4:7a:97 txqueuelen 1000 (Ethernet)
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.133.91.245 netmask 255.255.0.0 broadcast 10.133.255.255
inet6 fe80::407:2eff:fee7:7135 prefixlen 64 scopeid 0x20<link>
ether 06:07:2e:e7:71:35 txqueuelen 1000 (Ethernet)
RX packets 3320 bytes 1297431 (1.2 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3327 bytes 486607 (486.6 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 1933 bytes 123132 (123.1 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1933 bytes 123132 (123.1 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
...We can now attach TCPdump to any of the interfaces and record the traffic.
root@demo-pool1-lygt:/# tcpdump -i eth0 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:29:40.285086 IP 178.62.231.175.443 > 188.166.35.94.56492: Flags [P.], seq 3344525609:3344526062, ack 3498904092, win 249, options [nop,nop,TS val 1350708887 ecr 1279555160], length 453
...root@demo-pool1-lygt:/# tcpdump -i any -n port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
12:44:22.727590 IP 10.244.1.42.36658 > 10.244.0.232.80: Flags [S], seq 790184700, win 28200, options [mss 1410,sackOK,TS val 4002115421 ecr 0,nop,wscale 7], length 0
12:44:22.727695 IP 10.244.1.42.36658 > 10.244.0.232.80: Flags [S], seq 790184700, win 28200, options [mss 1410,sackOK,TS val 4002115421 ecr 0,nop,wscale 7], length 0
12:44:22.727738 IP 10.244.0.232.80 > 10.244.1.42.36658: Flags [S.], seq 2931529117, ack 790184701, win 27960, options [mss 1410,sackOK,TS val 3120146425 ecr 4002115421,nop,wscale 7], length 0
12:44:22.727751 IP 10.244.0.232.80 > 10.244.1.42.36658: Flags [S.], seq 2931529117, ack 790184701, win 27960, options [mss 1410,sackOK,TS val 3120146425 ecr 4002115421,nop,wscale 7], length 0
12:44:22.728424 IP 10.244.1.42.36658 > 10.244.0.232.80: Flags [.], ack 1, win 221, options [nop,nop,TS val 4002115422 ecr 3120146425], length 0
12:44:22.728460 IP 10.244.1.42.36658 > 10.244.0.232.80: Flags [.], ack 1, win 221, options [nop,nop,TS val 4002115422 ecr 3120146425], length 0Remark: The traffic being displayed in the second example in address space 10.244.0.0/16 is inter-pod traffic.
It is even possible to intercept the traffic in/out the kube-system pods ;-)
I hope this article provides an easy way to capture traffic and learn more on the inner workings of our K8S cluster. Thanks for reading, Philippe Bogaerts !!!
댓글