How to TCPdump effectively in Kubernetes (part 1)

Feb 3, 20222 min read

Updated: Feb 22, 2023

In a previous blog post, we focused on how to TCPdump in docker containers (see https://xxradar.medium.com/how-to-tcpdump-effectively-in-docker-2ed0a09b5406).

Although the information is still very useful and valid for troubleshooting K8S pods, it might get more difficult figuring out which containers to attach to on what node, etc … but it is still a very valid approach.

While focusing on an easier way, I came across the command

kubectl patch

This command allows to update an existing deployment for example. It basically does the trick outlined in the previous post but fully automatic

So if you have deployment running like for example

kubectl create -f https://raw.githubusercontent.com/xxradar/kuberneteslearning/master/radarhack-deploy.yaml

kubectl create -f https://raw.githubusercontent.com/xxradar/kuberneteslearning/master/radarhack-expose-clusterip.yaml

You should be able to access it on

# kubectl get services
...
my-radarhack-clusterip ClusterIP 10.104.201.226 <none> 80/TCP 36d

# curl http://10.104.201.226/
<HTML>
<HEAD>
<TITLE>RADARHACK.COM by XXRADAR</TITLE>
...

So far so good. Let’s focus on how to add the TCPdump container to the deployment. Create following file ex. patch.yaml

spec:
  template:
    spec:
      containers:
      - name: tcpdumper
        image: docker.io/dockersec/tcpdump

And apply it

kubectl patch deployment radarhack-deployment --patch “$(cat patch.yaml)”

You should be able to see that the TCPdump container is automatically added to the pods (please note that the pods are recreated, which is not exactly the same as in the previous blogpost, where you connect to a running pod/container)


# kubectl get deployment radarhack-deployment  --output yaml
apiVersion: extensions/v1beta1
kind: Deployment
..
      labels:
        app: radarhack
    spec:
      containers:
      - image: docker.io/dockersec/tcpdump
        imagePullPolicy: Always
        name: tcpdumper
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      - image: docker.io/xxradar/naxsi5
        imagePullPolicy: Always
        name: radarhack
        ports:
        - containerPort: 80
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 3
  conditions:
  ...

Now you can attach to the pod (and if traffic is generated)

# kubectl get pod
NAME                                   READY   STATUS    RESTARTS   AGE
...
radarhack-deployment-7c6b8f595-85dkt   3/3     Running   0          33m
radarhack-deployment-7c6b8f595-8qffp   3/3     Running   0          33m
radarhack-deployment-7c6b8f595-grdxz   3/3     Running   0          33m
radarhack22-56b59f7c7c-7cx2w           1/1     Running   0          36d
radarhack22-56b59f7c7c-bsnjk           1/1     Running   0          36d

# kubectl attach -it radarhack-deployment-7c6b8f595–85dkt
Defaulting container name to tcpdumper.
Use ‘kubectl describe pod/ -n default’ to see all of the containers in this pod.
Unable to use a TTY — container tcpdumper did not allocate one
If you don’t see a command prompt, try pressing enter.
l 3777658557 ecr 3777667637], length 0
08:14:38.740879 IP 10.244.0.0.48528 > radarhack-deployment-7c6b8f595–85dkt.80: Flags [.], ack 7945, win 360, options [nop,nop,TS val 3777667637 ecr 3777658557], length 0
08:14:38.936447 ARP, Request who-has radarhack-deployment-7c6b8f595–85dkt tell 10.244.2.1, length 28
08:14:38.936473 ARP, Reply radarhack-deployment-7c6b8f595–85dkt is-at fa:7a:24:85:f1:56 (oui Unknown), length 28
08:43:37.932430 IP 10.244.0.0.58290 > 10.244.2.50.80: Flags [S], seq 877954416, win 29200, options [mss 1460,sackOK,TS val 3779406828 ecr 0,nop,wscale 7], length 0
08:43:38.466295 IP 10.244.0.0.58292 > radarhack-deployment-7c6b8f595–85dkt.80: Flags [S], seq 3362221005, win 29200, options [mss 1460,sackOK,TS val 3779407363 ecr 0,nop,wscale 7], length 0
08:43:38.466343 IP radarhack-deployment-7c6b8f595–85dkt.80 > 10.244.0.0.58292: Flags [S.], seq 1653377327, ack 3362221006, win 27960, options [mss 1410,sackOK,TS val 3779398282 ecr 3779407363,nop,wscale 7], length 0

How to TCPdump effectively in Kubernetes (part 1)

Recent Posts

Comentários