Benjamin Sago / ogham / cairnrefinery / etc…

Technical notes › Watch a remote server’s packets in Wireshark, live

Like the heading says, it’s occasionally useful to see the exact network traffic that’s being sent and received by a server as it happens. If you ever need to debug something that’s happening at the network level, this functionality is indispensible. Sure, you can use tcpdump to save packets to a file, which you can then analyse later — but then you lose the real-time aspect.

To do this, you have to put a script on the server that runs tcpdump, configured to ignore the SSH packets for the current session. You can then either pipe this script into tshark, the command-line version of Wireshark, or you can send the packets back over SSH and analyse them inside Wireshark directly, so you get to keep using its GUI.

The script and the SSH problem

Here is the script that runs tcpdump. It dumps all packets to standard output as their raw network frames — so binary, not text. Don’t just run it in your terminal! It’ll barf strange characters everywhere.

/usr/local/bin/packet-capture
#!/bin/bash
if [ -z "$SSH_CLIENT" ]; then
  echo "[Not SSHing? Just run tcpdump manually]"
  exit 1
else
  read -r client_host _local_port client_port <<< "$SSH_CLIENT"
  sudo tcpdump -U -s 0 -w - "not (port $client_port and host $client_host)"
fi

It uses the following flags:

  • The -U flag disables buffering, so captured packets are shown immediately.
  • The -s 0 arguments tell tcpdump to capture everything, rather than to stop after a certain number of bytes. This functionality is usually used when saving all network traffic to multiple files for long-term storage.
  • The -w - arguments tell tcpdump to print the packets to standard output, rather than to a file.

If you were going to run this script on your local computer, these three arguments would be sufficient. But there’s an added hurdle when running the script over SSH. Because it logs all network traffic, each packet sent will result in some more SSH traffic — which then gets captured by the script itself, and sent over SSH, which gets captured, and sent, and so on, ad infinitum.

To solve this, we analyse the $SSH_CLIENT environment variable to specifically exclude the packets of the current SSH session from being returned.

Once you have put the script somewhere a regular user can run it, such as at /usr/local/bin/packet-capture, you can invoke it over SSH and pipe the results directly to Wireshark:

ssh $some-remote-host packet-capture | wireshark -k -i -

This will connect to the host using SSH and open Wireshark, and if everything goes correctly, that machine’s packets will be displayed in the local Wireshark.

One thing to keep in mind is that the Stop button in Wireshark just stops packets from being displayed, not from being captured. To stop the process entirely, close Wireshark or Control-C the shell pipeline.

The tcpdump problem

The annoying part about this is that for security reasons, by default in Linux, only the root user is allowed to see the entire machine’s network traffic like this. And as you’re (hopefully) not SSHing onto the machine as root, we need to find some way to do this.

  1. Use passwordless sudo, meaning you can just use sudo in the script with no further mess or fuss.

  2. Make an exception in /etc/sudoers that specifically allows passwordless sudo for this script in particular.

  3. Write a program in a compiled language that emulates this shell script’s functionality.

One way you might expect to work, but doesn’t, is setting the setuid bit of the shell script itself. Linux helpfully ignores this flag for interpreted scripts, only taking it into account when running binaries.

Which way you pick is up to you, but unfortunately you will have to pick one.∎

All tech notes