Measure and Troubleshoot Linux Network Resource Usage

Measuring network usage and identifying established connections on a Debian Linux server. 

Install Performance Monitoring Tools

Install sysstat, vnstat and iftop, if not yet installed:

# apt-get update && apt-get install sysstat vnstat iftop

For RHEL/CentOS, do the following (must have EPEL repository enabled):

# yum install -y sysstat vnstat iftop

The sysstat package contains sar and iostat system performance tools which we’ll be using today.

Make sure that sar is enabled in /etc/default/sysstat. If not enabled, do it.

You may also want to change the history value in /etc/sysstat/sysstat to something different than 7 days:

HISTORY=60

Note that if value is greater than 28, then log files will be kept in multiple directories, one for each month.

By default sysstat will collect data every 10 minutes. You can change this by modifying the cronjob /etc/cron.d/sysstat.

Finally, restart the service:

# service sysstat restart

Measure Network Usage

Network Usage with netstat

The netstat command prints information about the Linux networking subsystem. By default, netstat displays a list of open sockets.

Different arguments can be passed to print the desired type of information. Some of available arguments are listed below:

  1. -a: show both listening and non-listening sockets.
  2. -r: display the kernel routing table. Note that netstat -r and route -e produce the same output. However, route -e command requires root privileges to be run where netstat -r doesn’t.
  3. -i: display a table of all network interfaces.
  4. -e: display additional information.
  5. -p: show the PID and name of the program to which each socket belongs.
  6. -l: show listening sockets only.
  7. -t: show TCP connections.
  8. -u: show UDP connections.
  9. -c: print selected information every second continuously.
  10. -M: display a list of masqueraded connections (only works on pre-2.4 kernels). The ipt_MASQUERADE module has to be loaded.

On new kernels (3.2 f.e.), the following message appears:

$ netstat -M
netstat: no support for `ip_masquerade' on this system.

It is advised to use the /proc/net/ip_conntrack on new kernels to get a list of masqueraded connections.

Show all established TCP (-t) connections by using numerical ports and addresses (-n):

$ netstat -nt | grep -i established
tcp      0    0 10.10.1.17:12         10.10.1.10:56839     ESTABLISHED
tcp      0    0 10.10.1.17:53669      10.10.1.8:10050      ESTABLISHED
tcp6     0    0 10.10.1.17:443        10.10.1.10:42590     ESTABLISHED
tcp6     0    0 10.10.1.17:443        10.10.1.10:42606     ESTABLISHED
tcp6     0    0 10.10.1.17:443        10.10.1.10:42605     ESTABLISHED

Preview network statistics (-s) for all protocols:

$ netstat -s | less

Network Usage with vnstat

The vnstat is a console-based network traffic monitor. It keeps a log of hourly, daily and monthly network traffic for the selected interfaces.

Show network usage for the eth0 interface (-i):

$ vnstat -i eth0
Database updated: Sat Mar  1 17:16:49 2014

   eth0 since 02/17/14

          rx:  1.87 GiB      tx:  1.58 GiB      total:  3.45 GiB

   monthly
                     rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
       Feb '14      1.61 GiB |    1.36 GiB |    2.97 GiB |   10.30 kbit/s
       Mar '14    129.66 MiB |  119.31 MiB |  248.97 MiB |   32.79 kbit/s
     ------------------------+-------------+-------------+---------------
     estimated      5.42 GiB |    5.00 GiB |   10.43 GiB |

   daily
                     rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
     yesterday    156.53 MiB |  135.77 MiB |  292.30 MiB |   27.71 kbit/s
         today    129.66 MiB |  119.31 MiB |  248.97 MiB |   32.79 kbit/s
     ------------------------+-------------+-------------+---------------
     estimated       179 MiB |     165 MiB |     344 MiB |

Select the eth0 interface (-i) and display traffic by using the short mode (-s):

$ vnstat -i eth0 -s

                      rx      /      tx      /     total    /   estimated
 eth0:
       Feb '14      1.61 GiB  /    1.36 GiB  /    2.97 GiB
       Mar '14    129.66 MiB  /  119.31 MiB  /  248.97 MiB  /   10.43 GiB
     yesterday    156.53 MiB  /  135.77 MiB  /  292.30 MiB
         today    129.66 MiB  /  119.31 MiB  /  248.97 MiB  /     344 MiB

Display current transfer rate for the eth0 interface in real time until interrupted. Statistics will be shown after interruption if the runtime was more than 10 seconds:

# vnstat -i eth0 -l
Monitoring eth0...    (press CTRL-C to stop)

   rx:        0 kbit/s     0 p/s          tx:        0 kbit/s     0 p/s^C

 eth0  /  traffic statistics

                           rx         |       tx
--------------------------------------+------------------
  bytes                   176.99 MiB  |      704.42 MiB
--------------------------------------+------------------
          max          242.34 Mbit/s  |   251.37 Mbit/s
      average           16.29 Mbit/s  |    64.84 Mbit/s
          min               0 kbit/s  |        0 kbit/s
--------------------------------------+------------------
  packets                     325642  |          123123
--------------------------------------+------------------
          max              20681 p/s  |        5966 p/s
      average               3658 p/s  |        1383 p/s
          min                  0 p/s  |           0 p/s
--------------------------------------+------------------
  time                  1.48 minutes

Select the eth0 interface (-i) and show traffic for days (-d):

$ vnstat -i eth0 -d

 eth0  /  daily

         day         rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
      02/17/14     43.75 MiB |   16.66 MiB |   60.41 MiB |    5.73 kbit/s
      02/18/14     94.95 MiB |   88.40 MiB |  183.35 MiB |   17.38 kbit/s
      12/20/14         0 KiB |       0 KiB |       0 KiB |    0.00 kbit/s
      02/18/14     24.16 MiB |   16.39 MiB |   40.55 MiB |    3.84 kbit/s
      02/19/14    147.66 MiB |  131.22 MiB |  278.89 MiB |   26.44 kbit/s
      02/20/14    161.49 MiB |  134.28 MiB |  295.76 MiB |   28.04 kbit/s
      02/21/14    169.64 MiB |  137.29 MiB |  306.93 MiB |   29.10 kbit/s
      02/22/14    166.96 MiB |  142.06 MiB |  309.02 MiB |   29.30 kbit/s
      02/23/14    166.68 MiB |  139.80 MiB |  306.48 MiB |   29.06 kbit/s
      02/24/14    170.39 MiB |  144.82 MiB |  315.21 MiB |   29.89 kbit/s
      02/25/14    157.14 MiB |  137.00 MiB |  294.14 MiB |   27.89 kbit/s
      02/26/14    168.33 MiB |  138.85 MiB |  307.18 MiB |   29.13 kbit/s
      02/27/14    160.50 MiB |  136.23 MiB |  296.73 MiB |   28.13 kbit/s
      02/28/14    156.53 MiB |  135.77 MiB |  292.30 MiB |   27.71 kbit/s
      03/01/14    129.66 MiB |  119.31 MiB |  248.97 MiB |   32.79 kbit/s
     ------------------------+-------------+-------------+---------------
     estimated       179 MiB |     165 MiB |     344 MiB |

Network Usage with sar

Report network (-n) statistics from the network devices (DEV):

$ sar -n DEV 1 2
Linux 3.2.0-4-686-pae (flames) 	01/03/14 	_i686_	(2 CPU)

17:25:57   IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
17:25:58      lo     14.14     14.14      0.76      0.76      0.00      0.00      0.00
17:25:58    eth0     52.53     65.66      0.00      0.00      0.00      0.00      0.00

17:25:58   IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
17:25:59      lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
17:25:59    eth0    351.00    336.00      0.00      0.00      0.00      0.00      0.00

Average:   IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
Average:      lo      7.04      7.04      0.38      0.38      0.00      0.00      0.00
Average:    eth0    202.51    201.51      0.00      0.00      0.00      0.00      0.00

The following values are displayed (as per man page):

  1. IFACE: name of the network interface for which statistics are reported.
  2. rxpck/s: total number of packets received per second.
  3. txpck/s: total number of packets transmitted per second.
  4. rxkB/s: total number of kilobytes received per second.
  5. txkB/s: total number of kilobytes transmitted per second.
  6. rxcmp/s: number of compressed packets received per second.
  7. txcmp/s: number of compressed packets transmitted per second.
  8. rxmcst/s: number of multicast packets received per second.

Report historical network (-n) statistics from the network devices (DEV) starting (-s) 1 PM and ending (-e) 2 PM time interval:

$ sar -n DEV -s 13:00:00 -e 14:00:00
Linux 3.2.0-4-686-pae (flames) 	01/03/14 	_i686_	(2 CPU)

13:05:01   IFACE   rxpck/s  txpck/s  rxkB/s  txkB/s  rxcmp/s  txcmp/s  rxmcst/s
13:15:01      lo     22.17    22.17    6.04    6.04     0.00     0.00      0.00
13:15:01    eth0    144.48   149.90    0.00    0.00     0.00     0.00      0.00
13:25:01      lo     27.09    27.09   34.76   34.76     0.00     0.00      0.00
13:25:01    eth0    159.49   157.17    0.00    0.00     0.00     0.00      0.00
13:35:01      lo     21.91    21.91    6.13    6.13     0.00     0.00      0.00
13:35:01    eth0    141.16   146.25    0.00    0.00     0.00     0.00      0.00
13:45:01      lo     20.91    20.91    5.85    5.85     0.00     0.00      0.00
13:45:01    eth0    137.24   143.68    0.00    0.00     0.00     0.00      0.00
13:55:01      lo     26.50    26.50   39.18   39.18     0.00     0.00      0.00
13:55:01    eth0    148.30   151.32    0.00    0.00     0.00     0.00      0.00
Average:      lo     23.71    23.71   18.39   18.39     0.00     0.00      0.00
Average:    eth0    146.13   149.66    0.00    0.00     0.00     0.00      0.00

For ultimate sar experience, try the following command:

$ sar -n ALL 1 1

Network Usage with iftop

The iftop command displays network bandwidth usage for the interface specified or for the first external interface it can find. Iftop must be run with sufficient permissions (usually as root) to monitor all network traffic.

Get all TCP connections with sent and received traffic on separate lines:

# iftop -nNPb -i eth0 -f tcp

-----------------------------------------------------------------------------------
10.10.1.17:12         => 10.10.1.10:56536                   3.78kb  4.03kb  3.94kb
                      <=                                     416b    493b    316b
10.10.1.17:51909      => 10.10.1.8:10050                       0b    233b     61b
                      <=                                       0b    232b     61b
10.10.1.17:51918      => 10.10.1.8:10050                       0b    234b     61b
                      <=                                       0b    231b     61b 
10.10.1.17:51921      => 10.10.1.8:10050                    1.14kb   233b     61b
                      <=                                    1.13kb   232b     61b 
10.10.1.17:51902      => 10.10.1.8:10050                       0b    232b     61b
                      <=                                       0b    232b     61b 
10.10.1.17:51912      => 10.10.1.8:10050                       0b    232b     61b
                      <=                                       0b    232b     61b 
10.10.1.17:51915      => 10.10.1.8:10050                       0b    231b     61b
                      <=                                       0b    231b     61b 
10.10.1.17:443        => 10.10.1.10:42515                      0b     42b   76.2kb
                      <=                                       0b     42b   1.74kb 
10.10.1.17:443        => 10.10.1.10:42509                      0b     42b   1.94kb
                      <=                                       0b     42b   1.39kb 
10.10.1.17:443        => 10.10.1.10:42527                      0b     42b    433b
                      <=                                       0b     42b    495b
-----------------------------------------------------------------------------------
TX:             cum:    642kB   peak:   1.62Mb     rates:   4.92kb  5.63kb   135kb
RX:                     200kB            288kb              1.54kb  2.08kb  42.1kb
TOTAL:                  842kB           1.75Mb              6.46kb  7.72kb   177kb

Parameters used are below:

  1. -n: don’t do hostname lookups..
  2. -N: don’t resolve port number to service names.
  3. -P: turn on port display.
  4. -b: don’t display bar graphs of traffic.
  5. -i: listen to packets on the interface specified.
  6. -f: use filter code to select the IP packets to count.

Get total traffic shown for ssh port 12:

# iftop -nNPb -i eth0 -f "dst port 12"
-------------------------------------------------------------------------------
10.10.0.17:12       <=> 10.10.0.10:56536                  160b   2.17kb   555b
10.10.0.17:12       <=> 10.10.0.10:36030                    0b   1.68kb   431b
10.10.0.17:12       <=> 10.10.0.10:1692                   160b    160b    194b

-------------------------------------------------------------------------------
TX:          cum:      0B    peak:      0b      rates:      0b      0b      0b
RX:                 15.0kB           11.0kb               160b   4.00kb  1.78kb
TOTAL:              15.0kB           11.0kb               160b   4.00kb  1.78kb

Tip: use “t” to cycle through the four line display modes; the default 2-line display, with sent and received traffic on separate lines, and 3 1-line displays, with sent, received, or total traffic shown.

Related Posts

Finding Open Ports and Listening Services on Linux: Nmap, Netstat, lsof and ss

Measure and Troubleshoot Linux CPU Resource Usage

Measure and Troubleshoot Linux CPU Resource Usage

Measure and Troubleshoot Linux Disk I/O Resource Usage

Leave a Reply

Your email address will not be published. Required fields are marked *