Send Squid Logs to Graylog

Using GELF to send Squid logs to Graylog.

We are going to use the same approach as we did for Apache.

Squid 3.x Configuration

We are going to define a new logformat format called graylog_vhost to format a reverse proxy access log into a GELF format (JSON message):

logformat graylog_vhost {"version":"1.1","host":"%{Host}>h","short_message":"%rm %ru HTTP/%rv","level":6,"timestamp":"%tl","_client_ip":"%>a","_squid_ip":"%la","_server_ip":"%<a","_response_time":"%tr","_request_size":"%>st","_reply_size":"%<st","_http_url":"%ru","_http_status":"%>Hs","_http_method":"%rm","_http_referer":"%{Referer}>h","_user_agent":"%{User-Agent}>h","_squid_request_status":"%Ss","_squid_hierarchy_status":"%Sh","_from_squid":"true"}

Please check http://www.squid-cache.org/Versions/v3/3.5/cfgman/logformat.html for other format arguments that are available.

The below is a human-readable format:

{
   "version":"1.1",
   "host":"%{Host}>h",
   "short_message":"%rm %ru HTTP/%rv",
   "level":6,
   "timestamp":"%tl",
   "_client_ip":"%>a",
   "_squid_ip":"%la",
   "_server_ip":"%<a", 
   "_response_time":"%tr", 
   "_request_size":"%>st",
   "_reply_size":"%<st", 
   "_http_url":"%ru", 
   "_http_status":"%>Hs",
   "_http_method":"%rm",
   "_http_referer":"%{Referer}>h",
   "_user_agent":"%{User-Agent}>h",
   "_squid_request_status":"%Ss",
   "_squid_hierarchy_status":"%Sh",
   "_from_squid":"true"
}

Squid is capable of sending each access log line as text data to a TCP or a UDP receiver. We can use it to send logs to Graylog. The following line in the Squid configuration file squid.conf will do the job:

access_log udp://graylog.example.com:12201 graylog_vhost

The above assumes that a graylog.example.com server has a GELF input listener on a UDP port 12201.

References

http://docs.graylog.org/en/2.1/pages/gelf.html
http://www.squid-cache.org/Versions/v3/3.5/cfgman/access_log.html

Related Posts

Install Graylog Server 1.x on CentOS 7
Install Graylog Server 1.x on CentOS 6
Graylog Server Upgrade from 1.3.x to 2.0.x on CentOS 6
Set up MongoDB Authentication for Graylog

13 thoughts on “Send Squid Logs to Graylog

  1. Hello,
    How I get the size of download from user? How I get the total size of traffic in session by a user? The objective is build a report (in graylog) that shows the users that consumes more band.

    • Hi, I think that you’re going to need to check Squid’s size counters for this. There is a weblink provided (check the blog post) on where to find various format arguments for logformat.

    • This very blog post shows you a way to configure Squid log shipping to Graylog. Check logformat for more arguments if you need to, and add them to your JSON message. I don’t think there is anything else left to configure.

  2. Does anyone know how to generate reports with Graylog2? Squid access reports on screen and exporting to pdf?

    • The way I’ve got Squid (and other services) reports configured was to create an empty dashboard and add widgets. I’m sure there are dozens of different ways to achieve this, Graylog documentation is the best place to start. Not sure about exporting to PDF though.

  3. Hi,
    where do you save graylog_vhost format under graylog? It appears that we can use the same method to define new log format and send to graylog?

    • I need some directions in doing the following. I have jira server access_date.log (few hundred log files) copied on a local drive.
      I have create nxlog config following https://stackoverflow.com/questions/32410888/using-nxlog-with-iis-and-have-json-output (replaced the fields with access log fields). The nxlog starts fine with no error but there is no data in the graylog.

      the graylog has few others inputs which are working fine and receiving the data.
      ****************************************************************************************************
      ## This is a sample configuration file. See the nxlog reference manual about the
      ## configuration options. It should be installed locally and is also available
      ## online at http://nxlog.org/docs/

      ## Please set the ROOT to the folder your nxlog was installed into,
      ## otherwise it will not start.

      #define ROOT C:\Program Files\nxlog
      define ROOT C:\Program Files (x86)\nxlog

      Moduledir %ROOT%\modules
      CacheDir %ROOT%\data
      Pidfile %ROOT%\data\nxlog.pid
      SpoolDir %ROOT%\data
      LogFile %ROOT%\data\nxlog.log

      Module xm_json

      Module xm_fileop

      Module xm_syslog

      Module xm_gelf

      Module xm_csv
      Fields $IPAddress,$UserName,$DateTime,$HTTPAction,$ResponseCode,$Column10,$Column11
      FieldTypes string,string,string,string,string,string,string
      Delimiter “,”

      Module im_file
      File “C:\\Users\\jira\\Documents\\TempOut\\JiraAccessLogs\\accessLog.csv”

      if $raw_event =~ /^#/ drop(); \
      else
      {
      jira->parse_csv(); \
      to_json(); \
      }

      Module om_udp
      Host 172.17.1.87
      Port 5046
      OutputType GELF
      Exec $short_message = $raw_event; # Avoids truncation of the short_message field.
      Exec $collector_node_id = ‘JiraClient’;
      Exec $Hostname = hostname_fqdn();
      #Use the following line for debugging (uncomment the fileop extension above as well)
      Exec file_write(“C:\\Program Files (x86)\\nxlog\\data\\nxlog_output.log”, $raw_event);

      Path in => out

  4. hello
    I cannot make access_log udp://graylog.example.com:12201 graylog_vhost work
    Graylog never receiver the message, i tried in tcp as well…
    i succeed to send the message if i pass throught filebeat but the field message isnt parsed properly
    Any help with that ? thanks in advance

    • Try generating some traffic to see if it’s a network issue. You can use Netcat for both UDP and TCP. If you can connect, then you will see a hit on iptables (under pkts).

      If it’s not a network issue, then use Netcat to send a test packet to see if Graylog receives it. If it does, then check message formatting and make sure it matches Graylog’s parsing rules.

  5. I don’t get this working: The GELF messages arrive at the target host (checked via tcpdump), the format looks good (checked by writing the same format to a file on the squid host):

    {
    “version” : “1.1”,
    “_user_agent” : “curl/7.80.0”,
    “short_message” : “CONNECT http://www.cnn.com:443 HTTP/1.1″,
    “_squid_hierarchy_status” : “HIER_DIRECT”,
    “_request_size” : “1011”,
    “_http_status” : “200”,
    “_server_ip” : “151.101.113.67”,
    “_squid_request_status” : “TCP_TUNNEL”,
    “_http_method” : “CONNECT”,
    “_response_time” : “51”,
    “_client_ip” : “129.70.xx.yy”,
    “_squid_ip” : “129.70.xx.zz”,
    “level” : 6,
    “_http_referer” : “-“,
    “_http_url” : “www.cnn.com:443”,
    “timestamp” : 1641479750,
    “host” : “www.cnn.com:443”,
    “_reply_size” : “5572”,
    “_from_squid” : “true”
    }

    I did change

    “timestamp”:”%tl”

    to

    “timestamp”:%ts

    to comply with the specification (this should be seconds since epoch). But the importer shows no received messages. Any ideas? I am using Graylog 4.2.5.

Leave a Reply to Gianfrancisco Cancel reply

Your email address will not be published. Required fields are marked *