Send Squid Logs to Graylog

Using GELF to send Squid logs to Graylog.

We are going to use the same approach as we did for Apache.

Squid 3.x Configuration

We are going to define a new logformat format called graylog_vhost to format a reverse proxy access log into a GELF format (JSON message):

logformat graylog_vhost {"version":"1.1","host":"%{Host}>h","short_message":"%rm %ru HTTP/%rv","level":6,"timestamp":"%tl","_client_ip":"%>a","_squid_ip":"%la","_server_ip":"%<a","_response_time":"%tr","_request_size":"%>st","_reply_size":"%<st","_http_url":"%ru","_http_status":"%>Hs","_http_method":"%rm","_http_referer":"%{Referer}>h","_user_agent":"%{User-Agent}>h","_squid_request_status":"%Ss","_squid_hierarchy_status":"%Sh","_from_squid":"true"}

Please check http://www.squid-cache.org/Versions/v3/3.5/cfgman/logformat.html for other format arguments that are available.

The below is a human-readable format:

{
   "version":"1.1",
   "host":"%{Host}>h",
   "short_message":"%rm %ru HTTP/%rv",
   "level":6,
   "timestamp":"%tl",
   "_client_ip":"%>a",
   "_squid_ip":"%la",
   "_server_ip":"%<a", 
   "_response_time":"%tr", 
   "_request_size":"%>st",
   "_reply_size":"%<st", 
   "_http_url":"%ru", 
   "_http_status":"%>Hs",
   "_http_method":"%rm",
   "_http_referer":"%{Referer}>h",
   "_user_agent":"%{User-Agent}>h",
   "_squid_request_status":"%Ss",
   "_squid_hierarchy_status":"%Sh",
   "_from_squid":"true"
}

Squid is capable of sending each access log line as text data to a TCP or a UDP receiver. We can use it to send logs to Graylog. The following line in the Squid configuration file squid.conf will do the job:

access_log udp://graylog.example.com:12201 graylog_vhost

The above assumes that a graylog.example.com server has a GELF input listener on a UDP port 12201.

References

http://docs.graylog.org/en/2.1/pages/gelf.html
http://www.squid-cache.org/Versions/v3/3.5/cfgman/access_log.html

Related Posts

Install Graylog Server 1.x on CentOS 7
Install Graylog Server 1.x on CentOS 6
Graylog Server Upgrade from 1.3.x to 2.0.x on CentOS 6
Set up MongoDB Authentication for Graylog

6 thoughts on “Send Squid Logs to Graylog

  1. Hello,
    How I get the size of download from user? How I get the total size of traffic in session by a user? The objective is build a report (in graylog) that shows the users that consumes more band.

    • Hi, I think that you’re going to need to check Squid’s size counters for this. There is a weblink provided (check the blog post) on where to find various format arguments for logformat.

    • This very blog post shows you a way to configure Squid log shipping to Graylog. Check logformat for more arguments if you need to, and add them to your JSON message. I don’t think there is anything else left to configure.

  2. Does anyone know how to generate reports with Graylog2? Squid access reports on screen and exporting to pdf?

    • The way I’ve got Squid (and other services) reports configured was to create an empty dashboard and add widgets. I’m sure there are dozens of different ways to achieve this, Graylog documentation is the best place to start. Not sure about exporting to PDF though.

Leave a Reply

Your email address will not be published. Required fields are marked *