Graylog Server Upgrade from 1.3.x to 2.0.x on CentOS 6

Upgrading Graylog2 to Graylog (sounds confusing, I know).

We are going to upgrade Graylog 1.3.x that we set up some time ago to Graylog 2.0.x. Official upgrade instructions can be found here.

The reason for upgrading to Graylog 2.0.x and not to Graylog 2.1.x is that Graylog 2.0.x still uses separate listeners for the REST API and the web interface by default. Beginning with Graylog 2.1.0 it is possible to run both the REST API and the web interface on the same host/port-combination and this is now the default. Upgrade to Graylog 2.0.x is a major change on its own, therefore we want to keep things as less complicated as possible. We will upgrade to 2.1.x afterwards.

Software

Software used in this article:

  1. CentOS 6.8 minimal
  2. Elasticsearch 1.7.6 -> will be upgraded to Elasticsearch 2.4.2
  3. Graylog server 1.3.4 -> will be upgraded to Graylog server 2.0.3
  4. Graylog web 1.3.4 -> will be removed
  5. Java 1.8.0 openjdk -> no change
  6. MongoDB 3.2.11 -> no change

SELinux is set to enforcing mode. Firewall is on and is configured to allow incoming traffic on TCP ports 9000 and 12900. Graylog server’s internal IP is 10.8.143.13.

MongoDB

Graylog 2.x requires MongoDB 2.4 or newer. We are on 3.2 already, therefore no change is required.

# mongo --version
MongoDB shell version: 3.2.11

Elasticsearch 2.x

Graylog 2.0.0 and higher requires Elasticsearch 2.x.

Elasticsearch Migration Plugin

The first thing to do is to use the Elasticsearch Migration Plugin to help to highlight some potential pitfalls before an existing Elasticsearch 1.x cluster is upgraded to Elasticsearch 2.x. Do that before attempting an upgrade!

# find / -name plugin -type f
/usr/share/elasticsearch/bin/plugin
# cd /usr/share/elasticsearch/
# ./bin/plugin -i migration -u https://github.com/elastic/elasticsearch-migration/releases/download/v1.18/elasticsearch-migration-1.18.zip

Run the plugin: http://localhost:9200/_plugin/migration

In our case no issues were found, we can move forward.

Remove the plugin when done, as it won’t be compatible with Elasticsearch 2.x:

# ./bin/plugin remove migration

Elasticsearch Configuration Changes

Open Elasticsearch configuration file /etc/elasticsearch/elasticsearch.yml for editing and set the network.host value to a non-localhost address. Disable script.disable_dynamic as it is a deprecated setting. Make sure that you do those before starting Elasticsearch 2.x.

These are our settings for references, taken from the file /etc/elasticsearch/elasticsearch.yml:

cluster.name: graylog2
network.host: 10.8.143.13
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["10.8.143.13:9300"]

Note the existing cluster name, graylog2, we don’t change it.

The content of the file /etc/sysconfig/elasticsearch:

ES_HEAP_SIZE=4g
ES_STARTUP_SLEEP_TIME=5
MAX_OPEN_FILES=65536

Elasticsearch related security limits /etc/security/limits.conf:

elasticsearch soft nofile 65536
elasticsearch hard nofile 65536
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited

Index Range Types

This step needs to be performed before the update to Elasticsearch 2.x!

# curl -XGET localhost:9200/_all/_mapping/index_range; echo
{}

If the output is {} then we are not affected and can skip this step. Otherwise, we would need to delete the index_range type, Graylog does not use it anymore.

Graylog Index Template

Delete the index template:

# curl -X DELETE http://localhost:9200/_template/graylog-internal
{"acknowledged":true}

Graylog will automatically create the new index template on the next startup.

Index Retention and Rotation Settings

In 2.0.x the index rotation and retention settings have been moved from the Graylog server config file to the database and are now configurable via the web interface.

When we upgrade from a 1.x version and have modified any rotation/retention settings, we need to ensure that we keep our old settings in the config file so the migration process will add those old settings to the database. Otherwise the retention process will use the default settings and might remove a lot of indices.

Yum Repository

Edit the file /etc/yum.repos.d/elasticsearch.repo to include the new Elasticsearch 2.x repository:

[elasticsearch-1.7]
name=Elasticsearch repository for 1.7.x packages
baseurl=http://packages.elastic.co/elasticsearch/1.7/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=0

[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages
baseurl=https://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

Install Elasticsearch:

# yum clean all 
# yum install elasticsearch
# service elasticsearch restart

Check for any obvious issues:

# tail -f /var/log/elasticsearch/graylog2.log
# curl -X GET 'http://10.8.143.13:9200/'
{
  "name" : "Ororo Munroe",
  "cluster_name" : "graylog2",
  "cluster_uuid" : "59OCRIl5SOu8I61teQLpjQ",
  "version" : {
    "number" : "2.4.2",
    "build_hash" : "161c65a337d4b422ac0c805f284565cf2014bb84",
    "build_timestamp" : "2016-11-17T11:51:03Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.2"
  },
  "tagline" : "You Know, for Search"
}
# curl -X GET 'http://10.8.143.13:9200/_cluster/health?pretty'
{
  "cluster_name" : "graylog2",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 20,
  "active_shards" : 20,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

In case your Elasticsearch cluster status is red, be advised that it takes several minutes for it to become green.

Graylog 2.0.x

Remove the old Graylog 1.3 repository and install the new one for 2.0.

# yum remove graylog-1.3-repository-el6
# rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-2.0-repository_latest.rpm

Also remove the Graylog web package as it’s not compatible with Graylog 2.0.x.

# yum remove graylog-web

Install Graylog (this will upgrade to the version 2.0.x):

# yum clean all
# yum install graylog-server

Once finished, open the file /etc/graylog/server/server.conf for editing and put/modify the following settings:

is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = <passwd>
root_password_sha2 = <passwd>
rest_listen_uri = http://10.8.143.13:12900/
rest_transport_uri = http://10.8.143.13:12900/
web_listen_uri = http://10.8.143.13:9000/
elasticsearch_index_prefix = graylog2
elasticsearch_cluster_name = graylog2
elasticsearch_discovery_zen_ping_multicast_enabled = false
elasticsearch_discovery_zen_ping_unicast_hosts = 10.8.143.13:9300
elasticsearch_cluster_discovery_timeout = 10000
mongodb_useauth = false
mongodb_uri = mongodb://localhost/graylog2

Note that we don’t change the cluster name nor the DB from graylog2 to graylog. Restart the service:

# service graylog-server restart

Check for issues:

# tail -f /var/log/graylog-server/server.log
# curl -X GET 'http://10.8.143.13:12900/system/cluster/node'
{"cluster_id":"6ddf78b8-3fae-43aa-a3d2-8cdf7ddca900","node_id":"604e0a3a-891b-47ef-a08e-66c68e604b82","type":"server","transport_address":"http://10.8.143.13:12900/","last_seen":"2016-12-12T18:20:27.000Z","short_node_id":"604e0a3a","hostname":"vm-graylog","is_master":true}

All done, we should be able to log into Graylog WebUI on http://10.8.143.13:9000/.

Upgrade Graylog 2.0.x to 2.1.x

Once everything is working as expected and we are happy with the upgrade to the version 2.0.x, we can upgrade to 2.1.x. We need to add a new repository:

# rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-2.1-repository_latest.rpm

Upgrade Graylog. This will keep our existing configuration.

# yum clean all
# yum install graylog-server

Restart the service:

# service graylog-server restart

Optionally, to run the REST API and the web interface on the same port, we will need the following:

rest_listen_uri = http://10.8.143.13:9000/api/
web_enable = true
web_listen_uri = http://10.8.143.13:9000/
web_endpoint_uri = http://10.8.143.13:9000/api/
rest_transport_uri = http://10.8.143.13:9000/api/

More info about configuring Graylog 2.1.x web interface can be found here.

Leave a Reply

Your email address will not be published. Required fields are marked *