Set up Puppet Server with Hiera on CentOS 6

Part 7 of setting up a Linux home lab environment with VirtualBox. Check this blog post for more info.

Open Source Puppet is a declarative, model-based configuration management solution that lets us define the state of our home lab infrastructure, using the Puppet language.

In other words, Puppet is a configuration management system.

Hiera is a key/value lookup tool for configuration data, built to make Puppet better and let us set node-specific data without repeating ourselves.

Software

Software used in this article:

1. CentOS 6
2. Puppet 3.8

Installation

Add Puppet repository:

# rpm -ivh https://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm

Install Puppet server (Puppet version 3.8 installs Hiera as a dependency):

# yum install -y puppet-server

Configuration: Puppet Master Server

Open /etc/puppet/puppet.conf for editing and configure as below:

[main]
    # The Puppet log directory
    logdir = /var/log/puppet
    # Where Puppet PID files are kept
    rundir = /var/run/puppet
    # Where SSL certificates are kept
    ssldir = $vardir/ssl

    server = puppet.hl.local
    pluginsync = true
    parser = future

    # Merge recursively; in the event of a conflict,
    # allow higher priority values to win
    merge_behaviour = deeper

    # Environment variable that is used by Hiera
    environmet = prod

[agent]
    # The file in which puppetd stores a list of the classes
    # associated with the retrieved configuration
    classfile = $vardir/classes.txt
  
  # Where puppetd caches the local configuration
    localconfig = $vardir/localconfig

[master]
    # The comma-separated list of alternative DNS names to use for the local host.
    # When the node generates a CSR for itself, these are added to the request as 
    # the desired subjectAltName in the certificate
    dns_alt_names = puppet,puppet.hl.local
    always_cache_features = true

    # If set to false, never autosign even if an autosign.conf file is written
    autosign = true

When finished, start the puppetmaster service:

# /etc/init.d/puppetmaster start

Allow the service to start on boot:

# chkconfig puppetmaster on

Puppet master is now up and running, but does nothing useful.

Installation: Puppet Modules

We will use the following Puppet modules:

# puppet module install puppetlabs-firewall ;\
 puppet module install puppetlabs-ntp ;\
 puppet module install erwbgy-limits ;\
 puppet module install thias-sysctl ;\
 puppet module install spiette-selinux ;\
 puppet module install Aethylred-postfix ;\
 puppet module install puppetlabs-apache ;\
 puppet module install puppetlabs-mysql

Configuration: Firewall Module

Create a custom firewall module:

# mkdir -p /etc/puppet/modules/lsn_firewall/manifests

Put some basic metadata so that it is seen when puppet module list is run:

# cat << EOL > /etc/puppet/modules/lsn_firewall/metadata.json
{
  "name": "lsn_firewall",
  "version": "1.0.0",
  "author": "www.lisenet.com",
  "summary": "Manages iptables firewall",
  "license": "",
  "source": "",
  "project_page": "",
  "issues_url": "",
  "operatingsystem_support": [],
  "requirements": [],
  "dependencies": []
}
EOL

Create the lsn_firewall::pre and lsn_firewall::post Classes

Not much effort will be put on explaining this particular section as it is heavily based on the PuppetForge article: https://forge.puppetlabs.com/puppetlabs/firewall

Our approach will employ a whitelist setup, so we can define what rules we want and everything else is ignored rather than removed.

The run order of the firewall rules will be:

1. The rules in lsn_firewall::pre,
2. The rules defined in code (lsn_firewall/manifests/init.pp),
3. The rules lsn_firewall::post.

The rules in the pre and post classes will be fairly general: allow new incoming SSH connections, allow all related/established traffic, log and reject everything else.

# cat << EOL > /etc/puppet/modules/lsn_firewall/manifests/pre.pp
class lsn_firewall::pre {
        Firewall {
                require => undef,
        }

        # Default firewall rules
        firewall { '000 accept all icmp':
                proto   => 'icmp',
                action  => 'accept',
        }->
        firewall { '001 accept all lo interface':
                proto   => 'all',
                iniface => 'lo',
                action  => 'accept',
        }->
        firewall { '002 Allow ssh traffic':
                dport    => [ 22 ],
                proto   => 'tcp',
                state   => ['NEW'],
                action  => 'accept',
        }
}
EOL

# cat << EOL >/etc/puppet/modules/lsn_firewall/manifests/post.pp
class lsn_firewall::post {
  firewall { '997 accept related / established':
    proto   => 'all',
    state   => ['RELATED', 'ESTABLISHED'],
    action  => 'accept',
  }->
  firewall { '998 Log sessions':
    jump  => 'LOG',
    log_level => '4',
    log_prefix => 'iptables_input ',
  }->
  firewall { '999 drop all':
    proto   => 'all',
    action  => 'drop',
    before  => undef,
  }
}
EOL

As may be seen above, the rules allow basic networking and ensure that existing connections are not closed.

Create Firewall Rules

We are going to create firewall rules for the following services:

1. Puppet master (TCP 8140)
2. HTTP/S (TCP 80,443)
3. MySQL (TCP 3306)
4. DNS (TCP 53, UDP 53)
5. DHCP (UDP 67)
6. NTP (UDP 123)
7. SMTP (TCP 25)

Many more services can be added as and when required. Most of the rules will be restricted to internal 10.0.0.0/8 network.

# cat << EOL > /etc/puppet/modules/lsn_firewall/manifests/init.pp
class fw_puppet {
    firewall { '004 Allow puppet traffic':
        dport   => [ 8140 ],
        source  => '10.0.0.0/8',
        proto   => tcp,
        action  => accept,
    }
}

class fw_http {
    firewall { '005 Allow http/s traffic':
        dport   => [ 80, 443 ],
        source  => '10.0.0.0/8',
        proto   => tcp,
        action  => accept,
    }
}

class fw_mysql {
    firewall { '006 Allow MySQL traffic':
        dport   => [ 3306 ],
        source  => '10.0.0.0/8',
        proto   => tcp,
        action  => accept
    }
}

class fw_dns {
    firewall { '007 Allow BIND traffic TCP':
        dport   => [ 53 ],
        source  => '10.0.0.0/8',
        proto   => tcp,
        action  => accept
    }
    firewall { '008 Allow BIND traffic UDP':
        dport   => [ 53 ],
        source  => '10.0.0.0/8',
        proto   => udp,
        action  => accept
    }
}

class fw_dhcp {
    firewall { '009 Allow DHCP traffic':
        dport   => [ 67 ],
        proto   => udp,
        action  => accept
    }
}

class fw_ntp {
    firewall { '010 Allow NTP traffic':
        dport   => [ 123 ],
        source  => '10.0.0.0/8',
        proto   => udp,
        action  => accept
    }
}

class fw_smtp {
    firewall { '011 Allow SMTP traffic':
        dport   => [ 25 ],
        source  => '10.0.0.0/8',
        proto   => tcp,
        action  => accept
    }
}
EOL

Now, we want to set up a metatype to purge unmanaged firewall resources, and then set up the default parameters for all of the firewall rules. This should ensure that the pre and post classes are run in the correct order.

We also have to declare the lsn_firewall::pre and lsn_firewall::post classes to satisfy dependencies.

The last line is to assign classes to nodes using the hiera_include function (this is used later in the article).

# cat << EOL > /etc/puppet/manifests/site.pp
# Clear any existing IPv4 rules and make sure that only
# rules defined in Puppet exist on the machine
resources { "firewall":
  purge => true
}

# Ensure that the pre and post classes are run in the correct
# order to avoid locking us out of our box during the first
# Puppet run
Firewall {
  before  => Class['lsn_firewall::post'],
  require => Class['lsn_firewall::pre'],
}

# Declare the pre and post classes to satisfy dependencies
class { ['lsn_firewall::pre', 'lsn_firewall::post']: }

# Assign classes to nodes using the hiera_include function
hiera_include('classes')
EOL

Configuration: Hiera

Hierarchy and Tree Structure

This article assumes that the reader (at least) knows what Hiera is and how it is supposed to work.

We are going to use a json backend with a Hiera datadir of /etc/puppet/hieradata.

Hierarchy will be as follows:

1. fqdn – comes from facter,
2. server_role – comes from a configuration file (/etc/puppet/server_role), see below,
3. environment – comes from /etc/puppet/puppet.conf file.

We are going to use the following tree structure (in order of priority – lowest first) under /etc/puppet/hieradata/:

1. common.json – applies to all nodes but can have settings overridden by any other config,
2. env/envname.json – the environment name, for example “prod”,
3. server_role/role.json – the server role, for example “puppetmaster”, “mysql” etc,
4. node/fqdn.json – specific node configuration (not in use really, except where a node needs some specific configuration that cannot be picked up by all the above).

Many of these are defined by variables or facts (facter). Whenever a Hiera lookup is triggered from Puppet, Hiera receives a copy of all of the variables currently available to Puppet, including both top scope and local variables.

We can load and check Puppet specific facts with facter:

# facter -p

Create a new server_role fact:

# cat << EOL > /etc/puppet/modules/stdlib/lib/facter/server_role.rb
Facter.add("server_role") do
  setcode do
    Facter::Util::Resolution.exec("cat /etc/puppet/server_role")
  end
end
EOL

The server_role fact will be collected from /etc/puppet/server_role (this file needs to be created).

Config

Create a Hiera configuration file:

# cat << EOL > /etc/puppet/hiera.yaml
---
:merge_behaviour: deeper
:logger: console
:backends:
    - json
:json:
    :datadir: /etc/puppet/hieradata
:hierarchy:
    - "node/%{::fqdn}"
    - "server_role/%{::server_role}"
    - "env/%{::environment}"
    - common
EOL

Create a symlink:

# ln -sf /etc/puppet/hiera.yaml /etc/hiera.yaml

Create a hieradata folder structure:

# mkdir -p /etc/puppet/hieradata/{node,server_role,env}

Define the server role of the Puppet master server:

# echo "puppetmaster" > /etc/puppet/server_role

Create hiera data source files for environments:

# touch /etc/puppet/hieradata/env/{prod.json,dev.json}

Create hiera data source files for server roles:

# touch /etc/puppet/hieradata/server_role/{default.json,puppetmaster.json,mysql.json}

We have only three server roles in this example, a puppetmaster role for the Puppet master server, a mysql server role to configure MySQL servers, and a default server role for everything else.

We are going to configure environments in the way that SELinux is enabled in prod but is set to permissive in dev. Configuration also includes some Apache stuff for information disclosure.

Configure environments:

# cat << EOL > /etc/puppet/hieradata/env/prod.json
{
  "classes": [
      "selinux"
      ],
  "selinux::mode": "enforcing",
  "apache::server_signature": "Off",
  "apache::trace_enable": "Off",
  "apache::server_tokens": "Prod"
}
EOL

# cat << EOL > /etc/puppet/hieradata/env/dev.json
{
  "classes": [
      "selinux"
      ],
  "selinux::mode": "permissive",
  "apache::server_signature": "On",
  "apache::trace_enable": "On",
  "apache::server_tokens": "Full"
}
EOL

Now, we can create a Hiera configration for the puppetmaster server role. The configuration will do the following:

1. Configure firewall to allow incoming traffic for the following services:
   1. Puppet, HTTP/S, DNS, DHCP, NTP, SMTP,
2. Configure NTP server.

Create the file:

# cat << EOL > /etc/puppet/hieradata/server_role/puppetmaster.json
{
  "classes": [
    "fw_puppet",
    "fw_http",
    "fw_dns",
    "fw_dhcp",
    "fw_ntp",
    "fw_smtp",
    "ntp"
  ],
  "ntp::enable":  true,
    "ntp::servers": [
      "0.uk.pool.ntp.org iburst",
      "1.uk.pool.ntp.org iburst",
      "2.uk.pool.ntp.org iburst",
      "3.uk.pool.ntp.org iburst"
    ],
  "ntp::servers": [
      "0.uk.pool.ntp.org",
      "1.uk.pool.ntp.org",
      "2.uk.pool.ntp.org",
      "3.uk.pool.ntp.org"
    ],
  "ntp::interfaces": [ "0.0.0.0" ],
  "ntp::restrict": [
    "-4 default kod nomodify notrap nopeer noquery",
    "-6 default kod nomodify notrap nopeer noquery",
    "127.0.0.1",
    "-6 ::1",
    "10.0.0.0 mask 255.0.0.0 nomodify notrap nopeer"
  ]
}
EOL

We will also create a Hiera configuration for the mysql server role that will do the following:

1. Configure firewall to allow incoming MySQL traffic,
2. Configure Postfix relay,
3. Configure custom Linux security limits,
4. Configure MySQL server with sensible parameters.

Create the file:

# cat << EOL > /etc/puppet/hieradata/server_role/mysql.json
{
  "classes": [
      "fw_mysql",
      "postfix",
      "mysql::server"
  ],
  "postfix::relayhost": "smtp.hl.local",
  "postfix::inet_interfaces": "localhost",
  "limits": {
      "mysql": {
          "nproc": {
              "soft": "4096"
          },
          "nofile": {
              "soft": "4096"
          }
      }
  },
  "mysql::server::create_root_user": "true",
  "mysql::server::root_password": "passwd",
  "mysql::server::create_root_my_cnf": "true",
  "mysql::server::remove_default_accounts": "false",
  "mysql::server::restart": "true",
  "mysql::server::override_options": {
      "mysqld": {
          "bind-address": "0.0.0.0",
          "datadir": "/var/lib/mysql",
          "socket": "/var/lib/mysql/mysql.sock",
          "symbolic-links": "0",
          "max_allowed_packet": "16M",
          "max_heap_table_size": "256M",
          "max_connections": "100",
          "wait_timeout": "60",
          "open_files_limit": "4096",
          "general_log_file": "/var/log/mysql/mysql.log",
          "general_log": "0",
          "server_id": "1",
          "expire_logs_days": "7",
          "max_binlog_size": "1G",
          "query_cache_type": "1",
          "query_cache_limit": "16M",
          "thread_cache_size": "8",

          "key_buffer_size": "64M",
          "innodb_buffer_pool_size": "32M",
          "innodb_additional_mem_pool_size": "32M",
          "innodb_log_buffer_size": "10M",
          "tmp_table_size": "256M",
          "query_cache_size": "64M",

          "read_buffer_size": "1M",
          "read_rnd_buffer_size": "256K",
          "sort_buffer_size": "2M",
          "join_buffer_size": "1M",
          "thread_stack": "256K",
          "binlog_cache_size": "32K",

          "innodb_file_per_table": "1",
          "innodb_flush_method": "fsync",
          "innodb_flush_log_at_trx_commit": "1",
          "innodb_log_file_size": "32M",
          "innodb_lock_wait_timeout": "100",
          "innodb_data_file_path": "ibdata1:16M:autoextend:max:2048M"
      },
      "client": {
          "socket": "/var/lib/mysql/mysql.sock"
      },
      "mysqld_safe": {
          "socket": "/var/lib/mysql/mysql.sock"
      },
      "mysqldump": {
          "max_allowed_packet": "16M"
      }
  }
}
EOL

The default server role has a postfix relay configured only:

# cat << EOL > /etc/puppet/hieradata/server_role/default.json
{
  "classes": [
      "postfix"
  ],
  "postfix::relayhost": "smtp.hl.local",
  "postfix::inet_interfaces": "localhost"
}
EOL

The last bit is to create a Hiera common data source file. This file will provide any default values we want to use when Hiera cannot find a match for a given key elsewhere in our hierarchy.

The default configuration will cover the following:

1. Firewall,
2. NTP,
3. Postfix relay,
4. Linux security limits,
5. Sensible sysctl kernel parameters.

Create the file:

# cat << EOL > /etc/puppet/hieradata/common.json
{
  "classes": [
      "firewall",
      "ntp",
      "limits",
      "sysctl::base"
  ],
  "apache::serveradmin": "webmaster@localhost",
  "apache::server_signature": "Off",
  "apache::trace_enable": "Off",
  "apache::server_tokens": "Prod",
  "ntp::package_ensure": "present",
  "ntp::service_enable": true,
  "ntp::service_ensure": "running",
  "ntp::servers": [ "ntp.hl.local" ],
  "ntp::iburst_enable": true,
  "ntp::interfaces": [ "127.0.0.1" ],
  "ntp::restrict": [
    "default ignore",
    "-6 default ignore",
    "127.0.0.1",
    "-6 ::1"
  ],
  "limits": {
      "*": {
          "nofile": {
              "soft": "2048",
              "hard": "65536"
          },
          "nproc": {
              "soft": "2048",
              "hard": "16384"
          },
          "locks": {
              "soft": "2048",
              "hard": "2048"
          },
          "stack": {
              "soft": "10240",
              "hard": "32768"
          },
          "fsize": {
              "soft": " 33554432",
              "hard": "67108864"
          },
          "memlock": {
              "soft": "64",
              "hard": "64"
          },
          "core": {
              "hard": "0"
          }
      },
      "root": {
          "nofile": {
              "soft": "2048",
              "hard": "65536"
          },
          "nproc": {
              "soft": "2048",
              "hard": "16384"
          },
          "stack": {
              "soft": "10240",
              "hard": "32768"
          },
          "fsize": {
              "soft": "33554432"
          }
      }
  },
  "sysctl::base::values": {
      "fs.suid_dumpable": { "value": "0" },
      "vm.swappiness": { "value": "0" },
      "kernel.panic": { "value": "10" },
      "kernel.sysrq": { "value": "0" },
      "kernel.core_uses_pid": { "value": "1" },
      "kernel.dmesg_restrict": { "value": "1" },
      "kernel.kptr_restrict": { "value": "1" },
      "kernel.msgmnb": { "value": "65536" },
      "kernel.msgmax": { "value": "65536" },
      "kernel.shmmax": { "value": "1073741824" },
      "kernel.shmall": { "value": "262144" },
      "kernel.exec-shield": { "value": "1" },
      "kernel.randomize_va_space": { "value": "2" },
      "kernel.pid_max": { "value": "32768" },
      "net.core.wmem_max": { "value": "12582912" },
      "net.core.rmem_max": { "value": "12582912" },
      "net.core.netdev_max_backlog": { "value": "5000" },
      "net.ipv4.ip_forward": { "value": "0" },
      "net.ipv4.tcp_syncookies": { "value": "1" },
      "net.ipv4.tcp_rmem": { "value": "4096 87380 16777216" },
      "net.ipv4.tcp_wmem": { "value": "4096 65536 16777216" },
      "net.ipv4.tcp_window_scaling": { "value": "1" },
      "net.ipv4.tcp_sack": { "value": "1" },
      "net.ipv4.tcp_timestamps": { "value": "1" },
      "net.ipv4.tcp_no_metrics_save": { "value": "1" },
      "net.ipv4.conf.all.accept_source_route": { "value": "0"},
      "net.ipv4.conf.default.accept_source_route": { "value": "0" },
      "net.ipv4.conf.all.send_redirects": { "value": "0" },
      "net.ipv4.conf.all.accept_redirects": { "value": "0" },
      "net.ipv4.conf.all.secure_redirects": { "value": "0" },
      "net.ipv4.conf.default.send_redirects": { "value": "0" },
      "net.ipv4.conf.default.accept_redirects": { "value": "0" },
      "net.ipv4.conf.default.secure_redirects": { "value": "0" },
      "net.ipv4.conf.all.rp_filter": { "value": "1" },
      "net.ipv4.conf.default.rp_filter": { "value": "1" },
      "net.ipv4.conf.all.log_martians ": { "value": "1" },
      "net.ipv4.conf.default.log_martians": { "value": "1" },
      "net.ipv4.icmp_echo_ignore_broadcasts": { "value": "1" },
      "net.ipv4.icmp_ignore_bogus_error_messages": { "value": "1" },
      "net.ipv4.icmp_ignore_bogus_error_responses": { "value": "1" },
      "net.bridge.bridge-nf-call-iptables": { "value": "0" },
      "net.bridge.bridge-nf-call-arptables": { "value": "0" },
      "net.ipv6.conf.all.rp_filter": { "value": "1" },
      "net.ipv6.conf.all.accept_ra": { "value": "0" },
      "net.ipv6.conf.default.accept_ra": { "value": "0" },
      "net.ipv6.conf.all.accept_redirects": { "value": "0" },
      "net.ipv6.conf.default.accept_redirects": { "value": "0" }
  }
}
EOL

Check Puppet module list:

# puppet module list

It is time to manually trigger a Puppet client run on the puppetmaster:

# puppet agent -t

We should now see our master node connected:

# puppet cert list -all