Active/Passive MySQL High Availability Pacemaker Cluster with DRBD on CentOS 7

We are going to build a two-node active/passive MariaDB HA cluster using Pacemaker and Corosync. 

Before We Begin

We will use MariaDB, which is a binary drop in replacement for MySQL. MySQL is no longer in CentOS’s repositories and MariaDB has become the default database system offered.

MariaDB claims to have a lot of new options, extension, storage engines and bug fixes that are not in MySQL. Check here for more info about MariaDB.

No shared storage will be required. At any point in time, the MariaDB service will be active on one cluster node.

The convention followed in the article is that [ALL] # denotes a command that needs to be run on all cluster nodes.

Software

Software used in this article:

  1. CentOS Linux release 7.2.1511 (Core)
  2. kernel-3.10.0-327.4.4.el7
  3. pacemaker-1.1.13
  4. corosync-2.3.4
  5. pcs-0.9.143
  6. resource-agents-3.9.5
  7. drbd-8.4 (kmod-drbd84-8.4.7)
  8. MariaDB 5.5.44

Networking, Firewall and SELinux Configuration

We have two CentOS 7 virtual machines on VirtualBox, named pcmk01 and pcmk02.

Networking

The following networks will be used:

  1. 10.8.8.0/24 – LAN with access to the Internet (host-only adapter),
  2. 172.16.21.0/24 – non-routable cluster heartbeat vlan for Corosync (internal network adapter),
  3. 172.16.22.0/24 – non-routable cluster heartbeat vlan for DRBD (internal network adapter).

Hostnames and IPs as defined in /etc/hosts file:

10.8.8.60 pcmkvip
10.8.8.61 pcmk01
10.8.8.62 pcmk02
172.16.21.11 pcmk01-cr
172.16.21.12 pcmk02-cr
172.16.22.11 pcmk01-drbd
172.16.22.12 pcmk02-drbd

We have set the following hostnames:

[pcmk01]# hostnamectl set-hostname pcmk01
[pcmk02]# hostnamectl set-hostname pcmk02

A simplified network configuration can be seen below.

Network configuration for the first node can be seen below, it is the same for the second node except the IPs which are specified above.

[pcmk01]# cat /etc/sysconfig/network-scripts/ifcfg-enp0s8
#Corosync ring0
NAME="enp0s8"
DEVICE="enp0s8"
IPADDR="172.16.21.11"
PREFIX="24"
TYPE="Ethernet"
IPV4_FAILURE_FATAL="yes"
IPV6INIT="no"
DEFROUTE="no"
PEERDNS="no"
PEERROUTES="no"
ONBOOT="yes"
[pcmk01]# cat /etc/sysconfig/network-scripts/ifcfg-enp0s9
#DRBD
NAME="enp0s9"
DEVICE="enp0s9"
IPADDR="172.16.22.11"
PREFIX="24"
TYPE="Ethernet"
IPV4_FAILURE_FATAL="yes"
IPV6INIT="no"
DEFROUTE="no"
PEERDNS="no"
PEERROUTES="no"
ONBOOT="yes"
[pcmk01]# cat /etc/sysconfig/network-scripts/ifcfg-enp0s17
#LAN
NAME="enp0s17"
DEVICE="enp0s17"
IPADDR="10.8.8.61"
PREFIX="24"
GATEWAY="10.8.8.1"
DNS1="8.8.8.8"
DNS2="8.8.4.4"
TYPE="Ethernet"
IPV4_FAILURE_FATAL="yes"
IPV6INIT="no"
DEFROUTE="yes"
PEERDNS="yes"
ONBOOT="yes"

Iptables

This article uses Iptables firewall. Note that CentOS 7 utilises FirewallD as the default firewall management tool.

We replaced FirewallD service with Iptables:

[ALL]# systemctl stop firewalld.service
[ALL]# systemctl mask firewalld.service
[ALL]# systemctl daemon-reload
[ALL]# yum install -y iptables-services
[ALL]# systemctl enable iptables.service
[ALL]# service iptables save

These are the iptables firewall rules that we have in use:

# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -s 10.8.8.0/24 -p tcp -m tcp --dport 22 -m state --state NEW -j ACCEPT
-A INPUT -s 172.16.21.0/24 -d 172.16.21.0/24 -m comment --comment Corosync -j ACCEPT
-A INPUT -s 172.16.22.0/24 -d 172.16.22.0/24 -m comment --comment DRBD -j ACCEPT
-A INPUT -s 10.8.8.0/24 -p tcp -m tcp --dport 3306 -m state --state NEW -j ACCEPT
-A INPUT -p udp -m multiport --dports 67,68 -m state --state NEW -j ACCEPT
-A INPUT -p udp -m multiport --dports 137,138,139,445 -j DROP
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -j LOG --log-prefix "iptables_input "
-A INPUT -j DROP

We have also disabled IPv6, open /etc/sysctl.conf for editing and place the following:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
[ALL]# sysctl -p

SELinux

SELinux is set to enforcing mode.

Install Pacemaker and Corosync

[ALL]# yum install -y pcs

The pcs will install pacemaker, corosync and resource-agents as dependencies.

For SELinux management:

[ALL]# yum install -y policycoreutils-python

In RHEL 7, we have to set up a password for the pcs administration account named hacluster:

[ALL]# echo "passwd" | passwd hacluster --stdin

Start and enable the service:

[ALL]# systemctl start pcsd.service
[ALL]# systemctl enable pcsd.service

Configure Corosync

Authenticate as the hacluster user. Authorisation tokens are stored in the file /var/lib/pcsd/tokens.

[pcmk01]# pcs cluster auth pcmk01-cr pcmk02-cr -u hacluster -p passwd
pcmk01-cr: Authorized
pcmk02-cr: Authorized

Generate and synchronise the Corosync configuration.

[pcmk01]# pcs cluster setup --name mysql_cluster pcmk01-cr pcmk02-cr

Start the cluster on all nodes:

[pcmk01]# pcs cluster start --all

Install DRBD and MariaDB

DRBD Installation

DRBD refers to block devices designed as a building block to form high availability clusters. This is done by mirroring a whole block device via an assigned network. DRBD can be understood as network based RAID-1.

Import the ELRepo package signing key, enable the repository and install the DRBD kernel module with utilities:

[ALL]# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
[ALL]# rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
[ALL]# yum install -y kmod-drbd84 drbd84-utils

To avoid issues with SELinux, for the time being, we are going to exempt DRBD processes from SELinux control:

[ALL]# semanage permissive -a drbd_t

LVM Volume for DRBD

Create a new 1GB logical volume for DRBD:

[pcmk01]# vgs
  VG         #PV #LV #SN Attr   VSize  VFree
  vg_centos7   1   3   0 wz--n- 63.21g 45.97g
[ALL]# lvcreate --name lv_drbd --size 1024M vg_centos7

Configure DRBD

Configure DRBD, use single-primary mode with replication protocol C.

[ALL]# cat << EOL >/etc/drbd.d/mysql01.res
resource mysql01 {
 protocol C;
 meta-disk internal;
 device /dev/drbd0;
 disk   /dev/vg_centos7/lv_drbd;
 handlers {
  split-brain "/usr/lib/drbd/notify-split-brain.sh root";
 }
 net {
  allow-two-primaries no;
  after-sb-0pri discard-zero-changes;
  after-sb-1pri discard-secondary;
  after-sb-2pri disconnect;
  rr-conflict disconnect;
 }
 disk {
  on-io-error detach;
 }
 syncer {
  verify-alg sha1;
 }
 on pcmk01 {
  address  172.16.22.11:7789;
 }
 on pcmk02 {
  address  172.16.22.12:7789;
 }
}
EOL

We have a resource named mysql01 which uses /dev/vg_centos7/lv_drbd as the lower-level device, and is configured with internal meta data.

The resource uses TCP port 7789 for its network connections, and binds to the IP addresses 172.16.22.11 and 172.16.22.12, respectively.

If case we run into problems, we have to ensure that a TCP 7789 port is open on a firewall for the DRBD interface and that the resource name matches the file name.

Create the local metadata for the DRBD resource:

[ALL]# drbdadm create-md mysql01

Ensuring that a DRBD kernel module is loaded, bring up the DRBD resource:

[ALL]# drbdadm up mysql01

For data consistency, tell DRBD which node should be considered to have the correct data (can be run on any node as both have garbage at this point):

[pcmk01]# drbdadm primary --force mysql01

Observe the sync:

[pcmk01]# drbd-overview
 0:mysql01/0  SyncSource Primary/Secondary UpToDate/Inconsistent 
	[=>..................] sync'ed: 11.8% (926656/1048508)K

Create a filesystem on the DRBD device and tune as required:

[pcmk01]# mkfs.ext4 -m 0 -L drbd /dev/drbd0
[pcmk01]# tune2fs -c 30 -i 180d /dev/drbd0

Mount the disk, we will populate it with MariaDB content shortly:

[pcmk01]# mount /dev/drbd0 /mnt

MariaDB Installation

[ALL]# yum install -y mariadb-server mariadb

Ensure the MariaDB service is disabled, as it will managed by pacemaker:

[ALL]# systemctl disable mariadb.service

Now start the MariaDB service manually on one of the cluster nodes:

[pcmk01]# systemctl start mariadb

We can install a fresh MariaDB database with the mysql_install_db command:

[pcmk01]# mysql_install_db --datadir=/mnt --user=mysql

Run secure installation:

[pcmk01]# mysql_secure_installation

We need to give the same SELinux policy as the MariaDB datadir. The mysqld policy stores data with multiple different file context types under the /var/lib/mysql directory. If we want to store the data in a different directory, we can use the semanage command to add file context.

[pcmk01]# semanage fcontext -a -t mysqld_db_t "/mnt(/.*)?"
[pcmk01]# restorecon -Rv /mnt

Please be advised that changes made with the chcon command do not survive a file system relabel, or the execution of the restorecon command. Always use semanage.

At this point our preparation is complete, we can unmount the temporarily mounted filesystem and stop the MariaDB service:

[pcmk01]# umount /mnt
[pcmk01]# systemctl stop mariadb

Last thing to do, we have to put some very basic my.cnf configuration:

[ALL]# cat << EOL > /etc/my.cnf
[mysqld]
symbolic-links=0
bind_address            = 0.0.0.0
datadir                 = /var/lib/mysql
pid_file                = /var/run/mariadb/mysqld.pid
socket                  = /var/run/mariadb/mysqld.sock

[mysqld_safe]
bind_address            = 0.0.0.0
datadir                 = /var/lib/mysql
pid_file                = /var/run/mariadb/mysqld.pid
socket                  = /var/run/mariadb/mysqld.sock

!includedir /etc/my.cnf.d
EOL

Configure Pacemaker Cluster

We want the configuration logic and ordering to be as below:

  1. Start: mysql_fs01 -> mysql_service01 -> mysql_VIP01,
  2. Stop: mysql_VIP01 -> mysql_service01 -> mysql_fs01.

Where mysql_fs01 is the filesystem resource, mysql_service01 is the mysqld service resource, and mysql_VIP01 is the floating virtual IP 10.8.8.60.

One handy feature pcs has is the ability to queue up several changes into a file and commit those changes atomically. To do this, we start by populating the file with the current raw XML config from the CIB:

[pcmk01]# pcs cluster cib clust_cfg

Disable STONITH. Be advised that a node level fencing configuration depends heavily on environment. You can check this page for Pacemaker STONITH device configuration on VMware.

[pcmk01]# pcs -f clust_cfg property set stonith-enabled=false

Set quorum policy to ignore:

[pcmk01]# pcs -f clust_cfg property set no-quorum-policy=ignore

Prevent the resources from moving after recovery as it usually increases downtime:

[pcmk01]# pcs -f clust_cfg resource defaults resource-stickiness=200

Create a cluster resource named mysql_data01 for the DRBD device, and an additional clone resource MySQLClone01 to allow the resource to run on both cluster nodes at the same time:

[pcmk01]# pcs -f clust_cfg resource create mysql_data01 ocf:linbit:drbd \
  drbd_resource=mysql01 \
  op monitor interval=30s
[pcmk01]# pcs -f clust_cfg resource master MySQLClone01 mysql_data01 \
  master-max=1 master-node-max=1 \
  clone-max=2 clone-node-max=1 \
  notify=true

Note the meta variables used:

master-max: how many copies of the resource can be promoted to master status,
master-node-max: how many copies of the resource can be promoted to master status on a single node,
clone-max: how many copies of the resource to start. Defaults to the number of nodes in the cluster,
clone-node-max: how many copies of the resource can be started on a single node,
notify: when stopping or starting a copy of the clone, tell all the other copies beforehand and when the action was successful.

Create a cluster resource named mysql_fs01 for the filesystem. Tell the cluster that the clone resource MySQLClone01 must be run on the same node as the filesystem resource, and that the clone resource must be started before the filesystem resource.

[pcmk01]# pcs -f clust_cfg resource create mysql_fs01 Filesystem \
  device="/dev/drbd0" \
  directory="/var/lib/mysql" \
  fstype="ext4"
[pcmk01]# pcs -f clust_cfg constraint colocation add mysql_fs01 with MySQLClone01 \
  INFINITY with-rsc-role=Master
[pcmk01]# pcs -f clust_cfg constraint order promote MySQLClone01 then start mysql_fs01

Create a cluster resource named mysql_service01 for the MariaDB service. Tell the cluster that the MariaDB service must be run on the same node as the mysql_fs01 filesystem resource, and that the filesystem resource must be started first.

[pcmk01]# pcs -f clust_cfg resource create mysql_service01 ocf:heartbeat:mysql \
  binary="/usr/bin/mysqld_safe" \
  config="/etc/my.cnf" \
  datadir="/var/lib/mysql" \
  pid="/var/lib/mysql/mysql.pid" \
  socket="/var/lib/mysql/mysql.sock" \
  additional_parameters="--bind-address=0.0.0.0" \
  op start timeout=60s \
  op stop timeout=60s \
  op monitor interval=20s timeout=30s
[pcmk01]# pcs -f clust_cfg constraint colocation add mysql_service01 with mysql_fs01 INFINITY
[pcmk01]# pcs -f clust_cfg constraint order mysql_fs01 then mysql_service01

Finally, create a cluster resource named mysql_VIP01 for the virtual IP 10.8.8.60.

[pcmk01]# pcs -f clust_cfg resource create mysql_VIP01 ocf:heartbeat:IPaddr2 \
 ip=10.8.8.60 cidr_netmask=32 \
 op monitor interval=30s

Why to use IPaddr2 and not IPaddr:

  1. IPaddr – manages virtual IPv4 addresses (portable version),
  2. IPaddr2 – manages virtual IPv4 addresses (Linux specific version).

The virtual IP mysql_VIP01 resource must be run on the same node as the MariaDB resource, naturally, and must be started the last. This is to ensure that all other resources are already started before we can connect to the virtual IP.

[pcmk01]# pcs -f clust_cfg constraint colocation add mysql_VIP01 with mysql_service01 INFINITY
[pcmk01]# pcs -f clust_cfg constraint order mysql_service01 then mysql_VIP01

Let us check the configuration:

[pcmk01]# pcs -f clust_cfg constraint
Location Constraints:
Ordering Constraints:
  promote MySQLClone01 then start mysql_fs01 (kind:Mandatory)
  start mysql_fs01 then start mysql_service01 (kind:Mandatory)
  start mysql_service01 then start mysql_VIP01 (kind:Mandatory)
Colocation Constraints:
  mysql_fs01 with MySQLClone01 (score:INFINITY) (with-rsc-role:Master)
  mysql_service01 with mysql_fs01 (score:INFINITY)
  mysql_VIP01 with mysql_service01 (score:INFINITY)
[pcmk01]# pcs -f clust_cfg resource show
 Master/Slave Set: MySQLClone01 [mysql_data01]
     Stopped: [ pcmk01-cr pcmk02-cr ]
 mysql_fs01	(ocf::heartbeat:Filesystem):	Stopped
 mysql_service01	(ocf::heartbeat:mysql):	Stopped
 mysql_VIP01	(ocf::heartbeat:IPaddr2):	Stopped

We can commit changes now and check cluster status:

[pcmk01]# pcs cluster cib-push clust_cfg
[pcmk01]# pcs status
[...]

Online: [ pcmk01-cr pcmk02-cr ]

Full list of resources:

 Master/Slave Set: MySQLClone01 [mysql_data01]
     Masters: [ pcmk01-cr ]
     Stopped: [ pcmk02-cr ]
 mysql_fs01     (ocf::heartbeat:Filesystem):    Started pcmk01-cr
 mysql_service01        (ocf::heartbeat:mysql): Started pcmk01-cr
 mysql_VIP01    (ocf::heartbeat:IPaddr2):	Started pcmk01-cr

[...]

Once the configuration has been committed, Pacemaker will:

  1. Start DRBD on both cluster nodes,
  2. Select one node for promotion to the DRBD Primary role,
  3. Mount the filesystem, configure the cluster IP address, and start the MariaDB server on the same node,
  4. Commence resource monitoring

We can test the MariaDB service by telneting into the virtual IP 10.8.8.60 on a TCP por 3306:

# telnet 10.8.8.60 3306
Trying 10.8.8.60...
Connected to 10.8.8.60.
Escape character is '^]'.
Host 'puppet.hl.local' is not allowed to connect to this MariaDB serverConnection closed by foreign host.

References

http://www.linux-ha.org/wiki/Resource_agents
http://www.linux-ha.org/doc/man-pages/re-ra-mysql.html
http://albertolarripa.com/2012/05/13/install-configure-mysql-cluster/

72 thoughts on “Active/Passive MySQL High Availability Pacemaker Cluster with DRBD on CentOS 7

    • Thanks. I have to agree with this one, not that many constructive articles for CentOS 7 on the net. I had some troubles migrating Pacemaker from RHEL 6 to RHEL 7, therefore putting it online seemed like a good idea.

  1. Hi Thomas,
    Excelent post. I haven’t tried jet but doesn’t seems hard to follow.

    Why did you choose to replicate databases by files and not using mysql replication implementation? Do you know if is slower than syncing by the classic method?

    Thanks, and congratulations for the guide.

    • Hi Felipe, Thanks for your feedback.

      I’m used to SAN based high availability solutions with active-passive management software and it was natural for me to use such approach for MySQL. I was migrating from cman (CentOS 6) to pcs (CentOS 7) in this particular case.

  2. at drbdadm create-md mysql01
    I get an error: ‘mysql01’ not defined in your config (for this host).

    • Check the DRBD config, ensure you got the DRBD resource name right, also check hostnames of your DRBD nodes – they need to match with the ones defined in the config.

  3. Hi,

    i am currently working on a school project where im using PCS cluster. i managed to get it ip and running with my apache webservers. now im working on mariadb cluster. most of the config is the same and for the additional info i used your guide. the only thing that is different is that im using a MS iscsi target server for the shared directory. i managed to get it all up and running. both nodes have mariadb and the exact same. the parameters in the mysql_service resource are all specified to the shared storage location (/mnt/iscsi_share) exept the binary and config. i dident touch those.

    the cluster is now working only on 1 node. altough it has exact the same config file. ik getting the following error code:
    mysql_service_start_0 on linux06 ‘unknown error’ (1): call=20, status=complete, exitreason=’MySQL server failed to start (pid=3307) (rc=0), please check your installation’,
    last-rc-change=’Thu Jun 2 15:05:18 2016′, queued=0ms, exec=4969ms

    do you have any idea whear i need to start looking or what may couse this problem. the database on linux 5 works and i can reach it and modify it.

    i hope you can understand my problem with this rubish english grammer :)

    • Do you have SELinux enabled? If so, can you post me the output of the following command:

      # ls -dZ /mnt/iscsi_share

      What filesystem do you use on /mnt/iscsi_share? Is it mounted on all cluster nodes?

      What operating system do run the cluster on?

    • Hi Tomas,

      Sorry for the late reply but my deadline is closing in and im very busy with the documentation. I managed to get it up and running correctly. im using ext.4 and run it on CentOS 7. The error had something to do with the constraints. I followed your guide on this. the db worked but was not able to shutdown. so i decided to remove them. Now the cluster worked like i said in the previous comment. Later i decided to give the constraints another try. I had one combining stonith device and filesystem. this one was giving me the problems. Now my cluster works with the following constraints.

      Location Constraints:
      Ordering Constraints:
        start mysql_fs then start mysql_service (kind:Mandatory)
      Colocation Constraints:
        mysql_service with mysql_fs (score:INFINITY)
        virtual_ip with mysql_service (score:INFINITY)

      thanks for your help and your guide! i would not have managed to make ik without it!

    • I’m glad you managed to sort it out.

      I see that you have a colocation constraint for the virtual_ip, but there is no ordering constraint for it. You should add one to ensure that all MySQL resources are started before you can connect to the virtual IP.

  4. Hi Tomas,

    Many thanks for your article. Currently, I am trying to deploy it through a two servers in VirtualBox. I have a question regarding the interfaces. For each IP address ( 10.8.8.61, 172.16.21.11 and 172.16.22.11 ), do you have a multiple interfaces in your servers or are they virtual interface? I would like to know which interface do you have to create it through vagrant.

    BR

    • When I create the interfaces ifcfg-enp0s8, ifcfg-enp0s9 and ifcfg-enp0s17 in a host-only interface of VirtualMachine it cannot auth. Besides, I only see eth interfaces as older mode.

    • My apologies, the LAN is on a host-only adapter, but both non-routable cluster heartbeat vlans use internal network adapters. I’ve updated the article.

      What OS do you use? CentOS 7 uses so-called predictable network interface names by default, therefore the actual interface names on your system may be different compared to the ones mentioned in the article. If you want to go back to traditional interface naming scheme (“eth0”, “eth1” etc), you can do so in udev.

    • I am also using CentOS 7. The issue that I have found that I was creating a VM with just one interface and when I created the enp interfaces communication between VMs wasn’t properly. So, auth step doesn’t work and I had to provision three interface for each network. After, I provisioned three interfaces and it create eth interfaces and communication between VMs was properly.

      Do you advice use the same network for the communication of pcmk nodes?

    • You can use a single network interface for everything (LAN, Corosync and DRBD) if you wish, for testing it hardly matters.

      For production, you likely want to have redundant interfaces (teamed/bonded LAN, two Corosync rings etc).

  5. I really love this article! Hard to find good info on MySQL, everybody has Apache+DRBD, but this is the beezneez!

    A nice trick for CentOS 7 to make eno/enp back to eth0 I found on NerdVittles…

    sed -i 's|quiet|quiet net.ifnames=0 biosdevdame=0|' /etc/default/grub
    grub2-mkconfig -o /boot/grub2/grub.cfg
    mv /etc/sysconfig/network-scripts/ifcfg-eno* /etc/sysconfig/network-scripts/ifcfg-eth0
    nano /etc/sysconfig/network-scripts/ifcfg-eth0 && reboot

    I wanted to ask, if you have or could create an article for CentOS 6 as you mentioned that you used to use that. I am building HA VoIP Servers and currently all the pre-packaged distros are on CentOS 6 and I can only get HA working on CentOS 7 with pcsd lol, its been a nightmare figuring out if I want to build VoIP by hand on 7 or stuggle with HA on CentOS6 ontop of a custom Distro. After reading this, I am leaning towards building a custom server with PIAF/IncrediblePBX + Linux HA. Thanks!

    • Glad you found it helpful. I’m afraid I have no plans to go back to CentOS 6.

      Not sure if it helps you, but cluster stack on CentOS 6 is based on cman, I may be wrong, but I think that crmsh is no longer available on RHEL (thus no longer available on CentOS), but pcs is fully supported starting CentOS 6.5.

      CentOS 6 has been around for quite some time now, I think that you’ll be able to find dozens of articles covering HA setup online.

    • I think I found a mistake/problem in the guide…

      pid_file                = /var/run/mariadb/mysqld.pid
      socket                  = /var/run/mariadb/mysqld.sock

      NOT

      pid_file                = /var/run/mysqld/mysqld.pid
      socket                  = /var/run/mysqld/mysqld.sock

      Bonus :

      pcs resource group add SQL-Group mysql_service01 mysql_fs01

      Just organizes the output of pcs in a nice OU.

      Was wondering if your interested/available for some paid support helping me get my system running, I am having some trouble, but I am going to create a new test bed this weekend and follow your guide to the letter and hope that works.

       Master/Slave Set: MySQLClone01 [mysql_data01]
           mysql_data01       (ocf::linbit:drbd):     FAILED node2 (unmanaged)
           mysql_data01       (ocf::linbit:drbd):     FAILED node1 (unmanaged)
       Resource Group: SQL-Group
           mysqld     (ocf::heartbeat:mysql): Stopped
           mysql_fs01 (ocf::heartbeat:Filesystem):    Stopped
      
      mysql_data01_stop_0 on node2 'not configured' (6): call=29, status=complete, last-rc-change='Wed Jun  8 21:27:08 2016', queued=1ms, exec=15ms
      (same thing for node1)

      is where I am at right now… I am literally just starting my career and got the go-ahead to build a HA Voip Server @ work and have been struggling getting MySQL Services moved over to DRBD/PaceMaker.

      Thanks for the great guide again!

    • Not a mistake as such as I actually use a mysqld named folder under /var/run, however, I forgot to include a step for creating it. I updated the article, thanks for spotting. Just to make it clear, the folder that’s created by default is /var/run/mariadb/.

      With regards to resource groups, you’re right, grouping resources can simplify Pacemaker configuration, as in this article, but it’s not mandatory to get things working. You simply need to ensure that resources run on the same host by creating a colocation constraint with a score of INFINITY.

      Looking at the error you got above, it seems like your DRBD may not be configured properly.

      As for paid support, I’m usually open-minded, however, as a rule of thumb I only tend to provide support for systems that I build myself. Supporting something that was created by other people (however good or bad it might be) isn’t my cup of tea really as more often than not, such systems prove to be built in a haphazard way. Disclaimer alert – don’t take it for granted, it’s just my experience talking.

  6. I seem to be stuck, this time I am following you guide 99% (Changed IP’s and hostnames)

    [[email protected] ~]# mysql_install_db –datadir=/mnt –user=mysql
    chown: changing ownership of ‘/mnt’: Read-only file system
    Cannot change ownership of the database directories to the ‘mysql’
    user. Check that you have the necessary permissions and try again.

    I never did hear from you, but I am not really looking for ongoing support, just a some money to help troubleshoot my configs in a VM. (I am starting to wonder if its go something to do with the VM part of the equation, since sdb1 is virtual, also its only 2GB, but I don’t think that should matter…)

  7. I just wanted to say I got it! As a thank you I will make a donation!

    4 others who may have the same issues I was.

    My issues :

    1. I was started with a preformatted disk, start with a brand new disk and use fdisk
    2. My VIP needed to have nic=eth0 because I didn’t follow the IP’s/Nics part of this guide

  8. Hi Tomas,

    I have the same issue than @FreeSoftwareServers regarding with the resource of mysql_data01. I configured properly DRBD because I got in first place the status SyncSource and after when it finishes the synchronization is Completed:
    # drbd-overview
    0:mysql01/0 SyncSource Primary/Secondary UpToDate/Inconsistent
    [====>……………] sync’ed: 26.3% (242600/327632)K

    # drbd-overview
    0:mysql01/0 Connected Primary/Secondary UpToDate/UpToDate

    It seems that drbd works properly but I created the mysql_data01 as you show and after check the status. It fails despite the status in the monitor is complete:

    mysql_data01 (ocf::linbit:drbd): FAILED (unmanaged)[ pcmk02-cr pcmk01-cr ]

    Failed Actions:
    * mysql_data01_stop_0 on pcmk02-cr ‘not configured’ (6): call=6, status=complete, exitreason=’none’,
    last-rc-change=’Mon Jun 27 11:43:17 2016′, queued=0ms, exec=198ms
    * mysql_data01_stop_0 on pcmk01-cr ‘not configured’ (6): call=6, status=complete, exitreason=’none’,
    last-rc-change=’Mon Jun 27 11:43:17 2016′, queued=0ms, exec=117ms

    I have no errors with filesystem resource. Currently, I am using two vagrant machines to test it. Do you know wich kind of issue it could be? Because the status seems ok but pacemaker perform it like an error.

    Many thanks for your help in advance,
    BR

    • What’s the value of drbd_resource that you passed to the Pacemaker when creating the cluster resource named mysql_data01?

    • #pcs resource create mysql_data01 ocf:linbit:drbd drbd_resource=mysql01 op monitor interval=30s

    • The log is located in /var/log/cluster/corosync.log. Sorry for the stuff:

      Jun 27 14:30:06 [2053] pcmk01 crmd: error: crm_abort: pcmkRegisterNode: Triggered assert at xml.c:594 : node->type == XML_ELEMENT_NODE
      Jun 27 14:30:06 [2053] pcmk01 crmd: notice: process_lrm_event: Operation mysql_data01_monitor_0: not configured (node=pcmk01-cr, call=5, rc=6, cib-update=49, confirmed=true)
      Jun 27 14:30:06 [2048] pcmk01 cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/49)
      Jun 27 14:30:06 [2048] pcmk01 cib: info: cib_perform_op: Diff: — 0.9.0 2
      Jun 27 14:30:06 [2048] pcmk01 cib: info: cib_perform_op: Diff: +++ 0.9.1 (null)
      Jun 27 14:30:06 [2048] pcmk01 cib: info: cib_perform_op: + /cib: @num_updates=1
      Jun 27 14:30:06 [2048] pcmk01 cib: info: cib_perform_op: + /cib/status/node_state[@id=’1′]: @crm-debug-origin=do_update_resource
      Jun 27 14:30:06 [2048] pcmk01 cib: info: cib_perform_op: ++ /cib/status/node_state[@id=’1′]/lrm[@id=’1′]/lrm_resources:
      Jun 27 14:30:06 [2048] pcmk01 cib: info: cib_perform_op: ++ <lrm_rsc_op id="mysql_data01_last_failure_0" operation_key="mysql_data01_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.10" transition-key="4:5:7:0e367aab-5184-4c54-87da-0562471c2635" transition-magic="0:6;4:5:7:0e367aab-5184-4c54-87da-0562471c2635" on_node="pcmk01-cr" call-id="5" rc-code="6" op-status="0" interval="0" last-run="146
      Jun 27 14:30:06 [2048] pcmk01 cib: info: cib_perform_op: ++ <lrm_rsc_op id="mysql_data01_last_0" operation_key="mysql_data01_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.10" transition-key="4:5:7:0e367aab-5184-4c54-87da-0562471c2635" transition-magic="0:6;4:5:7:0e367aab-5184-4c54-87da-0562471c2635" on_node="pcmk01-cr" call-id="5" rc-code="6" op-status="0" interval="0" last-run="1467037806"
      Jun 27 14:30:06 [2048] pcmk01 cib: info: cib_perform_op: ++
      Jun 27 14:30:06 [2048] pcmk01 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=pcmk01-cr/crmd/49, version=0.9.1)
      Jun 27 14:30:06 [2053] pcmk01 crmd: warning: status_from_rc: Action 4 (mysql_data01_monitor_0) on pcmk01-cr failed (target: 7 vs. rc: 6): Error
      Jun 27 14:30:06 [2053] pcmk01 crmd: notice: abort_transition_graph: Transition aborted by mysql_data01_monitor_0 ‘create’ on pcmk01-cr: Event failed (magic=0:6;4:5:7:0e367aab-5184-4c54-87da-0562471c2635, cib=0.9.1, source=match_graph_event:381, 0)
      Jun 27 14:30:06 [2053] pcmk01 crmd: info: match_graph_event: Action mysql_data01_monitor_0 (4) confirmed on pcmk01-cr (rc=6)
      Jun 27 14:30:06 [2053] pcmk01 crmd: info: process_graph_event: Detected action (5.4) mysql_data01_monitor_0.5=not configured: failed
      Jun 27 14:30:06 [2053] pcmk01 crmd: warning: status_from_rc: Action 4 (mysql_data01_monitor_0) on pcmk01-cr failed (target: 7 vs. rc: 6): Error
      Jun 27 14:30:06 [2053] pcmk01 crmd: info: abort_transition_graph: Transition aborted by mysql_data01_monitor_0 ‘create’ on pcmk01-cr: Event failed (magic=0:6;4:5:7:0e367aab-5184-4c54-87da-0562471c2635, cib=0.9.1, source=match_graph_event:381, 0)
      Jun 27 14:30:06 [2053] pcmk01 crmd: info: match_graph_event: Action mysql_data01_monitor_0 (4) confirmed on pcmk01-cr (rc=6)
      Jun 27 14:30:06 [2053] pcmk01 crmd: info: process_graph_event: Detected action (5.4) mysql_data01_monitor_0.5=not configured: failed
      Jun 27 14:30:06 [2053] pcmk01 crmd: notice: te_rsc_command: Initiating action 3: probe_complete probe_complete-pcmk01-cr on pcmk01-cr (local) – no waiting
      Jun 27 14:30:06 [2053] pcmk01 crmd: info: te_rsc_command: Action 3 confirmed – no wait
      Jun 27 14:30:06 [2048] pcmk01 cib: info: cib_perform_op: Diff: — 0.9.1 2
      Jun 27 14:30:06 [2048] pcmk01 cib: info: cib_perform_op: Diff: +++ 0.9.2 (null)
      Jun 27 14:30:06 [2048] pcmk01 cib: info: cib_perform_op: + /cib: @num_updates=2
      Jun 27 14:30:06 [2048] pcmk01 cib: info: cib_perform_op: + /cib/status/node_state[@id=’2′]: @crm-debug-origin=do_update_resource
      Jun 27 14:30:06 [2048] pcmk01 cib: info: cib_perform_op: ++ /cib/status/node_state[@id=’2′]/lrm[@id=’2′]/lrm_resources:
      Jun 27 14:30:06 [2048] pcmk01 cib: info: cib_perform_op: ++ <lrm_rsc_op id="mysql_data01_last_failure_0" operation_key="mysql_data01_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.10" transition-key="6:5:7:0e367aab-5184-4c54-87da-0562471c2635" transition-magic="0:6;6:5:7:0e367aab-5184-4c54-87da-0562471c2635" on_node="pcmk02-cr" call-id="5" rc-code="6" op-status="0" interval="0" last-run="146
      Jun 27 14:30:06 [2048] pcmk01 cib: info: cib_perform_op: ++ <lrm_rsc_op id="mysql_data01_last_0" operation_key="mysql_data01_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.10" transition-key="6:5:7:0e367aab-5184-4c54-87da-0562471c2635" transition-magic="0:6;6:5:7:0e367aab-5184-4c54-87da-0562471c2635" on_node="pcmk02-cr" call-id="5" rc-code="6" op-status="0" interval="0" last-run="1467037806"
      Jun 27 14:30:06 [2048] pcmk01 cib: info: cib_perform_op: ++
      Jun 27 14:30:06 [2048] pcmk01 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=pcmk02-cr/crmd/16, version=0.9.2)
      Jun 27 14:30:06 [2053] pcmk01 crmd: warning: status_from_rc: Action 6 (mysql_data01_monitor_0) on pcmk02-cr failed (target: 7 vs. rc: 6): Error
      Jun 27 14:30:06 [2053] pcmk01 crmd: info: abort_transition_graph: Transition aborted by mysql_data01_monitor_0 ‘create’ on pcmk02-cr: Event failed (magic=0:6;6:5:7:0e367aab-5184-4c54-87da-0562471c2635, cib=0.9.2, source=match_graph_event:381, 0)
      Jun 27 14:30:06 [2053] pcmk01 crmd: info: match_graph_event: Action mysql_data01_monitor_0 (6) confirmed on pcmk02-cr (rc=6)
      Jun 27 14:30:06 [2053] pcmk01 crmd: info: process_graph_event: Detected action (5.6) mysql_data01_monitor_0.5=not configured: failed
      Jun 27 14:30:06 [2053] pcmk01 crmd: warning: status_from_rc: Action 6 (mysql_data01_monitor_0) on pcmk02-cr failed (target: 7 vs. rc: 6): Error
      Jun 27 14:30:06 [2053] pcmk01 crmd: info: abort_transition_graph: Transition aborted by mysql_data01_monitor_0 ‘create’ on pcmk02-cr: Event failed (magic=0:6;6:5:7:0e367aab-5184-4c54-87da-0562471c2635, cib=0.9.2, source=match_graph_event:381, 0)
      Jun 27 14:30:06 [2053] pcmk01 crmd: info: match_graph_event: Action mysql_data01_monitor_0 (6) confirmed on pcmk02-cr (rc=6)
      Jun 27 14:30:06 [2053] pcmk01 crmd: info: process_graph_event: Detected action (5.6) mysql_data01_monitor_0.5=not configured: failed
      Jun 27 14:30:06 [2053] pcmk01 crmd: notice: te_rsc_command: Initiating action 5: probe_complete probe_complete-pcmk02-cr on pcmk02-cr – no waiting
      Jun 27 14:30:06 [2053] pcmk01 crmd: info: te_rsc_command: Action 5 confirmed – no wait
      Jun 27 14:30:06 [2053] pcmk01 crmd: notice: run_graph: Transition 5 (Complete=5, Pending=0, Fired=0, Skipped=1, Incomplete=2, Source=/var/lib/pacemaker/pengine/pe-input-5.bz2): Stopped
      Jun 27 14:30:06 [2053] pcmk01 crmd: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
      Jun 27 14:30:06 [2052] pcmk01 pengine: notice: unpack_config: On loss of CCM Quorum: Ignore
      Jun 27 14:30:06 [2052] pcmk01 pengine: info: determine_online_status: Node pcmk01-cr is online
      Jun 27 14:30:06 [2052] pcmk01 pengine: info: determine_online_status: Node pcmk02-cr is online
      Jun 27 14:30:06 [2052] pcmk01 pengine: warning: unpack_rsc_op_failure: Processing failed op monitor for mysql_data01 on pcmk01-cr: not configured (6)
      Jun 27 14:30:06 [2052] pcmk01 pengine: error: unpack_rsc_op: Preventing mysql_data01 from re-starting anywhere: operation monitor failed ‘not configured’ (6)
      Jun 27 14:30:06 [2052] pcmk01 pengine: warning: unpack_rsc_op_failure: Processing failed op monitor for mysql_data01 on pcmk01-cr: not configured (6)
      Jun 27 14:30:06 [2052] pcmk01 pengine: error: unpack_rsc_op: Preventing mysql_data01 from re-starting anywhere: operation monitor failed ‘not configured’ (6)
      Jun 27 14:30:06 [2052] pcmk01 pengine: warning: unpack_rsc_op_failure: Processing failed op monitor for mysql_data01 on pcmk02-cr: not configured (6)
      Jun 27 14:30:06 [2052] pcmk01 pengine: error: unpack_rsc_op: Preventing mysql_data01 from re-starting anywhere: operation monitor failed ‘not configured’ (6)
      Jun 27 14:30:06 [2052] pcmk01 pengine: warning: unpack_rsc_op_failure: Processing failed op monitor for mysql_data01 on pcmk02-cr: not configured (6)
      Jun 27 14:30:06 [2052] pcmk01 pengine: error: unpack_rsc_op: Preventing mysql_data01 from re-starting anywhere: operation monitor failed ‘not configured’ (6)

    • Can you post the output of the following please:

      # ip -4 ad
      # cat /etc/hosts
      # cat /etc/drbd.d/mysql01.res
      # drbd-overview
      # cat /var/lib/pcsd/tokens
      # pcs resource show --full
      # pcs constraint --full
      # pcs property
      # pcs status
      # semanage permissive -ln
    • [[email protected] ~]# ip -4 ad
      1: lo:  mtu 65536 qdisc noqueue state UNKNOWN
          inet 127.0.0.1/8 scope host lo
             valid_lft forever preferred_lft forever
      2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
          inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic eth0
             valid_lft 83640sec preferred_lft 83640sec
      3: eth1:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
          inet 10.8.8.61/24 brd 10.8.8.255 scope global eth1
             valid_lft forever preferred_lft forever
      4: eth2:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
          inet 172.16.21.11/24 brd 172.16.21.255 scope global eth2
             valid_lft forever preferred_lft forever
      5: eth3:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
          inet 172.16.22.11/24 brd 172.16.22.255 scope global eth3
             valid_lft forever preferred_lft forever
      # cat /etc/hosts
      127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
      ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
      10.8.8.60 pcmkvip
       10.8.8.61 pcmk01
       10.8.8.62 pcmk02
       172.16.21.11 pcmk01-cr
       172.16.21.12 pcmk02-cr
       172.16.22.11 pcmk01-drbd
       172.16.22.12 pcmk02-drbd
      # cat /etc/drbd.d/mysql01.res
      resource mysql01 {
       protocol C;
       meta-disk internal;
       device /dev/drbd0;
       disk   /dev/VolGroup00/lv_drbd;
       handlers {
        split-brain "/usr/lib/drbd/notify-split-brain.sh root";
       }
       net {
        allow-two-primaries no;
        after-sb-0pri discard-zero-changes;
        after-sb-1pri discard-secondary;
        after-sb-2pri disconnect;
        rr-conflict disconnect;
       }
       disk {
        on-io-error detach;
       }
       syncer {
        verify-alg sha1;
       }
       on pcmk01 {
        address  172.16.22.11:7789;
       }
       on pcmk02 {
        address  172.16.22.12:7790; //because with vagrant I cannot use 7789 for both servers I forward 7790 to 7789
       }
      }
      # drbd-overview
       0:mysql01/0  Connected Primary/Secondary UpToDate/UpToDate /mnt ext4 302M 2.1M 296M 1%
      # cat /var/lib/pcsd/tokens
      {
        "format_version": 2,
        "data_version": 2,
        "tokens": {
          "pcmk01-cr": "828cbd4d-42a6-4ddc-aed1-2604735248ba",
          "pcmk02-cr": "e7fb15eb-82a3-47dd-bd53-3e1c71af9569"
        }
      # pcs resource show --full
       Master: MySQLClone01
        Meta Attrs: master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
        Resource: mysql_data01 (class=ocf provider=linbit type=drbd)
         Attributes: drbd_resource=mysql01
         Operations: start interval=0s timeout=240 (mysql_data01-start-interval-0s)
                     promote interval=0s timeout=90 (mysql_data01-promote-interval-0s)
                     demote interval=0s timeout=90 (mysql_data01-demote-interval-0s)
                     stop interval=0s timeout=100 (mysql_data01-stop-interval-0s)
                     monitor interval=30s (mysql_data01-monitor-interval-30s)
      # pcs constraint --full
      Location Constraints:
      Ordering Constraints:
      Colocation Constraints:
      # pcs property
      Cluster Properties:
       cluster-infrastructure: corosync
       cluster-name: mysql_cluster
       dc-version: 1.1.13-10.el7_2.2-44eb2dd
       have-watchdog: false
       no-quorum-policy: ignore
       stonith-enabled: false
      # pcs status
      Cluster name: mysql_cluster
      Last updated: Mon Jun 27 19:04:40 2016		Last change: Mon Jun 27 18:58:37 2016 by root via cibadmin on pcmk01-cr
      Stack: corosync
      Current DC: pcmk02-cr (version 1.1.13-10.el7_2.2-44eb2dd) - partition with quorum
      2 nodes and 2 resources configured
      
      Online: [ pcmk01-cr pcmk02-cr ]
      
      Full list of resources:
      
       Master/Slave Set: MySQLClone01 [mysql_data01]
           mysql_data01	(ocf::linbit:drbd):	FAILED pcmk02-cr (unmanaged)
           mysql_data01	(ocf::linbit:drbd):	FAILED pcmk01-cr (unmanaged)
      
      Failed Actions:
      * mysql_data01_stop_0 on pcmk02-cr 'not configured' (6): call=6, status=complete, exitreason='none',
          last-rc-change='Mon Jun 27 18:57:20 2016', queued=0ms, exec=57ms
      * mysql_data01_stop_0 on pcmk01-cr 'not configured' (6): call=6, status=complete, exitreason='none',
          last-rc-change='Mon Jun 27 18:57:20 2016', queued=0ms, exec=35ms
      
      
      PCSD Status:
        pcmk01-cr: Online
        pcmk02-cr: Online
      
      Daemon Status:
        corosync: active/disabled
        pacemaker: active/disabled
        pcsd: active/enabled
      # semanage permissive -ln
      drbd_t
      mon_procd_t
      mon_statd_t
      timemaster_t
      brltty_t
      cpuplug_t
      phc2sys_t
      systemd_networkd_t
      kmscon_t
      gear_t
      iotop_t
      osad_t
      ptp4l_t
    • My sincerest apologies, I was trying since yesterday but never it sent. I though that my responses were discarded. How could I delete the messages? Once again apologies

    • Ah, I see, not to worry. Due to huge amount of spam all comments are held for moderation. They only appear once manually approved. I’ll get rid of the duplicates.

    • I’m afraid I’m unable to replicate this, it works fine for me. Do you have firewall and SELinux configured?

      I’ve got the following packages:

      kernel-3.10.0-327.22.2.el7.x86_64
      pacemaker-1.1.13-10.el7_2.2.x86_64
      corosync-2.3.4-7.el7_2.3.x86_64
      pcs-0.9.143-15.el7.x86_64
      kmod-drbd84-8.4.7-1_1.el7.elrepo.x86_64
      drbd84-utils-8.9.5-1.el7.elrepo.x86_64

      Can you confirm they match with your system? Also, not sure it’s going to help much, but can you run the following:

      # pcs resource cleanup mysql_data01
    • I doesn’t use firewalld. I use iptables:

      # cat /etc/sysconfig/iptables
      *filter
      :INPUT ACCEPT [0:0]
      :FORWARD ACCEPT [0:0]
      :OUTPUT ACCEPT [47243:114710948]
      -A INPUT -p icmp -j ACCEPT
      -A INPUT -i lo -j ACCEPT
      -A INPUT -s 10.8.8.0/24 -p tcp -m tcp --dport 22 -m state --state NEW -j ACCEPT
      -A INPUT -s 172.16.21.0/24 -d 172.16.21.0/24 -m comment --comment Corosync -j ACCEPT
      -A INPUT -s 172.16.22.0/24 -d 172.16.22.0/24 -m comment --comment DRBD -j ACCEPT
      -A INPUT -s 10.8.8.0/24 -p tcp -m tcp --dport 3306 -m state --state NEW -j ACCEPT
      -A INPUT -p tcp -m tcp --dport 2224 -m state --state NEW -j ACCEPT
      -A INPUT -p tcp -m tcp --dport 3121 -m state --state NEW -j ACCEPT
      -A INPUT -p tcp -m tcp --dport 21064 -m state --state NEW -j ACCEPT
      -A INPUT -p tcp -m tcp --dport 7789 -m state --state NEW -j ACCEPT
      -A INPUT -p tcp -m tcp --dport 7790 -m state --state NEW -j ACCEPT
      -A INPUT -p udp -m multiport --dports 67,68,5405 -m state --state NEW -j ACCEPT
      -A INPUT -p udp -m multiport --dports 137,138,139,445 -j DROP
      -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
      -A INPUT -j LOG --log-prefix "iptables_input "
      -A INPUT -j DROP
      COMMIT

      The selinux status:

      # sestatus
      SELinux status:                 enabled
      SELinuxfs mount:                /sys/fs/selinux
      SELinux root directory:         /etc/selinux
      Loaded policy name:             targeted
      Current mode:                   enforcing
      Mode from config file:          enforcing
      Policy MLS status:              enabled
      Policy deny_unknown status:     allowed
      Max kernel policy version:      28

      The packages:

      kernel-3.10.0-327.13.1.el7.x86_64
      kernel-3.10.0-327.22.2.el7.x86_64
      pacemaker-1.1.13-10.el7_2.2.x86_64
      corosync-2.3.4-7.el7_2.3.x86_64
      pcs-0.9.143-15.el7.x86_64
      kmod-drbd84-8.4.7-1_1.el7.elrepo.x86_64
      drbd84-utils-8.9.5-1.el7.elrepo.x86_64
      # pcs resource cleanup mysql_data01
      Waiting for 2 replies from the CRMd.. OK
      Cleaning up mysql_data01 on pcmk01-cr, removing fail-count-mysql_data01
      Cleaning up mysql_data01 on pcmk02-cr, removing fail-count-mysql_data01
    • That all looks good. You have some duplicate rules on iptables, but it’s not important. Try setting SELinux to permissive and stopping iptables, then do a resource cleanup and let me know if it makes a difference. I cannot spot any misconfiguration really.

    • I have tried stopping iptables and setting selinux permissive both CLI and conf file and doesn’t work. I cannot see some conclusive errors to indicate where is the issue. Besides, when I turn off the VMs I cannot turn up the VMs and I have to deploy it from the beginning. Only I get this output when it tries turn up:

      [ 6433.804567 ] drbd mysql01: meta connection shut down by peer 

      Finally, I doesn’t configure the network interfaces enp because I create it as eth with vagrant.

    • It looks like an issue within your environment. If I were you, I would try to narrow the problem down. Remove Vagrant from the equation. Get two servers up and running, stick with predictable network interface device names, don’t change them and keep everything simple until you get Pacemaker working with DRBD.

    • Hi Tomas, I come back again. I have mounted the active/passive infraestructura in two physical servers discarding Vagrant. But I am getting the same error, drbd is connected in both servers but when I started to create the resources without cib, drbd resource called mysql_data01 continue unmanaged. In failed actioncs I see the same error

      * mysql_data01_stop_0 on pcmk01-cr 'not configured' (6): call=6, status=complete, exitreason='none',
          last-rc-change='Tue Jul 12 15:23:43 2016', queued=0ms, exec=27ms
      * mysql_data01_stop_0 on pcmk02-cr 'not configured' (6): call=6, status=complete, exitreason='none',
          last-rc-change='Tue Jul 12 15:23:43 2016', queued=0ms, exec=33ms


      Is there a drbd log where I can check that all is succesfully? On the same hand, is there a cluster log where I can check why this resource is unmanaged? At the moment the only that I have seen in path /var/log/cluster/corosync.log:

      Jul 12 16:16:55 [88488] pcmk01       crmd:     info: crm_timer_popped:	PEngine Recheck Timer (I_PE_CALC) just popped (900000ms)
      Jul 12 16:16:55 [88488] pcmk01       crmd:   notice: do_state_transition:	State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
      Jul 12 16:16:55 [88488] pcmk01       crmd:     info: do_state_transition:	Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
      Jul 12 16:16:55 [88487] pcmk01    pengine:     info: process_pe_message:	Input has not changed since last time, not saving to disk
      Jul 12 16:16:55 [88487] pcmk01    pengine:   notice: unpack_config:	On loss of CCM Quorum: Ignore
      Jul 12 16:16:55 [88487] pcmk01    pengine:     info: determine_online_status:	Node pcmk01-cr is online
      Jul 12 16:16:55 [88487] pcmk01    pengine:     info: determine_online_status:	Node pcmk02-cr is online
      Jul 12 16:16:55 [88487] pcmk01    pengine:  warning: unpack_rsc_op_failure:	Processing failed op stop for mysql_data01:0 on pcmk01-cr: not configured (6)
      Jul 12 16:16:55 [88487] pcmk01    pengine:    error: unpack_rsc_op:	Preventing MySQLClone01 from re-starting anywhere: operation stop failed 'not configured' (6)
      Jul 12 16:16:55 [88487] pcmk01    pengine:  warning: unpack_rsc_op_failure:	Processing failed op stop for mysql_data01:0 on pcmk01-cr: not configured (6)
      Jul 12 16:16:55 [88487] pcmk01    pengine:    error: unpack_rsc_op:	Preventing MySQLClone01 from re-starting anywhere: operation stop failed 'not configured' (6)
      Jul 12 16:16:55 [88487] pcmk01    pengine:     info: native_add_running:	resource mysql_data01:0 isnt managed
      Jul 12 16:16:55 [88487] pcmk01    pengine:  warning: unpack_rsc_op_failure:	Processing failed op stop for mysql_data01:1 on pcmk02-cr: not configured (6)
      Jul 12 16:16:55 [88487] pcmk01    pengine:    error: unpack_rsc_op:	Preventing MySQLClone01 from re-starting anywhere: operation stop failed 'not configured' (6)
      Jul 12 16:16:55 [88487] pcmk01    pengine:  warning: unpack_rsc_op_failure:	Processing failed op stop for mysql_data01:1 on pcmk02-cr: not configured (6)
      Jul 12 16:16:55 [88487] pcmk01    pengine:    error: unpack_rsc_op:	Preventing MySQLClone01 from re-starting anywhere: operation stop failed 'not configured' (6)
      Jul 12 16:16:55 [88487] pcmk01    pengine:     info: native_add_running:	resource mysql_data01:1 isnt managed
      Jul 12 16:16:55 [88487] pcmk01    pengine:     info: clone_print:	 Master/Slave Set: MySQLClone01 [mysql_data01]
      Jul 12 16:16:55 [88487] pcmk01    pengine:     info: native_print:	     mysql_data01	(ocf::linbit:drbd):	FAILED pcmk01-cr (unmanaged)
      Jul 12 16:16:55 [88487] pcmk01    pengine:     info: native_print:	     mysql_data01	(ocf::linbit:drbd):	FAILED pcmk02-cr (unmanaged)
      Jul 12 16:16:55 [88487] pcmk01    pengine:     info: get_failcount_full:	mysql_data01:0 has failed INFINITY times on pcmk01-cr
      Jul 12 16:16:55 [88487] pcmk01    pengine:  warning: common_apply_stickiness:	Forcing MySQLClone01 away from pcmk01-cr after 1000000 failures (max=1000000)
      Jul 12 16:16:55 [88487] pcmk01    pengine:     info: get_failcount_full:	mysql_data01:1 has failed INFINITY times on pcmk01-cr
      Jul 12 16:16:55 [88487] pcmk01    pengine:  warning: common_apply_stickiness:	Forcing MySQLClone01 away from pcmk01-cr after 1000000 failures (max=1000000)
      Jul 12 16:16:55 [88487] pcmk01    pengine:     info: get_failcount_full:	mysql_data01:0 has failed INFINITY times on pcmk02-cr
      Jul 12 16:16:55 [88487] pcmk01    pengine:  warning: common_apply_stickiness:	Forcing MySQLClone01 away from pcmk02-cr after 1000000 failures (max=1000000)
      Jul 12 16:16:55 [88487] pcmk01    pengine:     info: get_failcount_full:	mysql_data01:1 has failed INFINITY times on pcmk02-cr
      Jul 12 16:16:55 [88487] pcmk01    pengine:  warning: common_apply_stickiness:	Forcing MySQLClone01 away from pcmk02-cr after 1000000 failures (max=1000000)
      Jul 12 16:16:55 [88487] pcmk01    pengine:     info: native_color:	Unmanaged resource mysql_data01:0 allocated to pcmk01-cr: failed
      Jul 12 16:16:55 [88487] pcmk01    pengine:     info: native_color:	Unmanaged resource mysql_data01:1 allocated to pcmk02-cr: failed
      Jul 12 16:16:55 [88487] pcmk01    pengine:     info: master_color:	MySQLClone01: Promoted 0 instances of a possible 1 to master
      Jul 12 16:16:55 [88487] pcmk01    pengine:     info: LogActions:	Leave   mysql_data01:0	(Slave unmanaged)
      Jul 12 16:16:55 [88487] pcmk01    pengine:     info: LogActions:	Leave   mysql_data01:1	(Slave unmanaged)
      Jul 12 16:16:55 [88487] pcmk01    pengine:   notice: process_pe_message:	Calculated Transition 13: /var/lib/pacemaker/pengine/pe-input-494.bz2
      Jul 12 16:16:55 [88488] pcmk01       crmd:     info: do_state_transition:	State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
      Jul 12 16:16:55 [88488] pcmk01       crmd:     info: do_te_invoke:	Processing graph 13 (ref=pe_calc-dc-1468340215-35) derived from /var/lib/pacemaker/pengine/pe-input-494.bz2
      Jul 12 16:16:55 [88488] pcmk01       crmd:   notice: run_graph:	Transition 13 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-494.bz2): Complete
      Jul 12 16:16:55 [88488] pcmk01       crmd:     info: do_log:	FSA: Input I_TE_SUCCESS from notify_crmd() received in state S_TRANSITION_ENGINE
      Jul 12 16:16:55 [88488] pcmk01       crmd:   notice: do_state_transition:	State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]


      It seems corosync problem but when I check the cluster status it is ok in both servers:

      Online: [ pcmk01-cr pcmk02-cr ]
      PCSD Status:
        pcmk01-cr: Online
        pcmk02-cr: Online
      
      Daemon Status:
        corosync: active/disabled
        pacemaker: active/disabled
        pcsd: active/enabled


      I am sure that I am following all steps but it is very strange.

  9. I wonder that how to configure active-active mode. Please let me know that way.

    I set
    ———————————————————————————

    resource mysql01 {
     protocol C;
     meta-disk internal;
     device /dev/drbd0;
     disk   /dev/sdb;
     handlers {
      split-brain "/usr/lib/drbd/notify-split-brain.sh root";
     }
     net {
      allow-two-primaries yes;
      after-sb-0pri discard-zero-changes;
      after-sb-1pri discard-secondary;
      after-sb-2pri disconnect;
      rr-conflict disconnect;
     }
     disk {
      on-io-error detach;
     }
     syncer {
      verify-alg sha1;
     }
     on pcmk01 {
      address  172.16.22.11:7789;
     }
     on pcmk02 {
      address  172.16.22.12:7790;
     }
    }

    ———————————————————–

    pcs -f clust_cfg resource master MySQLClone01 mysql_data01 \
    > master-max=2 master-node-max=2 \
    > clone-max=2 clone-node-max=2 \
    > notify=true

    ——————————————————–

    but pcs status is

    Cluster name: mysql_cluster
    Last updated: Mon Jul  4 16:28:30 2016          Last change: Mon Jul  4 16:28:26 2016 by root via cibadmin on pcmk01-cr
    Stack: corosync
    Current DC: pcmk02-cr (version 1.1.13-10.el7_2.2-44eb2dd) - partition with quorum
    2 nodes and 5 resources configured
    
    Online: [ pcmk01-cr pcmk02-cr ]
    
    Full list of resources:
    
     Master/Slave Set: MySQLClone01 [mysql_data01]
         mysql_data01       (ocf::linbit:drbd):     FAILED pcmk02-cr (unmanaged)
         mysql_data01       (ocf::linbit:drbd):     FAILED pcmk01-cr (unmanaged)
     mysql_fs01     (ocf::heartbeat:Filesystem):    Stopped
     mysql_service01        (ocf::heartbeat:mysql): Stopped
     mysql_VIP01    (ocf::heartbeat:IPaddr2):       Stopped
    
    Failed Actions:
    * mysql_data01_stop_0 on pcmk02-cr 'not configured' (6): call=20, status=complete, exitreason='none',
        last-rc-change='Mon Jul  4 16:28:27 2016', queued=0ms, exec=32ms
    * mysql_data01_stop_0 on pcmk01-cr 'not configured' (6): call=20, status=complete, exitreason='none',
        last-rc-change='Mon Jul  4 16:28:27 2016', queued=0ms, exec=25ms
    
    
    PCSD Status:
      pcmk01-cr: Online
      pcmk02-cr: Online
    
    Daemon Status:
      corosync: active/enabled
      pacemaker: active/enabled
      pcsd: active/enabled
    • Did you ever receive an answer for these?
      “mysql_data01 (ocf::linbit:drbd): FAILED pcmk02-cr (unmanaged)
      mysql_data01 (ocf::linbit:drbd): FAILED pcmk01-cr (unmanaged)”

  10. Thank you for reply.
    But i want to use just mariadb duplication instead of galera cluster.
    Is it possible active-active mode in this post?

  11. I just love this article, thanks for the time in putting this together. I would just like to know, if it would be hard to add Apache into this active / passive setup? I’m trying to make a active / passive LAMP server that’s all

    Again thanks for your time.

    • I tend to keep databases isolated (back-end) and use a separate cluster for Apache (front-end) with front-end proxies on top of it really. Check this post for how to add Apache as a cluster service. Hope this helps.

    • Thanks for your last reply.

      I’ve got two servers running the system from this post without any problems that I know of.
      I just have one of two questions if you can help, I have two physical servers would I need to setup this STONITH? If one of my servers goes offline and comes back on, how to I re-sync the SQL and re-add this server back in the cluster as if I do a pc status at commandline, it says no in cluster?

      Thanks again

    • It doesn’t really matter that much what you use, whether it be physical servers, blades, virtual machines etc. The question really is how do you handle split-brain scenarions? You don’t need STONITH if you have some well-tested and proven mechanism that takes care of it.

      As for your second question, I don’t know your setup, so cannot advise.

  12. I want to do a Mysql High Availability Cluster But I am not able to do it. When I created Mysql resource

    pcs resource create Mysql ocf:heartbeat:mysql params binary=”/usr/bin/mysql” datadir=”/var/lib/mysql” config=”/etc/my.cnf” log=”/var/log/mysqld.log” pid=”/var/run/mysqld/mysqld.pid” socket=”/var/lib/mysql/mysql.sock” op start interval=0s timeout=20s op stop interval=0s timeout=20s op monitor interval=20s

    pcs status show that resource Mysql:started [node1 node2] but after some time (5 seconds) It is stopped and also Mysql service or status also stopped on both the nodes

    Thank you

  13. Log:

    2016-07-05T06:49:21.166141Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use –explicit_defaults_for_timestamp server option (see documentation for more details).
    2016-07-05T06:49:22.384391Z 0 [Warning] InnoDB: New log files created, LSN=45790
    2016-07-05T06:49:22.540803Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
    2016-07-05T06:49:22.599347Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 9b0bb85c-427c-11e6-b344-0800274eaf5b.
    2016-07-05T06:49:22.609981Z 0 [Warning] Gtid table is not ready to be used. Table ‘mysql.gtid_executed’ cannot be opened.
    2016-07-05T06:49:22.611483Z 1 [Note] A temporary password is generated for [email protected]: lvwXr+i>_3/!
    2016-07-05T06:49:27.861283Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use –explicit_defaults_for_timestamp server option (see documentation for more details).
    2016-07-05T06:49:27.862086Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.13) starting as process 2939 …
    2016-07-05T06:49:27.869138Z 0 [Note] InnoDB: PUNCH HOLE support available
    2016-07-05T06:49:27.869186Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
    2016-07-05T06:49:27.869196Z 0 [Note] InnoDB: Uses event mutexes

  14. pcs status
    Cluster name: hacluster1
    Last updated: Thu Jul 21 05:50:40 2016 Last change: Thu Jul 21 05:50:13 2016 by root via cibadmin on node2
    Stack: corosync
    Current DC: node1 (version 1.1.13-10.el7_2.2-44eb2dd) – partition with quorum
    2 nodes and 2 resources configured

    Online: [ node1 node2 ]

    Full list of resources:

    VIRTUAL_IP_ (ocf::heartbeat:IPaddr2): Started node2
    Mysql (ocf::heartbeat:mysql): Stopped

    Failed Actions:
    * Mysql_start_0 on node1 ‘unknown error’ (1): call=33, status=complete, exitreason=’MySQL server failed to start (pid=10542) (rc=7), please check your installation’,
    last-rc-change=’Thu Jul 21 05:50:26 2016′, queued=0ms, exec=550ms
    * Mysql_start_0 on node2 ‘unknown error’ (1): call=29, status=complete, exitreason=’MySQL server failed to start (pid=11933) (rc=7), please check your installation’,
    last-rc-change=’Thu Jul 21 05:50:27 2016′, queued=0ms, exec=938ms

    PCSD Status:
    node1: Online
    node2: Online

    Daemon Status:
    corosync: active/enabled
    pacemaker: active/enabled
    pcsd: active/enabled

  15. Thank you your guide was helping me a lot , i had a lot issues doing this setup with others tuto’s.
    Your guide was the good one , all works perfect , with MAC/Parallel.
    Question How this setup is good for prod env??
    I had setup heartbeat+drbd+mysql on centos 5.5 in the past and still working in prod very good.
    but this setup can go to prod env?

    • This particular setup does not have fencing (neither resource level nor node level), therefore I would not use it in production.

  16. when doing mysql ha cluster I done replication after making mysql resource it works well when there is failover but checking mysql status (service mysqld status) it shows inactive(dead) on both nodes why……???????

  17. please tell
    How to create a Mysql resource using pacemaker and corosync on centos7 i am succesfully created two nodes and a resource for virtual ip it works well now i want to create a Mysql resource but not able to create as i mention it stops the mysql service but on the node Mysql resource started i am able to login in mysql on that node but the service is stopped…..?????

    Thanks

    • The article explains how to create what you asked for. If you run into problems, check logs to see what’s failing, and troubleshoot accordingly.

  18. Hey Tomas,
    Successfully created Postgresql cluster following your guidelines, awesome write-up, thank you very much!

  19. Hello,

    Does anyone know what is causing this failure below?

    “Master/Slave Set: MySQLClone01 [mysql_data01]
    mysql_data01 (ocf::linbit:drbd): FAILED pcmk02-cr (unmanaged)
    mysql_data01 (ocf::linbit:drbd): FAILED pcmk01-cr (unmanaged)”

  20. Hi.

    Do you have any idea how to resolve this?

    [[email protected] ~]# yum install -y kmod-drbd84 drbd84-utils
    Loaded plugins: fastestmirror
    Loading mirror speeds from cached hostfile
    * base: centos.serverspace.co.uk
    * elrepo: mirrors.coreix.net
    * extras: mirrors.coreix.net
    * updates: mirror.sov.uk.goscomb.net
    Package drbd84-utils-8.9.6-1.el7.elrepo.x86_64 already installed and latest version
    Resolving Dependencies
    –> Running transaction check
    —> Package kmod-drbd84.x86_64 0:8.4.8-1_2.el7.elrepo will be installed
    –> Processing Dependency: kernel(crypto_shash_setkey) = 0xad34f5e2 for package: kmod-drbd84-8.4.8-1_2.el7.elrepo.x86_64
    –> Processing Dependency: kernel(crypto_shash_digest) = 0xa2f07728 for package: kmod-drbd84-8.4.8-1_2.el7.elrepo.x86_64
    –> Processing Dependency: kernel(crypto_destroy_tfm) = 0x8f944a92 for package: kmod-drbd84-8.4.8-1_2.el7.elrepo.x86_64
    –> Processing Dependency: kernel(crypto_alloc_shash) = 0x848fdfcb for package: kmod-drbd84-8.4.8-1_2.el7.elrepo.x86_64
    –> Processing Dependency: kernel(crypto_alloc_ahash) = 0x035deac4 for package: kmod-drbd84-8.4.8-1_2.el7.elrepo.x86_64
    –> Processing Dependency: kernel(crypto_ahash_finup) = 0xc30b20a1 for package: kmod-drbd84-8.4.8-1_2.el7.elrepo.x86_64
    –> Processing Dependency: kernel(crypto_ahash_final) = 0xb2c3ef50 for package: kmod-drbd84-8.4.8-1_2.el7.elrepo.x86_64
    –> Processing Dependency: kernel >= 3.10.0-514.el7 for package: kmod-drbd84-8.4.8-1_2.el7.elrepo.x86_64
    –> Finished Dependency Resolution
    Error: Package: kmod-drbd84-8.4.8-1_2.el7.elrepo.x86_64 (elrepo)
    Requires: kernel(crypto_ahash_finup) = 0xc30b20a1
    Installed: kernel-3.10.0-327.el7.x86_64 (@anaconda)
    kernel(crypto_ahash_finup) = 0xecb40033
    Installed: kernel-3.10.0-327.36.3.el7.x86_64 (@updates)
    kernel(crypto_ahash_finup) = 0xecb40033
    Installed: kernel-debug-3.10.0-327.36.3.el7.x86_64 (@updates)
    kernel(crypto_ahash_finup) = 0x7e16c908
    Available: kernel-3.10.0-327.3.1.el7.x86_64 (updates)
    kernel(crypto_ahash_finup) = 0xecb40033
    Available: kernel-3.10.0-327.4.4.el7.x86_64 (updates)
    kernel(crypto_ahash_finup) = 0xecb40033
    Available: kernel-3.10.0-327.4.5.el7.x86_64 (updates)
    kernel(crypto_ahash_finup) = 0xecb40033
    Available: kernel-3.10.0-327.10.1.el7.x86_64 (updates)
    kernel(crypto_ahash_finup) = 0xecb40033
    Available: kernel-3.10.0-327.13.1.el7.x86_64 (updates)
    kernel(crypto_ahash_finup) = 0xecb40033
    Available: kernel-3.10.0-327.18.2.el7.x86_64 (updates)
    kernel(crypto_ahash_finup) = 0xecb40033
    Available: kernel-3.10.0-327.22.2.el7.x86_64 (updates)
    kernel(crypto_ahash_finup) = 0xecb40033
    Available: kernel-3.10.0-327.28.2.el7.x86_64 (updates)
    kernel(crypto_ahash_finup) = 0xecb40033
    Available: kernel-3.10.0-327.28.3.el7.x86_64 (updates)
    kernel(crypto_ahash_finup) = 0xecb40033
    Available: kernel-3.10.0-327.36.1.el7.x86_64 (updates)
    kernel(crypto_ahash_finup) = 0xecb40033
    Available: kernel-3.10.0-327.36.2.el7.x86_64 (updates)
    kernel(crypto_ahash_finup) = 0xecb40033
    Available: kernel-debug-3.10.0-327.el7.x86_64 (base)
    kernel(crypto_ahash_finup) = 0x7e16c908
    Available: kernel-debug-3.10.0-327.3.1.el7.x86_64 (updates)
    kernel(crypto_ahash_finup) = 0x7e16c908
    Available: kernel-debug-3.10.0-327.4.4.el7.x86_64 (updates)
    kernel(crypto_ahash_finup) = 0x7e16c908
    Available: kernel-debug-3.10.0-327.4.5.el7.x86_64 (updates)
    kernel(crypto_ahash_finup) = 0x7e16c908
    Available: kernel-debug-3.10.0-327.10.1.el7.x86_64 (updates)
    kernel(crypto_ahash_finup) = 0x7e16c908
    Available: kernel-debug-3.10.0-327.13.1.el7.x86_64 (updates)
    kernel(crypto_ahash_finup) = 0x7e16c908
    Available: kernel-debug-3.10.0-327.18.2.el7.x86_64 (updates)
    kernel(crypto_ahash_finup) = 0x7e16c908
    Available: kernel-debug-3.10.0-327.22.2.el7.x86_64 (updates)
    kernel(crypto_ahash_finup) = 0x7e16c908
    Available: kernel-debug-3.10.0-327.28.2.el7.x86_64 (updates)
    kernel(crypto_ahash_finup) = 0x7e16c908
    Available: kernel-debug-3.10.0-327.28.3.el7.x86_64 (updates)
    kernel(crypto_ahash_finup) = 0x7e16c908
    Available: kernel-debug-3.10.0-327.36.1.el7.x86_64 (updates)
    kernel(crypto_ahash_finup) = 0x7e16c908
    Available: kernel-debug-3.10.0-327.36.2.el7.x86_64 (updates)
    kernel(crypto_ahash_finup) = 0x7e16c908
    Error: Package: kmod-drbd84-8.4.8-1_2.el7.elrepo.x86_64 (elrepo)
    Requires: kernel(crypto_shash_setkey) = 0xad34f5e2
    Installed: kernel-3.10.0-327.el7.x86_64 (@anaconda)
    kernel(crypto_shash_setkey) = 0xeedcd625
    Installed: kernel-3.10.0-327.36.3.el7.x86_64 (@updates)
    kernel(crypto_shash_setkey) = 0xeedcd625
    Installed: kernel-debug-3.10.0-327.36.3.el7.x86_64 (@updates)
    kernel(crypto_shash_setkey) = 0x3a236648
    Available: kernel-3.10.0-327.3.1.el7.x86_64 (updates)
    kernel(crypto_shash_setkey) = 0xeedcd625
    Available: kernel-3.10.0-327.4.4.el7.x86_64 (updates)
    kernel(crypto_shash_setkey) = 0xeedcd625
    Available: kernel-3.10.0-327.4.5.el7.x86_64 (updates)
    kernel(crypto_shash_setkey) = 0xeedcd625
    Available: kernel-3.10.0-327.10.1.el7.x86_64 (updates)
    kernel(crypto_shash_setkey) = 0xeedcd625
    Available: kernel-3.10.0-327.13.1.el7.x86_64 (updates)
    kernel(crypto_shash_setkey) = 0xeedcd625
    Available: kernel-3.10.0-327.18.2.el7.x86_64 (updates)
    kernel(crypto_shash_setkey) = 0xeedcd625
    Available: kernel-3.10.0-327.22.2.el7.x86_64 (updates)
    kernel(crypto_shash_setkey) = 0xeedcd625
    Available: kernel-3.10.0-327.28.2.el7.x86_64 (updates)
    kernel(crypto_shash_setkey) = 0xeedcd625
    Available: kernel-3.10.0-327.28.3.el7.x86_64 (updates)
    kernel(crypto_shash_setkey) = 0xeedcd625
    Available: kernel-3.10.0-327.36.1.el7.x86_64 (updates)
    kernel(crypto_shash_setkey) = 0xeedcd625
    Available: kernel-3.10.0-327.36.2.el7.x86_64 (updates)
    kernel(crypto_shash_setkey) = 0xeedcd625
    Available: kernel-debug-3.10.0-327.el7.x86_64 (base)
    kernel(crypto_shash_setkey) = 0x3a236648
    Available: kernel-debug-3.10.0-327.3.1.el7.x86_64 (updates)
    kernel(crypto_shash_setkey) = 0x3a236648
    Available: kernel-debug-3.10.0-327.4.4.el7.x86_64 (updates)
    kernel(crypto_shash_setkey) = 0x3a236648
    Available: kernel-debug-3.10.0-327.4.5.el7.x86_64 (updates)
    kernel(crypto_shash_setkey) = 0x3a236648
    Available: kernel-debug-3.10.0-327.10.1.el7.x86_64 (updates)
    kernel(crypto_shash_setkey) = 0x3a236648
    Available: kernel-debug-3.10.0-327.13.1.el7.x86_64 (updates)
    kernel(crypto_shash_setkey) = 0x3a236648
    Available: kernel-debug-3.10.0-327.18.2.el7.x86_64 (updates)
    kernel(crypto_shash_setkey) = 0x3a236648
    Available: kernel-debug-3.10.0-327.22.2.el7.x86_64 (updates)
    kernel(crypto_shash_setkey) = 0x3a236648
    Available: kernel-debug-3.10.0-327.28.2.el7.x86_64 (updates)
    kernel(crypto_shash_setkey) = 0x3a236648
    Available: kernel-debug-3.10.0-327.28.3.el7.x86_64 (updates)
    kernel(crypto_shash_setkey) = 0x3a236648
    Available: kernel-debug-3.10.0-327.36.1.el7.x86_64 (updates)
    kernel(crypto_shash_setkey) = 0x3a236648
    Available: kernel-debug-3.10.0-327.36.2.el7.x86_64 (updates)
    kernel(crypto_shash_setkey) = 0x3a236648
    Error: Package: kmod-drbd84-8.4.8-1_2.el7.elrepo.x86_64 (elrepo)
    Requires: kernel(crypto_shash_digest) = 0xa2f07728
    Installed: kernel-3.10.0-327.el7.x86_64 (@anaconda)
    kernel(crypto_shash_digest) = 0x8033678b
    Installed: kernel-3.10.0-327.36.3.el7.x86_64 (@updates)
    kernel(crypto_shash_digest) = 0x8033678b
    Installed: kernel-debug-3.10.0-327.36.3.el7.x86_64 (@updates)
    kernel(crypto_shash_digest) = 0xf53f7270
    Available: kernel-3.10.0-327.3.1.el7.x86_64 (updates)
    kernel(crypto_shash_digest) = 0x8033678b
    Available: kernel-3.10.0-327.4.4.el7.x86_64 (updates)
    kernel(crypto_shash_digest) = 0x8033678b
    Available: kernel-3.10.0-327.4.5.el7.x86_64 (updates)
    kernel(crypto_shash_digest) = 0x8033678b
    Available: kernel-3.10.0-327.10.1.el7.x86_64 (updates)
    kernel(crypto_shash_digest) = 0x8033678b
    Available: kernel-3.10.0-327.13.1.el7.x86_64 (updates)
    kernel(crypto_shash_digest) = 0x8033678b
    Available: kernel-3.10.0-327.18.2.el7.x86_64 (updates)
    kernel(crypto_shash_digest) = 0x8033678b
    Available: kernel-3.10.0-327.22.2.el7.x86_64 (updates)
    kernel(crypto_shash_digest) = 0x8033678b
    Available: kernel-3.10.0-327.28.2.el7.x86_64 (updates)
    kernel(crypto_shash_digest) = 0x8033678b
    Available: kernel-3.10.0-327.28.3.el7.x86_64 (updates)
    kernel(crypto_shash_digest) = 0x8033678b
    Available: kernel-3.10.0-327.36.1.el7.x86_64 (updates)
    kernel(crypto_shash_digest) = 0x8033678b
    Available: kernel-3.10.0-327.36.2.el7.x86_64 (updates)
    kernel(crypto_shash_digest) = 0x8033678b
    Available: kernel-debug-3.10.0-327.el7.x86_64 (base)
    kernel(crypto_shash_digest) = 0xf53f7270
    Available: kernel-debug-3.10.0-327.3.1.el7.x86_64 (updates)
    kernel(crypto_shash_digest) = 0xf53f7270
    Available: kernel-debug-3.10.0-327.4.4.el7.x86_64 (updates)
    kernel(crypto_shash_digest) = 0xf53f7270
    Available: kernel-debug-3.10.0-327.4.5.el7.x86_64 (updates)
    kernel(crypto_shash_digest) = 0xf53f7270
    Available: kernel-debug-3.10.0-327.10.1.el7.x86_64 (updates)
    kernel(crypto_shash_digest) = 0xf53f7270
    Available: kernel-debug-3.10.0-327.13.1.el7.x86_64 (updates)
    kernel(crypto_shash_digest) = 0xf53f7270
    Available: kernel-debug-3.10.0-327.18.2.el7.x86_64 (updates)
    kernel(crypto_shash_digest) = 0xf53f7270
    Available: kernel-debug-3.10.0-327.22.2.el7.x86_64 (updates)
    kernel(crypto_shash_digest) = 0xf53f7270
    Available: kernel-debug-3.10.0-327.28.2.el7.x86_64 (updates)
    kernel(crypto_shash_digest) = 0xf53f7270
    Available: kernel-debug-3.10.0-327.28.3.el7.x86_64 (updates)
    kernel(crypto_shash_digest) = 0xf53f7270
    Available: kernel-debug-3.10.0-327.36.1.el7.x86_64 (updates)
    kernel(crypto_shash_digest) = 0xf53f7270
    Available: kernel-debug-3.10.0-327.36.2.el7.x86_64 (updates)
    kernel(crypto_shash_digest) = 0xf53f7270
    Error: Package: kmod-drbd84-8.4.8-1_2.el7.elrepo.x86_64 (elrepo)
    Requires: kernel(crypto_ahash_final) = 0xb2c3ef50
    Installed: kernel-3.10.0-327.el7.x86_64 (@anaconda)
    kernel(crypto_ahash_final) = 0x1995b8e7
    Installed: kernel-3.10.0-327.36.3.el7.x86_64 (@updates)
    kernel(crypto_ahash_final) = 0x1995b8e7
    Installed: kernel-debug-3.10.0-327.36.3.el7.x86_64 (@updates)
    kernel(crypto_ahash_final) = 0x14cffb96
    Available: kernel-3.10.0-327.3.1.el7.x86_64 (updates)
    kernel(crypto_ahash_final) = 0x1995b8e7
    Available: kernel-3.10.0-327.4.4.el7.x86_64 (updates)
    kernel(crypto_ahash_final) = 0x1995b8e7
    Available: kernel-3.10.0-327.4.5.el7.x86_64 (updates)
    kernel(crypto_ahash_final) = 0x1995b8e7
    Available: kernel-3.10.0-327.10.1.el7.x86_64 (updates)
    kernel(crypto_ahash_final) = 0x1995b8e7
    Available: kernel-3.10.0-327.13.1.el7.x86_64 (updates)
    kernel(crypto_ahash_final) = 0x1995b8e7
    Available: kernel-3.10.0-327.18.2.el7.x86_64 (updates)
    kernel(crypto_ahash_final) = 0x1995b8e7
    Available: kernel-3.10.0-327.22.2.el7.x86_64 (updates)
    kernel(crypto_ahash_final) = 0x1995b8e7
    Available: kernel-3.10.0-327.28.2.el7.x86_64 (updates)
    kernel(crypto_ahash_final) = 0x1995b8e7
    Available: kernel-3.10.0-327.28.3.el7.x86_64 (updates)
    kernel(crypto_ahash_final) = 0x1995b8e7
    Available: kernel-3.10.0-327.36.1.el7.x86_64 (updates)
    kernel(crypto_ahash_final) = 0x1995b8e7
    Available: kernel-3.10.0-327.36.2.el7.x86_64 (updates)
    kernel(crypto_ahash_final) = 0x1995b8e7
    Available: kernel-debug-3.10.0-327.el7.x86_64 (base)
    kernel(crypto_ahash_final) = 0x14cffb96
    Available: kernel-debug-3.10.0-327.3.1.el7.x86_64 (updates)
    kernel(crypto_ahash_final) = 0x14cffb96
    Available: kernel-debug-3.10.0-327.4.4.el7.x86_64 (updates)
    kernel(crypto_ahash_final) = 0x14cffb96
    Available: kernel-debug-3.10.0-327.4.5.el7.x86_64 (updates)
    kernel(crypto_ahash_final) = 0x14cffb96
    Available: kernel-debug-3.10.0-327.10.1.el7.x86_64 (updates)
    kernel(crypto_ahash_final) = 0x14cffb96
    Available: kernel-debug-3.10.0-327.13.1.el7.x86_64 (updates)
    kernel(crypto_ahash_final) = 0x14cffb96
    Available: kernel-debug-3.10.0-327.18.2.el7.x86_64 (updates)
    kernel(crypto_ahash_final) = 0x14cffb96
    Available: kernel-debug-3.10.0-327.22.2.el7.x86_64 (updates)
    kernel(crypto_ahash_final) = 0x14cffb96
    Available: kernel-debug-3.10.0-327.28.2.el7.x86_64 (updates)
    kernel(crypto_ahash_final) = 0x14cffb96
    Available: kernel-debug-3.10.0-327.28.3.el7.x86_64 (updates)
    kernel(crypto_ahash_final) = 0x14cffb96
    Available: kernel-debug-3.10.0-327.36.1.el7.x86_64 (updates)
    kernel(crypto_ahash_final) = 0x14cffb96
    Available: kernel-debug-3.10.0-327.36.2.el7.x86_64 (updates)
    kernel(crypto_ahash_final) = 0x14cffb96
    Error: Package: kmod-drbd84-8.4.8-1_2.el7.elrepo.x86_64 (elrepo)
    Requires: kernel(crypto_destroy_tfm) = 0x8f944a92
    Installed: kernel-3.10.0-327.el7.x86_64 (@anaconda)
    kernel(crypto_destroy_tfm) = 0x5dbe8093
    Installed: kernel-3.10.0-327.36.3.el7.x86_64 (@updates)
    kernel(crypto_destroy_tfm) = 0x5dbe8093
    Installed: kernel-debug-3.10.0-327.36.3.el7.x86_64 (@updates)
    kernel(crypto_destroy_tfm) = 0x94d1a071
    Available: kernel-3.10.0-327.3.1.el7.x86_64 (updates)
    kernel(crypto_destroy_tfm) = 0x5dbe8093
    Available: kernel-3.10.0-327.4.4.el7.x86_64 (updates)
    kernel(crypto_destroy_tfm) = 0x5dbe8093
    Available: kernel-3.10.0-327.4.5.el7.x86_64 (updates)
    kernel(crypto_destroy_tfm) = 0x5dbe8093
    Available: kernel-3.10.0-327.10.1.el7.x86_64 (updates)
    kernel(crypto_destroy_tfm) = 0x5dbe8093
    Available: kernel-3.10.0-327.13.1.el7.x86_64 (updates)
    kernel(crypto_destroy_tfm) = 0x5dbe8093
    Available: kernel-3.10.0-327.18.2.el7.x86_64 (updates)
    kernel(crypto_destroy_tfm) = 0x5dbe8093
    Available: kernel-3.10.0-327.22.2.el7.x86_64 (updates)
    kernel(crypto_destroy_tfm) = 0x5dbe8093
    Available: kernel-3.10.0-327.28.2.el7.x86_64 (updates)
    kernel(crypto_destroy_tfm) = 0x5dbe8093
    Available: kernel-3.10.0-327.28.3.el7.x86_64 (updates)
    kernel(crypto_destroy_tfm) = 0x5dbe8093
    Available: kernel-3.10.0-327.36.1.el7.x86_64 (updates)
    kernel(crypto_destroy_tfm) = 0x5dbe8093
    Available: kernel-3.10.0-327.36.2.el7.x86_64 (updates)
    kernel(crypto_destroy_tfm) = 0x5dbe8093
    Available: kernel-debug-3.10.0-327.el7.x86_64 (base)
    kernel(crypto_destroy_tfm) = 0x94d1a071
    Available: kernel-debug-3.10.0-327.3.1.el7.x86_64 (updates)
    kernel(crypto_destroy_tfm) = 0x94d1a071
    Available: kernel-debug-3.10.0-327.4.4.el7.x86_64 (updates)
    kernel(crypto_destroy_tfm) = 0x94d1a071
    Available: kernel-debug-3.10.0-327.4.5.el7.x86_64 (updates)
    kernel(crypto_destroy_tfm) = 0x94d1a071
    Available: kernel-debug-3.10.0-327.10.1.el7.x86_64 (updates)
    kernel(crypto_destroy_tfm) = 0x94d1a071
    Available: kernel-debug-3.10.0-327.13.1.el7.x86_64 (updates)
    kernel(crypto_destroy_tfm) = 0x94d1a071
    Available: kernel-debug-3.10.0-327.18.2.el7.x86_64 (updates)
    kernel(crypto_destroy_tfm) = 0x94d1a071
    Available: kernel-debug-3.10.0-327.22.2.el7.x86_64 (updates)
    kernel(crypto_destroy_tfm) = 0x94d1a071
    Available: kernel-debug-3.10.0-327.28.2.el7.x86_64 (updates)
    kernel(crypto_destroy_tfm) = 0x94d1a071
    Available: kernel-debug-3.10.0-327.28.3.el7.x86_64 (updates)
    kernel(crypto_destroy_tfm) = 0x94d1a071
    Available: kernel-debug-3.10.0-327.36.1.el7.x86_64 (updates)
    kernel(crypto_destroy_tfm) = 0x94d1a071
    Available: kernel-debug-3.10.0-327.36.2.el7.x86_64 (updates)
    kernel(crypto_destroy_tfm) = 0x94d1a071
    Error: Package: kmod-drbd84-8.4.8-1_2.el7.elrepo.x86_64 (elrepo)
    Requires: kernel(crypto_alloc_shash) = 0x848fdfcb
    Installed: kernel-3.10.0-327.el7.x86_64 (@anaconda)
    kernel(crypto_alloc_shash) = 0x5be5c8ee
    Installed: kernel-3.10.0-327.36.3.el7.x86_64 (@updates)
    kernel(crypto_alloc_shash) = 0x5be5c8ee
    Installed: kernel-debug-3.10.0-327.36.3.el7.x86_64 (@updates)
    kernel(crypto_alloc_shash) = 0x7946d063
    Available: kernel-3.10.0-327.3.1.el7.x86_64 (updates)
    kernel(crypto_alloc_shash) = 0x5be5c8ee
    Available: kernel-3.10.0-327.4.4.el7.x86_64 (updates)
    kernel(crypto_alloc_shash) = 0x5be5c8ee
    Available: kernel-3.10.0-327.4.5.el7.x86_64 (updates)
    kernel(crypto_alloc_shash) = 0x5be5c8ee
    Available: kernel-3.10.0-327.10.1.el7.x86_64 (updates)
    kernel(crypto_alloc_shash) = 0x5be5c8ee
    Available: kernel-3.10.0-327.13.1.el7.x86_64 (updates)
    kernel(crypto_alloc_shash) = 0x5be5c8ee
    Available: kernel-3.10.0-327.18.2.el7.x86_64 (updates)
    kernel(crypto_alloc_shash) = 0x5be5c8ee
    Available: kernel-3.10.0-327.22.2.el7.x86_64 (updates)
    kernel(crypto_alloc_shash) = 0x5be5c8ee
    Available: kernel-3.10.0-327.28.2.el7.x86_64 (updates)
    kernel(crypto_alloc_shash) = 0x5be5c8ee
    Available: kernel-3.10.0-327.28.3.el7.x86_64 (updates)
    kernel(crypto_alloc_shash) = 0x5be5c8ee
    Available: kernel-3.10.0-327.36.1.el7.x86_64 (updates)
    kernel(crypto_alloc_shash) = 0x5be5c8ee
    Available: kernel-3.10.0-327.36.2.el7.x86_64 (updates)
    kernel(crypto_alloc_shash) = 0x5be5c8ee
    Available: kernel-debug-3.10.0-327.el7.x86_64 (base)
    kernel(crypto_alloc_shash) = 0x7946d063
    Available: kernel-debug-3.10.0-327.3.1.el7.x86_64 (updates)
    kernel(crypto_alloc_shash) = 0x7946d063
    Available: kernel-debug-3.10.0-327.4.4.el7.x86_64 (updates)
    kernel(crypto_alloc_shash) = 0x7946d063
    Available: kernel-debug-3.10.0-327.4.5.el7.x86_64 (updates)
    kernel(crypto_alloc_shash) = 0x7946d063
    Available: kernel-debug-3.10.0-327.10.1.el7.x86_64 (updates)
    kernel(crypto_alloc_shash) = 0x7946d063
    Available: kernel-debug-3.10.0-327.13.1.el7.x86_64 (updates)
    kernel(crypto_alloc_shash) = 0x7946d063
    Available: kernel-debug-3.10.0-327.18.2.el7.x86_64 (updates)
    kernel(crypto_alloc_shash) = 0x7946d063
    Available: kernel-debug-3.10.0-327.22.2.el7.x86_64 (updates)
    kernel(crypto_alloc_shash) = 0x7946d063
    Available: kernel-debug-3.10.0-327.28.2.el7.x86_64 (updates)
    kernel(crypto_alloc_shash) = 0x7946d063
    Available: kernel-debug-3.10.0-327.28.3.el7.x86_64 (updates)
    kernel(crypto_alloc_shash) = 0x7946d063
    Available: kernel-debug-3.10.0-327.36.1.el7.x86_64 (updates)
    kernel(crypto_alloc_shash) = 0x7946d063
    Available: kernel-debug-3.10.0-327.36.2.el7.x86_64 (updates)
    kernel(crypto_alloc_shash) = 0x7946d063
    Error: Package: kmod-drbd84-8.4.8-1_2.el7.elrepo.x86_64 (elrepo)
    Requires: kernel >= 3.10.0-514.el7
    Installed: kernel-3.10.0-327.el7.x86_64 (@anaconda)
    kernel = 3.10.0-327.el7
    Installed: kernel-3.10.0-327.36.3.el7.x86_64 (@updates)
    kernel = 3.10.0-327.36.3.el7
    Installed: kernel-debug-3.10.0-327.36.3.el7.x86_64 (@updates)
    kernel = 3.10.0-327.36.3.el7
    Available: kernel-3.10.0-327.3.1.el7.x86_64 (updates)
    kernel = 3.10.0-327.3.1.el7
    Available: kernel-3.10.0-327.4.4.el7.x86_64 (updates)
    kernel = 3.10.0-327.4.4.el7
    Available: kernel-3.10.0-327.4.5.el7.x86_64 (updates)
    kernel = 3.10.0-327.4.5.el7
    Available: kernel-3.10.0-327.10.1.el7.x86_64 (updates)
    kernel = 3.10.0-327.10.1.el7
    Available: kernel-3.10.0-327.13.1.el7.x86_64 (updates)
    kernel = 3.10.0-327.13.1.el7
    Available: kernel-3.10.0-327.18.2.el7.x86_64 (updates)
    kernel = 3.10.0-327.18.2.el7
    Available: kernel-3.10.0-327.22.2.el7.x86_64 (updates)
    kernel = 3.10.0-327.22.2.el7
    Available: kernel-3.10.0-327.28.2.el7.x86_64 (updates)
    kernel = 3.10.0-327.28.2.el7
    Available: kernel-3.10.0-327.28.3.el7.x86_64 (updates)
    kernel = 3.10.0-327.28.3.el7
    Available: kernel-3.10.0-327.36.1.el7.x86_64 (updates)
    kernel = 3.10.0-327.36.1.el7
    Available: kernel-3.10.0-327.36.2.el7.x86_64 (updates)
    kernel = 3.10.0-327.36.2.el7
    Available: kernel-debug-3.10.0-327.el7.x86_64 (base)
    kernel = 3.10.0-327.el7
    Available: kernel-debug-3.10.0-327.3.1.el7.x86_64 (updates)
    kernel = 3.10.0-327.3.1.el7
    Available: kernel-debug-3.10.0-327.4.4.el7.x86_64 (updates)
    kernel = 3.10.0-327.4.4.el7
    Available: kernel-debug-3.10.0-327.4.5.el7.x86_64 (updates)
    kernel = 3.10.0-327.4.5.el7
    Available: kernel-debug-3.10.0-327.10.1.el7.x86_64 (updates)
    kernel = 3.10.0-327.10.1.el7
    Available: kernel-debug-3.10.0-327.13.1.el7.x86_64 (updates)
    kernel = 3.10.0-327.13.1.el7
    Available: kernel-debug-3.10.0-327.18.2.el7.x86_64 (updates)
    kernel = 3.10.0-327.18.2.el7
    Available: kernel-debug-3.10.0-327.22.2.el7.x86_64 (updates)
    kernel = 3.10.0-327.22.2.el7
    Available: kernel-debug-3.10.0-327.28.2.el7.x86_64 (updates)
    kernel = 3.10.0-327.28.2.el7
    Available: kernel-debug-3.10.0-327.28.3.el7.x86_64 (updates)
    kernel = 3.10.0-327.28.3.el7
    Available: kernel-debug-3.10.0-327.36.1.el7.x86_64 (updates)
    kernel = 3.10.0-327.36.1.el7
    Available: kernel-debug-3.10.0-327.36.2.el7.x86_64 (updates)
    kernel = 3.10.0-327.36.2.el7
    Error: Package: kmod-drbd84-8.4.8-1_2.el7.elrepo.x86_64 (elrepo)
    Requires: kernel(crypto_alloc_ahash) = 0x035deac4
    Installed: kernel-3.10.0-327.el7.x86_64 (@anaconda)
    kernel(crypto_alloc_ahash) = 0x28226cfc
    Installed: kernel-3.10.0-327.36.3.el7.x86_64 (@updates)
    kernel(crypto_alloc_ahash) = 0x28226cfc
    Installed: kernel-debug-3.10.0-327.36.3.el7.x86_64 (@updates)
    kernel(crypto_alloc_ahash) = 0x0cd07bd3
    Available: kernel-3.10.0-327.3.1.el7.x86_64 (updates)
    kernel(crypto_alloc_ahash) = 0x28226cfc
    Available: kernel-3.10.0-327.4.4.el7.x86_64 (updates)
    kernel(crypto_alloc_ahash) = 0x28226cfc
    Available: kernel-3.10.0-327.4.5.el7.x86_64 (updates)
    kernel(crypto_alloc_ahash) = 0x28226cfc
    Available: kernel-3.10.0-327.10.1.el7.x86_64 (updates)
    kernel(crypto_alloc_ahash) = 0x28226cfc
    Available: kernel-3.10.0-327.13.1.el7.x86_64 (updates)
    kernel(crypto_alloc_ahash) = 0x28226cfc
    Available: kernel-3.10.0-327.18.2.el7.x86_64 (updates)
    kernel(crypto_alloc_ahash) = 0x28226cfc
    Available: kernel-3.10.0-327.22.2.el7.x86_64 (updates)
    kernel(crypto_alloc_ahash) = 0x28226cfc
    Available: kernel-3.10.0-327.28.2.el7.x86_64 (updates)
    kernel(crypto_alloc_ahash) = 0x28226cfc
    Available: kernel-3.10.0-327.28.3.el7.x86_64 (updates)
    kernel(crypto_alloc_ahash) = 0x28226cfc
    Available: kernel-3.10.0-327.36.1.el7.x86_64 (updates)
    kernel(crypto_alloc_ahash) = 0x28226cfc
    Available: kernel-3.10.0-327.36.2.el7.x86_64 (updates)
    kernel(crypto_alloc_ahash) = 0x28226cfc
    Available: kernel-debug-3.10.0-327.el7.x86_64 (base)
    kernel(crypto_alloc_ahash) = 0x0cd07bd3
    Available: kernel-debug-3.10.0-327.3.1.el7.x86_64 (updates)
    kernel(crypto_alloc_ahash) = 0x0cd07bd3
    Available: kernel-debug-3.10.0-327.4.4.el7.x86_64 (updates)
    kernel(crypto_alloc_ahash) = 0x0cd07bd3
    Available: kernel-debug-3.10.0-327.4.5.el7.x86_64 (updates)
    kernel(crypto_alloc_ahash) = 0x0cd07bd3
    Available: kernel-debug-3.10.0-327.10.1.el7.x86_64 (updates)
    kernel(crypto_alloc_ahash) = 0x0cd07bd3
    Available: kernel-debug-3.10.0-327.13.1.el7.x86_64 (updates)
    kernel(crypto_alloc_ahash) = 0x0cd07bd3
    Available: kernel-debug-3.10.0-327.18.2.el7.x86_64 (updates)
    kernel(crypto_alloc_ahash) = 0x0cd07bd3
    Available: kernel-debug-3.10.0-327.22.2.el7.x86_64 (updates)
    kernel(crypto_alloc_ahash) = 0x0cd07bd3
    Available: kernel-debug-3.10.0-327.28.2.el7.x86_64 (updates)
    kernel(crypto_alloc_ahash) = 0x0cd07bd3
    Available: kernel-debug-3.10.0-327.28.3.el7.x86_64 (updates)
    kernel(crypto_alloc_ahash) = 0x0cd07bd3
    Available: kernel-debug-3.10.0-327.36.1.el7.x86_64 (updates)
    kernel(crypto_alloc_ahash) = 0x0cd07bd3
    Available: kernel-debug-3.10.0-327.36.2.el7.x86_64 (updates)
    kernel(crypto_alloc_ahash) = 0x0cd07bd3
    You could try using –skip-broken to work around the problem
    You could try running: rpm -Va –nofiles –nodigest
    [[email protected] ~]#

    It would seem the packages are not compatible with the kernel.

    • I use kernel 3.10.0-327.36.3.el7 with kmod-drbd84-8.4.7. I haven’t upgraded to kmod-drbd84-8.4.8 yet.

      According to changelog, kmod-drbd84-8.4.8-1_2.el7 requires kernel 3.10.0-514.el7. Therefore upgrade your RHEL kernel to the version specified, wait for CentOS 7.3 to be released, or downgrade to kmod-drbd84-8.4.7.

    • Hi.

      That worked. Maybe a slight change to your guide but:

      yum install -y kmod-drbd84-8.4.7 drbd84-utils

      It works without all these errors. I guess its due to a newer version in yum.

    • As Tomas has said it’s the kernel version. I’ve just checked and CentOS 7.3 (1611) has kernel 3.10.0-514.el7 in it. You can get it now if you enable the continuous release repository (# yum-config-manager –enable cr), just be aware that the CR repo doesn’t have the same QA testing on the packages as the official release, which I would expect to be released relatively soon. Good to test with in a pre-production environment, not so much in production.

Leave a Reply

Your email address will not be published. Required fields are marked *