Setting up a Kerberised NFS Server on RHEL 7

We are going to set up a Kerberised NFSv4 server.
[UPDATE]: instructions have been tested on RHEL 7.0, 7.1 and 7.2. Notes for different versions have also been added, where necessary.

Requirements

To establish a Kerberised session between NFS client and host, a few things are required (credit goes to Sander van Vugt).

/etc/krb5.keytab: This file contains the security principals for both the NFS server as the NFS client, and it is required to join the Kerberos realm. In the /etc/krb5.keytab file, the host principal, the NFS principal, or both principals are contained.

A Kerberised user session: After configuring remote authentication against a Kerberos server, the login procedure is already Kerberised and no additional action is required.

The sec=method option, both in the share definition and in the mount options. This enables the desired Kerberos method. The following ones are available: none, sys, krb5, krb5i and krb5p.

The nfs-server as well as the nfs-secure-server services must be active on the NFS server. We have to manually load these services and enable them on RHEL 7.0.

The nfs-secure service must be loaded on the client. We have to manually load and enable this service on RHEL 7.0.

The Lab

We have three RHEL 7.0 servers (with no updates installed) available in our lab:

ipa.rhce.local (10.8.8.70)FreeIPA server, provides DNS, LDAP, NTP and Kerberos services,
srv1.rhce.local (10.8.8.71) – will be configured as a Kerberised NFS server,
srv2.rhce.local (10.8.8.72) – will be configured as an NFS client.

All three servers have SELinux set to enforcing mode. Our FreeIPA server setup can be found here.

Servers srv1 and srv2 have been configured for LDAP and Kerberos authentication using sssd, check this blog post for details.

On a RHEL 7.0 with no updates installed, we have to import the Kerberos key tabs on both srv1 and srv2 servers, run nfs-secure-server and nfs-secure and mount the share with sec=krb5p.

Keep Time in Sync

To get Kerberos running, NTP synchronisation and hostname resolution must be working. This is required for both the NFS server srv1 and the NFS client srv2.

Our FreeIPA server provides NTP and DNS services. Time synchronisation can be achieved by using ntpd, or chronyd. We use the latter in this article.

Install package on both srv1 and srv2:

# yum install -y chrony

Open /etc/chrony.conf and add the line pointing to our NTP server:

server ipa.rhce.local iburst

Enable and start the service:

# systemctl enable chronyd && systemctl start chronyd

Enable NTP based network time synchronisation:

# timedatectl set-ntp true

Check:

# chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* ipa.rhce.local                3   6    17    41  -1761ns[  +10us] +/-   83ms

Base NFS Server Setup

All commands in this section are run on the server srv1.

The nfs-utils package version used in the article is 1.3.0.

Packages, Services and Firewall

# yum install -y nfs-utils
# systemctl enable nfs-server && systemctl start nfs-server

We can optionally enable the rpcbind daemon to convert RPC program numbers into addresses, this is used by NFSv3, but not required for NFSv4.

# systemctl enable rpcbind && systemctl start rpcbind

On RHEL 7.0, we have to enable the nfs-secure-server service:

# systemctl enable nfs-secure-server

These are nfs related services available on RHEL 7.0:

# ls -lon /usr/lib/systemd/system/|grep " nfs-"
-rw-r--r--. 1 0 286 Mar 26 2014 nfs-blkmap.service
-rw-r--r--. 1 0 265 Mar 26 2014 nfs-idmap.service
-rw-r--r--. 1 0 491 Mar 26 2014 nfs-lock.service
lrwxrwxrwx. 1 0 16 Jun 3 21:02 nfslock.service -> nfs-lock.service
-rw-r--r--. 1 0 228 Mar 26 2014 nfs-mountd.service
-rw-r--r--. 1 0 263 Mar 26 2014 nfs-rquotad.service
-rw-r--r--. 1 0 332 Mar 26 2014 nfs-secure-server.service
-rw-r--r--. 1 0 279 Mar 26 2014 nfs-secure.service
-rw-r--r--. 1 0 650 Mar 26 2014 nfs-server.service
lrwxrwxrwx. 1 0 18 Jun 3 21:02 nfs.service -> nfs-server.service
lrwxrwxrwx. 1 0 18 Jun 3 21:02 rpcgssd.service -> nfs-secure.service
lrwxrwxrwx. 1 0 17 Jun 3 21:02 rpcidmapd.service -> nfs-idmap.service
lrwxrwxrwx. 1 0 25 Jun 3 21:02 rpcsvcgssd.service -> nfs-secure-server.service

On RHEL 7.1 and RHEL 7.2, the nfs-secure-server service becomes a static service and cannot be enabled. It is started by nfs-server service automatically assuming the file /etc/krb5.keytab is present on the system.

These are nfs related services that are available on RHEL 7.1 and RHEL 7.2:

# ls -lon /usr/lib/systemd/system/|grep " nfs-"
-rw-r--r--. 1 0 345 Jan 23 2015 nfs-blkmap.service
-rw-r--r--. 1 0 215 Jan 23 2015 nfs-blkmap.target
-rw-r--r--. 1 0 477 Jan 23 2015 nfs-client.target
-rw-r--r--. 1 0 144 Jan 23 2015 nfs-config.service
-rw-r--r--. 1 0 240 Jan 23 2015 nfs-idmapd.service
lrwxrwxrwx. 1 0 18 Jun 23 12:41 nfs-idmap.service -> nfs-idmapd.service
lrwxrwxrwx. 1 0 17 Jun 23 12:41 nfs-lock.service -> rpc-statd.service
-rw-r--r--. 1 0 300 Jan 23 2015 nfs-mountd.service
lrwxrwxrwx. 1 0 19 Jun 23 12:41 nfs-secure-server.service -> rpc-svcgssd.service
lrwxrwxrwx. 1 0 16 Jun 23 12:41 nfs-secure.service -> rpc-gssd.service
-rw-r--r--. 1 0 881 Jan 23 2015 nfs-server.service
lrwxrwxrwx. 1 0 18 Jun 23 12:41 nfs.service -> nfs-server.service
-rw-r--r--. 1 0 567 Jan 23 2015 nfs-utils.service

Notice that the nfs-idmap service becomes a symbolic link to nfs-idmapd.

On RHEL 7.0, don’t start the nfs-secure-server just yet, as there is no keytab file created. Secure NFS server will fail without it.

Configure firewall:

# firewall-cmd --permanent --add-service=nfs
# firewall-cmd --reload

Be advised that rpc-bind and mountd services are only required by NFSv3, and can be skipped when setting up an NFSv4 server.

# firewall-cmd --permanent --add-service={mountd,rpc-bind}
# firewall-cmd --reload

Configure NFS Exports

We are going to create three different exports as explained below:

/srv/nfs_pub – a public NFS share with ‘all_squash’,
/srv/nfs_group – a share for group collaboration,
/srv/nfs_secure – a secure Kerberised NFS share.

Create directories:

# mkdir -m 0755 /srv/nfs_{pub,group,secure}

Change ownership for the public NFS share. All users will be squashed. We will map all uids and gids to the NFS anonymous user, in this case the nfsnobody account:

# chown nfsnobody:nfsnobody /srv/nfs_pub

Configure collaboration for the group share:

# groupadd devops
# chgrp devops /srv/nfs_group
# chmod 2770 /srv/nfs_group

For the kerberised NFS share, we are going to make an LDAP user alice the owner of the share so that we can easily test authentication:

# chown alice /srv/nfs_secure

Let us check the default SELinux context:

# ls -Z /srv
drwxrws---. root      devops    unconfined_u:object_r:var_t:s0   /srv/nfs_group
drwxr-xr-x. alice     root      unconfined_u:object_r:var_t:s0   /srv/nfs_secure
drwxr-xr-x. nfsnobody nfsnobody unconfined_u:object_r:var_t:s0   /srv/nfs_pub

For the public share, we want to the public_content_rw_t context type. For other shares, the nfs_t type should suffice.

# yum install -y policycoreutils-python
# semanage fcontext -a -t public_content_rw_t "/srv/nfs_pub(/.*)?"
# semanage fcontext -a -t nfs_t "/srv/nfs_group(/.*)?"
# semanage fcontext -a -t nfs_t "/srv/nfs_secure(/.*)?"

Note that files labeled with the public_content_t type allow them to be read by FTP, Apache, Samba and rsync. Files labeled with the public_content_rw_t type require booleans to be set before services can write to files labeled with the public_content_rw_t type.

To allow the NFS server to modify public files used for public file transfer services, we must turn on the nfsd_anon_write boolean:

# setsebool -P nfsd_anon_write=1

Restore SELinux context:

# restorecon -Rv /srv/nfs_*

Verify:

# ls -Z /srv
drwxrws---. root      devops    unconfined_u:object_r:nfs_t:s0   nfs_group
drwxr-xr-x. nfsnobody nfsnobody unconfined_u:object_r:public_content_rw_t:s0 nfs_pub
drwxr-xr-x. alice     root      unconfined_u:object_r:nfs_t:s0   nfs_secure

Open the file /etc/exports and add the following lines:

/srv/nfs_pub     10.8.8.0/24(sec=sys,rw,sync,root_squash,all_squash)
/srv/nfs_group   10.8.8.0/24(sec=sys,rw,sync,root_squash)
/srv/nfs_secure  srv2.rhce.local(sec=krb5p,rw,sync)

We can see that access to the secure share is allowed from the server srv2 only. Access to the other two shares is allowed from the whole LAN.

Export the shares:

# exportfs -rav
exporting srv2.rhce.local:/srv/nfs_secure
exporting 10.8.8.0/24:/srv/nfs_group
exporting 10.8.8.0/24:/srv/nfs_pub

Check what NFS protocols are enabled:

# cat /proc/fs/nfsd/versions
-2 +3 +4 +4.1 -4.2

By default, NFS mounts have the SELinux context nfs_t independently of the SELinux context they have on the NFS server. The NFS server can be forced to properly export the SELinux context of a share by switching to NFS version 4.2. Note that this option needs to be turned on explicitly.

Enable NFS v4.2 by adding the following line to the file /etc/sysconfig/nfs:

RPCNFSDARGS="-V 4.2"

On the client side, mount -o vers=4.2 must be specified as the mount option.

One other thing to check, by default the nfs_export_all_ro and the nfs_export_all_rw SELinux booleans are both enabled:

# getsebool -a | grep nfs_ex
nfs_export_all_ro --> on
nfs_export_all_rw --> on

This allows the NFS daemon to read and write almost any file. Disable the booleans to lock down the capabilities.

Kerberos Keytab File

There are a couple of ways of creating the keytab file /etc/krb5.keytab.

If we know the IPA admin credentials, we can obtain a Kerberos ticket, and use ipa-getkeytab command.

# yum install -y ipa-client

Obtain a Kerberos ticket and verify:

# kinit admin
# klist

Get a keytab for our Kerberos NFS principal:

# ipa-getkeytab -s ipa.rhce.local -p nfs/srv1.rhce.local -k /etc/krb5.keytab
Keytab successfully retrieved and stored in: /etc/krb5.keytab

Verify with:

# klist -k
   2 [email protected]
   2 [email protected]
   2 [email protected]
   2 [email protected]
   2 [email protected]
   2 [email protected]

Now if we don’t have Kerberos admin credentials, we can alternatively download the keytab file for the server via FTP (assuming it was made available on the Kerberos server):

# wget -O /etc/krb5.keytab ftp://ipa.rhce.local/pub/srv1.keytab
# chmod 0600 /etc/krb5.keytab
# restorecon -v /etc/krb5.keytab

Verify:

klist -k
   3 [email protected]
   3 [email protected]
   3 [email protected]
   3 [email protected]
   3 [email protected]
   3 [email protected]

Once the keytab is created, start the nfs-secure-server service:

# systemctl start nfs-secure-server

For verbose logging, open the file /etc/sysconfig/nfs, assign the -vvv string to the RPCGSSDARGS variable and restart the nfs-secure-server service.

NFS Client Setup

All commands in this section are run on the server srv2.

Packages and Services

# yum install -y nfs-utils

On RHEL 7.0, enable the nfs-secure service:

# systemctl enable nfs-secure

On RHEL 7.1 and RHEL 7.2, the nfs-secure is a static service and cannot be enabled. It is started by the nfs-client.target service assuming the file /etc/krb5.keytab is present on the system:

# systemctl enable nfs-client.target

Mount a Kerberised NFS Share

If the NFSv3 server has firewall configured to allow mountd and rpcbind traffic, we can use the showmount command to verify that the client can see the shares:

# showmount -e srv1.rhce.local
Export list for srv1.rhce.local:
/srv/nfs_group  10.8.8.0/24
/srv/nfs_pub    10.8.8.0/24
/srv/nfs_secure srv2.rhce.local

Because NFSv4 does not use the mountd daemon, showmount will not return information about version 4 mounts.

As with the NFS server, if we know the Kerberos admin credentials, we can obtain the keytab with ipa-client:

# yum install -y ipa-client

Obtain a Kerberos ticket and verify:

# kinit admin
# klist

Get a keytab for our Kerberos NFS principal:

# ipa-getkeytab -s ipa.rhce.local -p nfs/srv2.rhce.local -k /etc/krb5.keytab
Keytab successfully retrieved and stored in: /etc/krb5.keytab

Verify with:

# klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   4 [email protected]
   4 [email protected]
   4 [email protected]
   4 [email protected]
   4 [email protected]
   4 [email protected]

As we can see above, our keytab does not contain host credentials as we haven’t created them when setting up a FreeIPA server. Therefore it’s vital that our NFS client is configured for Kerberos authentication (with authconfig-tui f.e.), otherwise we won’t be able to mount the kerberised NFS share, and rpc.gssd will complain a lot:

ERROR: gssd_refresh_krb5_machine_credential: no usable keytab entry found in keytab /etc/krb5.keytab for connection with host srv1.rhce.local

If the Kerberos credentials are not available, we can get the keytab from FTP (as we have configured it for our system). Download the keytab file from the Kerberos server:

# wget ftp://ipa.rhce.local/pub/srv2.keytab -O /etc/krb5.keytab
# chmod 0600 /etc/krb5.keytab 
# restorecon -v /etc/krb5.keytab
# klist
klist: No credentials cache found (ticket cache FILE:/tmp/krb5cc_0)
# klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   5 [email protected]
   5 [email protected]
   5 [email protected]
   5 [email protected]
   5 [email protected]
   5 [email protected]

Start the nfs-secure service:

# systemctl start nfs-secure

For verbose logging, open the file /etc/sysconfig/nfs, assign the -vvv string to the RPCGSSDARGS variable and restart the nfs-secure service. You may need it if you run into problems.

Mount the share:

# mkdir /mnt/nfs_secure
# mount -t nfs4 -o vers=4.2,sec=krb5p srv1.rhce.local:/srv/nfs_secure /mnt/nfs_secure

If you get the error message “mount.nfs: an incorrect mount option was specified, check that you started nfs-secure daemon on the client and nfs-secure-server on the server.

Verify:

# mount | grep secure
srv1.rhce.local:/srv/nfs_secure on /mnt/nfs_secure type nfs (rw,relatime,seclabel,vers=4.2,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=10.8.8.72,local_lock=none,addr=10.8.8.71)

To mount on boot, add the following to /etc/fstab:

srv1.rhce.local:/srv/nfs_secure /mnt/nfs_secure nfs4 rw,vers=4.2,sec=krb5p 0 0

Note that the _netdev option should not be required to mount the NFSv4 share.

It is my understanding that under NFSv3 (mount type nfs), the _netdev option tells the system to wait to mount until the network is available. With a mount type of nfs4 the _netdev option is ignored as remote mounts are pulled by remote-fs.target.

However, Sander van Vugt recommends that all remote file systems that need to be mounted through /etc/fstab include the _netdev and the x-systemd.automount mount options.

The _netdev mount option ensures that the mount is delayed until the network is fully available. The x-systemd.automount option ensures optimal integration with systemd and will ensure that the mount is made a lot faster.

Test Kerberos, change to user alice to access the share:

# su - alice
Last login: Wed May 18 16:06:51 BST 2016 on pts/2
su: warning: cannot change directory to /home/alice: No such file or directory
id: cannot find name for group ID 1219400005
$ touch /mnt/nfs_secure/file
touch: cannot touch ‘/mnt/nfs_secure/file’: Permission denied
$ kinit
Password for [email protected]:
$ touch /mnt/nfs_secure/file
$ ls -l /mnt/nfs_secure/file
-rw-r--r--. 1 alice 1219400005 0 Jun  3 20:57 /mnt/nfs_secure/file
$ kdestroy
$ exit

Mount Non-Kerberised NFS Shares

Create mountpoints:

# mkdir /mnt/nfs_{pub,group}

Mount the public and group shares as these require no Kerberos configuration:

# mount -t nfs4 -o vers=4.2,sec=sys srv1.rhce.local:/srv/nfs_pub /mnt/nfs_pub 
# mount -t nfs4 -o vers=4.2,sec=sys srv1.rhce.local:/srv/nfs_group /mnt/nfs_group

Problems With Key Version Numbers

Sometimes the key version number (KVNO) used by the KDC and the service principal keys stored in /etc/krb5.keytab for services hosted on the system do not match. The KVNO can get out of synchronisation when a new set of keys are created on the KDC without updating the keytable file with the new keys.

To diagnose, list the keytab entries:

# klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   3 [email protected]
   3 [email protected]
   3 [email protected]
   3 [email protected]
   3 [email protected]
   3 [email protected]

Acquire Kerberos ticket by using the host key:

# kinit -k

Or use admin credentials:

# kinit admin

Check the KVNO that is used by the KDC:

# kvno nfs/srv2.rhce.local
[email protected]: kvno = 4

References

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/kerb-nfs.html

54 thoughts on “Setting up a Kerberised NFS Server on RHEL 7

  1. Good day

    I have been reading your site alongside the certdeport one after finishing the sander’s vedios and jangs book on RHEL 7…and i am really impressed…I will be writing my RHCE on the 23rd of Septemeber and i think i have managed to prepare very ,well thanks to all those sources and mainly this one for being specific.

    I have a few grey areas ,the kerberos server can be the redhat ipa or the standalone server installed from stand alone packages…

    my questions this config below when configuring the redhat cleint

    [libdefaults]
    default_realm = EXAMPLE.COM
    dns_lookup_realm = false
    dns_lookup_kdc = false
    rdns = false
    forwardable = yes
    ticket_lifetime = 24h
    
    [realms]
    EXAMPLE.COM = {
          kdc = ipaserver.example.com:88
          admin_server = ipaserver.example.com:749
          }
    [domain_realm]
    .example.com = EXAMPLE.COM
    example.com = EXAMPLE.COM

    the above is for the redhat IPA configuration
    and for a standalone configuration there are no ports 88 and port 749
    my question is ,how do you then know how to configure a client without knowing the ports if its a redhat IPA server or not ?

    • I’m glad you found it helpful. To configure a Kerberos client I use authconfig-tui. You need to know the REALM and have DNS configured to point to the Kerberos server. No manual configuration of krb5.conf is required.

  2. @tomas so basically what you are saying is the kerberos server should provide DNS services for the client to pick it up? i have a KDC configured but with no DNS service running and it seems i have to configure the clients via authconfig-tui but i have to manually place the server name and ports when it asks for the kdc and the kdc_admin server

    • You need to ensure the hostname resolution is working. It can be a DNS server, or it can be static entries in /etc/hosts, up to you really.

    • I have also noticed that the /etc/sssd/sssd.conf file does get automatically created which makes the whole process complex

    • To use ipa-client-install, you need to know the Kerberos admin password, otherwise you cannot join the domain. I doubt such information will be supplied during the RHCE exam.

  3. Thank you @tomas , i had actually overlooked the fact that there is a password needed to join the IPA server and its mostly like not going to be supplied during the exam and that makes sense to use the other method…

    I luv sssd because its newer so i will stick to sssd.

  4. Interesting information regarding _netdev, I spoke to a Red Hat trainer who advised that it should not be required for SMB/NFS mounting, because when you mount the file system is either specified or automatically detected as either Samba or NFS, and as such it knows that these mounts should come up after the network. Despite this I think I’ll still add it in just to be safe!

  5. @tomas confirm something on this nfs share below

    /srv/nfs_group – a share for group collaboration,

    I am sure we need the devops groups on the client also ,with users in it to be able to create files and directories

  6. Omg! Because of your tutorial I’ve finally got my Kerberized NFS setup to work using CentOS 7.2. I could finally move on with the rest of my labs. Thank you very much!

  7. Hey Tomas, I cannot connect to a Kerberized share in my CentOS 7.2 environment. The service is registered and the key is downloaded from IPA server, LDAP clients are configured correctly. nfs-server service is started on both server and client, but on client, I cannot start nfs-client service:

    [[email protected] /]# systemctl start nfs-client
    Failed to start nfs-client.service: Unit nfs-client.service failed to load: No such file or directory.

    So, getting this:

    [[email protected] /]# mount -o sec=krb5p server1.example.com:/nfs-secureshare /nfssecure
    mount.nfs: an incorrect mount option was specified
    [[email protected] /]# mount -o sec=sys server1.example.com:/nfs-secureshare /nfssecure
    mount.nfs: access denied by server while mounting server1.example.com:/nfs-secureshare

    • The error is self-explanatory, the service file does not exist.

      You need a target, not a service. It’s explained in the blog post.

  8. Yes, thank, Tomas! I have started the target, now I have this:
    [[email protected] ~]# mount -o sec=krb5p -V 4.2 server1.example.com:/nfs-secureshare /nfssecure
    mount from util-linux 2.23.2 (libmount 2.23.0: selinux, debug, assert)
    [[email protected] ~]# echo $?
    0
    [[email protected] ~]# mount | grep nfssecure
    [[email protected] ~]#

    /var/log/messages shows nothing

  9. [[email protected] ~]# mount | grep nfs
    sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
    nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
    [[email protected] ~]# df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/mapper/centos-root 9.4G 1.1G 8.3G 12% /
    devtmpfs 483M 0 483M 0% /dev
    tmpfs 493M 0 493M 0% /dev/shm
    tmpfs 493M 6.7M 486M 2% /run
    tmpfs 493M 0 493M 0% /sys/fs/cgroup
    /dev/sda1 497M 166M 332M 34% /boot
    //server1.example.com/smb 9.4G 1.3G 8.1G 14% /smb
    tmpfs 99M 0 99M 0% /run/user/0

    • This is useless information I’m afraid and doesn’t help at all.

      Try mounting in verbose mode and post the logs.

  10. -v option for “mount” did not show anything. And I dont know where to see the logs (/var/log/messages shows nothing) Here is strace output (you may not to post it all here, just have a look):

    [[email protected] ~]# strace -vts1024 mount -o sec=krb5p -V 4.2 server1.example.com:/nfs-secureshare /nfssecure

    • See “man mount” for the -V parameter that you use. You should figure it out.

      Did you read the blog post? There is an example on how to mount a Kerberised NFS share.

  11. yes, I have read it more carefully now:

    [[email protected] ~]# mount -t nfs4 -o sec=krb5p,vers=4.2 server1.example.com:/nfs-secureshare /nfssecure –verbose
    mount.nfs4: timeout set for Thu Nov 24 02:08:11 2016
    mount.nfs4: trying text-based options ‘sec=krb5p,vers=4.2,addr=10.39.92.67,clientaddr=10.39.92.77’
    mount.nfs4: mount(2): Permission denied
    mount.nfs4: access denied by server while mounting server1.example.com:/nfs-secureshare

    I have a few issues here: time is not synced between server and client (chronyd is configured to sync with IPA server, ports are opened and tcpdump shows traffic on port 123, timedatectl set-ntp true)

    [[email protected] ~]# systemctl status nfs-secure
    Б≈▐ rpc-gssd.service – RPC security service for NFS client and server
    Loaded: loaded (/usr/lib/systemd/system/rpc-gssd.service; static; vendor preset: disabled)
    Active: active (running) since Wed 2016-11-23 23:25:31 +07; 50min ago
    ERROR: gssd_refresh_krb5_machine_credential: no usable keytab entry found in keytab /etc/krb5.keytab for connection with host

    Key version is not 4:

    [[email protected] ~]# klist -k
    Keytab name: FILE:/etc/krb5.keytab
    KVNO Principal
    —- ————————————————————————–
    1 [email protected]
    1 [email protected]
    1 [email protected]
    1 [email protected]

    I will dig into it all….

    • “no usable keytab entry found in keytab /etc/krb5.keytab for connection with host” – what’s the KVNO that is used by the KDC (FreeIPA)?

  12. [[email protected] ~]# klist -k
    Keytab name: FILE:/etc/krb5.keytab
    KVNO Principal
    —- ————————————————————————–
    2 [email protected]
    2 [email protected]
    2 [email protected]
    2 [email protected]
    2 [email protected]
    2 [email protected]

    [[email protected] ~]# klist -k
    Keytab name: FILE:/etc/krb5.keytab
    KVNO Principal
    —- ————————————————————————–
    1 [email protected]
    1 [email protected]
    1 [email protected]
    1 [email protected]

    [[email protected] ~]# klist -k
    Keytab name: FILE:/etc/krb5.keytab
    KVNO Principal
    —- ————————————————————————–
    1 [email protected]
    1 [email protected]
    1 [email protected]
    1 [email protected]

    • [[email protected] ~]# kvno [email protected]
      [email protected]: kvno = 1

    • By using kadmin.local, see here for details. You can also use ipa-getkeytab to generate the keys but do not save them in the host keytab.

  13. kadmin.local
    ktadd NFS/server1.example.com
    ktadd NFS/server2.example.com
    quit

    ipa-getkeytab -s ipa.example.com -p NFS/server1.example.com -k /root/server1.nfs.keytab
    ipa-getkeytab -s ipa.example.com -p NFS/server2.example.com -k /root/server2.nfs.keytab

    copy them to server1 and server2 to /etc/krb5.tab

    Is everything correct?

  14. [[email protected] ~]# mount -o vers=4.2,sec=krb5p server1.example.com:/nfs-secureshare /nfssecure –verbose
    mount.nfs: timeout set for Fri Nov 25 08:31:13 2016
    mount.nfs: trying text-based options ‘vers=4.2,sec=krb5p,addr=192.168.0.102,clientaddr=192.168.0.104’
    mount.nfs: mount(2): Invalid argument
    mount.nfs: an incorrect mount option was specified

    [[email protected] ~]# systemctl status nfs-secure
    ● rpc-gssd.service – RPC security service for NFS client and server
    Loaded: loaded (/usr/lib/systemd/system/rpc-gssd.service; static; vendor preset: disabled)
    Active: active (running) since Fri 2016-11-25 08:26:42 +07; 5s ago
    Process: 3139 ExecStart=/usr/sbin/rpc.gssd $GSSDARGS (code=exited, status=0/SUCCESS)
    Main PID: 3140 (rpc.gssd)
    CGroup: /system.slice/rpc-gssd.service
    └─3140 /usr/sbin/rpc.gssd

    [[email protected] ~]# systemctl status nfs-secure-server
    ● rpc-svcgssd.service – RPC security service for NFS server
    Loaded: loaded (/usr/lib/systemd/system/rpc-svcgssd.service; static; vendor preset: disabled)
    Active: inactive (dead)
    Condition: start condition failed at Fri 2016-11-25 08:28:00 +07; 24s ago
    none of the trigger conditions were met

    This is CentOS 7.2 and nfs-secure-servver should be autostarted by nfs-server, right?

  15. mount.nfs4: trying text-based options ‘sec=krb5p,vers=4.2,addr=192.168.0.102,clientaddr=192.168.0.104’
    mount.nfs4: mount(2): Invalid argument
    mount.nfs4: an incorrect mount option was specified

    [[email protected] ~]# cat /proc/fs/nfsd/versions
    -2 +3 +4 +4.1 +4.2

    rolled-back the whole environment and reinstalled the services and the keys. now getting this.
    knvo shows the same key versions.
    investigating further. from the IPA server installation instructions, I haven’t configured DNS zone transfer
    investigating this further…

  16. set it, /var/log/messages shows nothing.
    its strange: kvno version is changes randomly:

    ipa
    2 [email protected]
    1 [email protected]
    1 [email protected]

    server1
    2 [email protected]
    server2
    2 [email protected]

    previously, with the same actions done, the versions on the servers were 1
    still getting
    mount.nfs: mount(2): Invalid argument
    mount.nfs: an incorrect mount option was specified

  17. the following command can be used during troubleshooting to get the creds manually:

    # kinit -k -t /etc/krb5.keytab [email protected]

    I tried and got an error, something like cannot contact to KDC

  18. when I used authconfig-tui to configure LDAP client, I set “User DNS to find servers”, something like that
    as I can see, specifying ipa.example.com in /etc/hosts was not enough

    so, to sum-up:

    1. principals were not added to keytab
    2. nfs version was not specified in /etc/sysconfig/nfs
    3. DNS was not configured correctly

    Thank you very much!

    • There are two links mentioned for how to configure LDAP auth, one with SSSD and another with nslcd. I take if you haven’t checked them? :)

  19. in your sample exam a keytab for srv1 is already provided on FTP
    how will it be on the exam? both keytabs for srv1 and srv2 will be on FTP or I will make these files using ipa-getkeytab?

  20. if one keytab is already provided, where is the second one? if I kame it, the kvno versions can differ

Leave a Reply

Your email address will not be published. Required fields are marked *