Setting up a Kerberised NFS Server on RHEL 7

We are going to set up a Kerberised NFSv4 server.

[UPDATE]: instructions have been tested on RHEL 7.0, 7.1 and 7.2. Notes for different versions have also been added, where necessary.

Requirements

To establish a Kerberised session between NFS client and host, a few things are required (credit goes to Sander van Vugt).

/etc/krb5.keytab: This file contains the security principals for both the NFS server as the NFS client, and it is required to join the Kerberos realm. In the /etc/krb5.keytab file, the host principal, the NFS principal, or both principals are contained.

A Kerberised user session: After configuring remote authentication against a Kerberos server, the login procedure is already Kerberised and no additional action is required.

The sec=method option, both in the share definition and in the mount options. This enables the desired Kerberos method. The following ones are available: none, sys, krb5, krb5i and krb5p.

The nfs-server as well as the nfs-secure-server services must be active on the NFS server. We have to manually load these services and enable them on RHEL 7.0.

The nfs-secure service must be loaded on the client. We have to manually load and enable this service on RHEL 7.0.

The Lab

We have three RHEL 7.0 servers (with no updates installed) available in our lab:

ipa.rhce.local (10.8.8.70)FreeIPA server, provides DNS, LDAP, NTP and Kerberos services,
srv1.rhce.local (10.8.8.71) – will be configured as a Kerberised NFS server,
srv2.rhce.local (10.8.8.72) – will be configured as an NFS client.

All three servers have SELinux set to enforcing mode. Our FreeIPA server setup can be found here.

Servers srv1 and srv2 have been configured for LDAP and Kerberos authentication using sssd, check this blog post for details.

On a RHEL 7.0 with no updates installed, we have to import the Kerberos key tabs on both srv1 and srv2 servers, run nfs-secure-server and nfs-secure and mount the share with sec=krb5p.

Keep Time in Sync

To get Kerberos running, NTP synchronisation and hostname resolution must be working. This is required for both the NFS server srv1 and the NFS client srv2.

Our FreeIPA server provides NTP and DNS services. Time synchronisation can be achieved by using ntpd, or chronyd. We use the latter in this article.

Install package on both srv1 and srv2:

# yum install -y chrony

Open /etc/chrony.conf and add the line pointing to our NTP server:

server ipa.rhce.local iburst

Enable and start the service:

# systemctl enable chronyd && systemctl start chronyd

Enable NTP based network time synchronisation:

# timedatectl set-ntp true

Check:

# chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* ipa.rhce.local                3   6    17    41  -1761ns[  +10us] +/-   83ms

Base NFS Server Setup

All commands in this section are run on the server srv1.

The nfs-utils package version used in the article is 1.3.0.

Packages, Services and Firewall

# yum install -y nfs-utils
# systemctl enable nfs-server && systemctl start nfs-server

We can optionally enable the rpcbind daemon to convert RPC program numbers into addresses, this is used by NFSv3, but not required for NFSv4.

# systemctl enable rpcbind && systemctl start rpcbind

On RHEL 7.0, we have to enable the nfs-secure-server service:

# systemctl enable nfs-secure-server

These are nfs related services available on RHEL 7.0:

# ls -lon /usr/lib/systemd/system/|grep " nfs-"
-rw-r--r--. 1 0 286 Mar 26 2014 nfs-blkmap.service
-rw-r--r--. 1 0 265 Mar 26 2014 nfs-idmap.service
-rw-r--r--. 1 0 491 Mar 26 2014 nfs-lock.service
lrwxrwxrwx. 1 0 16 Jun 3 21:02 nfslock.service -> nfs-lock.service
-rw-r--r--. 1 0 228 Mar 26 2014 nfs-mountd.service
-rw-r--r--. 1 0 263 Mar 26 2014 nfs-rquotad.service
-rw-r--r--. 1 0 332 Mar 26 2014 nfs-secure-server.service
-rw-r--r--. 1 0 279 Mar 26 2014 nfs-secure.service
-rw-r--r--. 1 0 650 Mar 26 2014 nfs-server.service
lrwxrwxrwx. 1 0 18 Jun 3 21:02 nfs.service -> nfs-server.service
lrwxrwxrwx. 1 0 18 Jun 3 21:02 rpcgssd.service -> nfs-secure.service
lrwxrwxrwx. 1 0 17 Jun 3 21:02 rpcidmapd.service -> nfs-idmap.service
lrwxrwxrwx. 1 0 25 Jun 3 21:02 rpcsvcgssd.service -> nfs-secure-server.service

On RHEL 7.1 and RHEL 7.2, the nfs-secure-server service becomes a static service and cannot be enabled. It is started by nfs-server service automatically assuming the file /etc/krb5.keytab is present on the system.

These are nfs related services that are available on RHEL 7.1 and RHEL 7.2:

# ls -lon /usr/lib/systemd/system/|grep " nfs-"
-rw-r--r--. 1 0 345 Jan 23 2015 nfs-blkmap.service
-rw-r--r--. 1 0 215 Jan 23 2015 nfs-blkmap.target
-rw-r--r--. 1 0 477 Jan 23 2015 nfs-client.target
-rw-r--r--. 1 0 144 Jan 23 2015 nfs-config.service
-rw-r--r--. 1 0 240 Jan 23 2015 nfs-idmapd.service
lrwxrwxrwx. 1 0 18 Jun 23 12:41 nfs-idmap.service -> nfs-idmapd.service
lrwxrwxrwx. 1 0 17 Jun 23 12:41 nfs-lock.service -> rpc-statd.service
-rw-r--r--. 1 0 300 Jan 23 2015 nfs-mountd.service
lrwxrwxrwx. 1 0 19 Jun 23 12:41 nfs-secure-server.service -> rpc-svcgssd.service
lrwxrwxrwx. 1 0 16 Jun 23 12:41 nfs-secure.service -> rpc-gssd.service
-rw-r--r--. 1 0 881 Jan 23 2015 nfs-server.service
lrwxrwxrwx. 1 0 18 Jun 23 12:41 nfs.service -> nfs-server.service
-rw-r--r--. 1 0 567 Jan 23 2015 nfs-utils.service

Notice that the nfs-idmap service becomes a symbolic link to nfs-idmapd.

On RHEL 7.0, don’t start the nfs-secure-server just yet, as there is no keytab file created. Secure NFS server will fail without it.

Configure firewall:

# firewall-cmd --permanent --add-service=nfs
# firewall-cmd --reload

Be advised that rpc-bind and mountd services are only required by NFSv3, and can be skipped when setting up an NFSv4 server.

# firewall-cmd --permanent --add-service={mountd,rpc-bind}
# firewall-cmd --reload

Configure NFS Exports

We are going to create three different exports as explained below:

/srv/nfs_pub – a public NFS share with ‘all_squash’,
/srv/nfs_group – a share for group collaboration,
/srv/nfs_secure – a secure Kerberised NFS share.

Create directories:

# mkdir -m 0755 /srv/nfs_{pub,group,secure}

Change ownership for the public NFS share. All users will be squashed. We will map all uids and gids to the NFS anonymous user, in this case the nfsnobody account:

# chown nfsnobody:nfsnobody /srv/nfs_pub

Configure collaboration for the group share:

# groupadd devops
# chgrp devops /srv/nfs_group
# chmod 2770 /srv/nfs_group

For the kerberised NFS share, we are going to make an LDAP user alice the owner of the share so that we can easily test authentication:

# chown alice /srv/nfs_secure

Let us check the default SELinux context:

# ls -Z /srv
drwxrws---. root      devops    unconfined_u:object_r:var_t:s0   /srv/nfs_group
drwxr-xr-x. alice     root      unconfined_u:object_r:var_t:s0   /srv/nfs_secure
drwxr-xr-x. nfsnobody nfsnobody unconfined_u:object_r:var_t:s0   /srv/nfs_pub

For the public share, we want to the public_content_rw_t context type. For other shares, the nfs_t type should suffice.

# yum install -y policycoreutils-python
# semanage fcontext -a -t public_content_rw_t "/srv/nfs_pub(/.*)?"
# semanage fcontext -a -t nfs_t "/srv/nfs_group(/.*)?"
# semanage fcontext -a -t nfs_t "/srv/nfs_secure(/.*)?"

Note that files labeled with the public_content_t type allow them to be read by FTP, Apache, Samba and rsync. Files labeled with the public_content_rw_t type require booleans to be set before services can write to files labeled with the public_content_rw_t type.

To allow the NFS server to modify public files used for public file transfer services, we must turn on the nfsd_anon_write boolean:

# setsebool -P nfsd_anon_write=1

Restore SELinux context:

# restorecon -Rv /srv/nfs_*

Verify:

# ls -Z /srv
drwxrws---. root      devops    unconfined_u:object_r:nfs_t:s0   nfs_group
drwxr-xr-x. nfsnobody nfsnobody unconfined_u:object_r:public_content_rw_t:s0 nfs_pub
drwxr-xr-x. alice     root      unconfined_u:object_r:nfs_t:s0   nfs_secure

Open the file /etc/exports and add the following lines:

/srv/nfs_pub     10.8.8.0/24(sec=sys,rw,sync,root_squash,all_squash)
/srv/nfs_group   10.8.8.0/24(sec=sys,rw,sync,root_squash)
/srv/nfs_secure  srv2.rhce.local(sec=krb5p,rw,sync)

We can see that access to the secure share is allowed from the server srv2 only. Access to the other two shares is allowed from the whole LAN.

Export the shares:

# exportfs -rav
exporting srv2.rhce.local:/srv/nfs_secure
exporting 10.8.8.0/24:/srv/nfs_group
exporting 10.8.8.0/24:/srv/nfs_pub

Check what NFS protocols are enabled:

# cat /proc/fs/nfsd/versions
-2 +3 +4 +4.1 -4.2

By default, NFS mounts have the SELinux context nfs_t independently of the SELinux context they have on the NFS server. The NFS server can be forced to properly export the SELinux context of a share by switching to NFS version 4.2. Note that this option needs to be turned on explicitly.

Enable NFS v4.2 by adding the following line to the file /etc/sysconfig/nfs:

RPCNFSDARGS="-V 4.2"

On the client side, mount -o vers=4.2 must be specified as the mount option.

One other thing to check, by default the nfs_export_all_ro and the nfs_export_all_rw SELinux booleans are both enabled:

# getsebool -a | grep nfs_ex
nfs_export_all_ro --> on
nfs_export_all_rw --> on

This allows the NFS daemon to read and write almost any file. Disable the booleans to lock down the capabilities.

Kerberos Keytab File

There are a couple of ways of creating the keytab file /etc/krb5.keytab.

If we know the IPA admin credentials, we can obtain a Kerberos ticket, and use ipa-getkeytab command.

# yum install -y ipa-client

Obtain a Kerberos ticket and verify:

# kinit admin
# klist

Get a keytab for our Kerberos NFS principal:

# ipa-getkeytab -s ipa.rhce.local -p nfs/srv1.rhce.local -k /etc/krb5.keytab
Keytab successfully retrieved and stored in: /etc/krb5.keytab

Verify with:

# klist -k
   2 nfs/[email protected]
   2 nfs/[email protected]
   2 nfs/[email protected]
   2 nfs/[email protected]
   2 nfs/[email protected]
   2 nfs/[email protected]

Now if we don’t have Kerberos admin credentials, we can alternatively download the keytab file for the server via FTP (assuming it was made available on the Kerberos server):

# wget -O /etc/krb5.keytab ftp://ipa.rhce.local/pub/srv1.keytab
# chmod 0600 /etc/krb5.keytab
# restorecon -v /etc/krb5.keytab

Verify:

klist -k
   3 nfs/[email protected]
   3 nfs/[email protected]
   3 nfs/[email protected]
   3 nfs/[email protected]
   3 nfs/[email protected]
   3 nfs/[email protected]

Once the keytab is created, start the nfs-secure-server service:

# systemctl start nfs-secure-server

Check the NFS server status for any errors.

For verbose logging, open the file /etc/sysconfig/nfs, assign the -vvv string to the RPCGSSDARGS variable and restart the nfs-secure-server service.

Reboot the NFS server before proceeding to the client configuration. After restart, ensure that both services are running:

# systemctl status nfs-server nfs-secure-server

NFS Client Setup

All commands in this section are run on the server srv2.

Packages and Services

# yum install -y nfs-utils

On RHEL 7.0, enable the nfs-secure service:

# systemctl enable nfs-secure

On RHEL 7.1 and RHEL 7.2, the nfs-secure is a static service and cannot be enabled. It is started by the nfs-client.target service assuming the file /etc/krb5.keytab is present on the system:

# systemctl enable nfs-client.target

Mount a Kerberised NFS Share

If the NFSv3 server has firewall configured to allow mountd and rpcbind traffic, we can use the showmount command to verify that the client can see the shares:

# showmount -e srv1.rhce.local
Export list for srv1.rhce.local:
/srv/nfs_group  10.8.8.0/24
/srv/nfs_pub    10.8.8.0/24
/srv/nfs_secure srv2.rhce.local

Because NFSv4 does not use the mountd daemon, showmount will not return information about version 4 mounts.

As with the NFS server, if we know the Kerberos admin credentials, we can obtain the keytab with ipa-client:

# yum install -y ipa-client

Obtain a Kerberos ticket and verify:

# kinit admin
# klist

Get a keytab for our Kerberos NFS principal:

# ipa-getkeytab -s ipa.rhce.local -p nfs/srv2.rhce.local -k /etc/krb5.keytab
Keytab successfully retrieved and stored in: /etc/krb5.keytab

Verify with:

# klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   4 nfs/[email protected]
   4 nfs/[email protected]
   4 nfs/[email protected]
   4 nfs/[email protected]
   4 nfs/[email protected]
   4 nfs/[email protected]

As we can see above, our keytab does not contain host credentials as we haven’t created them when setting up a FreeIPA server. Therefore it’s vital that our NFS client is configured for Kerberos authentication (with authconfig-tui f.e.), otherwise we won’t be able to mount the kerberised NFS share, and rpc.gssd will complain a lot:

ERROR: gssd_refresh_krb5_machine_credential: no usable keytab entry found in keytab /etc/krb5.keytab for connection with host srv1.rhce.local

If the Kerberos credentials are not available, we can get the keytab from FTP (as we have configured it for our system). Download the keytab file from the Kerberos server:

# wget ftp://ipa.rhce.local/pub/srv2.keytab -O /etc/krb5.keytab
# chmod 0600 /etc/krb5.keytab 
# restorecon -v /etc/krb5.keytab
# klist
klist: No credentials cache found (ticket cache FILE:/tmp/krb5cc_0)
# klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   5 nfs/[email protected]
   5 nfs/[email protected]
   5 nfs/[email protected]
   5 nfs/[email protected]
   5 nfs/[email protected]
   5 nfs/[email protected]

Start the nfs-secure service:

# systemctl start nfs-secure

For verbose logging, open the file /etc/sysconfig/nfs, assign the -vvv string to the RPCGSSDARGS variable and restart the nfs-secure service. You may need it if you run into problems.

Mount the share:

# mkdir /mnt/nfs_secure
# mount -t nfs4 -o vers=4.2,sec=krb5p srv1.rhce.local:/srv/nfs_secure /mnt/nfs_secure

If you get the error message “mount.nfs: an incorrect mount option was specified, check that you started nfs-secure daemon on the client and nfs-secure-server on the server.

Verify:

# mount | grep secure
srv1.rhce.local:/srv/nfs_secure on /mnt/nfs_secure type nfs (rw,relatime,seclabel,vers=4.2,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=10.8.8.72,local_lock=none,addr=10.8.8.71)

To mount on boot, add the following to /etc/fstab:

srv1.rhce.local:/srv/nfs_secure /mnt/nfs_secure nfs4 rw,vers=4.2,sec=krb5p 0 0

Note that the _netdev option should not be required to mount the NFSv4 share.

It is my understanding that under NFSv3 (mount type nfs), the _netdev option tells the system to wait to mount until the network is available. With a mount type of nfs4 the _netdev option is ignored as remote mounts are pulled by remote-fs.target.

However, Sander van Vugt recommends that all remote file systems that need to be mounted through /etc/fstab include the _netdev and the x-systemd.automount mount options.

The _netdev mount option ensures that the mount is delayed until the network is fully available. The x-systemd.automount option ensures optimal integration with systemd and will ensure that the mount is made a lot faster.

Test Kerberos, change to user alice to access the share:

# su - alice
Last login: Wed May 18 16:06:51 BST 2016 on pts/2
su: warning: cannot change directory to /home/alice: No such file or directory
id: cannot find name for group ID 1219400005
$ touch /mnt/nfs_secure/file
touch: cannot touch ‘/mnt/nfs_secure/file’: Permission denied
$ kinit
Password for [email protected]:
$ touch /mnt/nfs_secure/file
$ ls -l /mnt/nfs_secure/file
-rw-r--r--. 1 alice 1219400005 0 Jun  3 20:57 /mnt/nfs_secure/file
$ kdestroy
$ exit

Mount Non-Kerberised NFS Shares

Create mountpoints:

# mkdir /mnt/nfs_{pub,group}

Mount the public and group shares as these require no Kerberos configuration:

# mount -t nfs4 -o vers=4.2,sec=sys srv1.rhce.local:/srv/nfs_pub /mnt/nfs_pub 
# mount -t nfs4 -o vers=4.2,sec=sys srv1.rhce.local:/srv/nfs_group /mnt/nfs_group

Problems With Key Version Numbers

Sometimes the key version number (KVNO) used by the KDC and the service principal keys stored in /etc/krb5.keytab for services hosted on the system do not match. The KVNO can get out of synchronisation when a new set of keys are created on the KDC without updating the keytable file with the new keys.

To diagnose, list the keytab entries:

# klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   3 nfs/[email protected]
   3 nfs/[email protected]
   3 nfs/[email protected]
   3 nfs/[email protected]
   3 nfs/[email protected]
   3 nfs/[email protected]

Acquire Kerberos ticket by using the host key:

# kinit -k

Or use admin credentials:

# kinit admin

Check the KVNO that is used by the KDC:

# kvno nfs/srv2.rhce.local
nfs/[email protected]: kvno = 4

References

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/kerb-nfs.html

159 thoughts on “Setting up a Kerberised NFS Server on RHEL 7

  1. Good day

    I have been reading your site alongside the certdeport one after finishing the sander’s vedios and jangs book on RHEL 7…and i am really impressed…I will be writing my RHCE on the 23rd of Septemeber and i think i have managed to prepare very ,well thanks to all those sources and mainly this one for being specific.

    I have a few grey areas ,the kerberos server can be the redhat ipa or the standalone server installed from stand alone packages…

    my questions this config below when configuring the redhat cleint

    [libdefaults]
    default_realm = EXAMPLE.COM
    dns_lookup_realm = false
    dns_lookup_kdc = false
    rdns = false
    forwardable = yes
    ticket_lifetime = 24h
    
    [realms]
    EXAMPLE.COM = {
          kdc = ipaserver.example.com:88
          admin_server = ipaserver.example.com:749
          }
    [domain_realm]
    .example.com = EXAMPLE.COM
    example.com = EXAMPLE.COM

    the above is for the redhat IPA configuration
    and for a standalone configuration there are no ports 88 and port 749
    my question is ,how do you then know how to configure a client without knowing the ports if its a redhat IPA server or not ?

    • I’m glad you found it helpful. To configure a Kerberos client I use authconfig-tui. You need to know the REALM and have DNS configured to point to the Kerberos server. No manual configuration of krb5.conf is required.

    • Hello Martin,
      I’m also impressed from your detail level, however somewhat confused…(Regarding keytab).
      I’ve also seen the krb nfs lesson from Sander, ITFountains and more….
      Both,
      demonstrated they are generating the keytab only for the principal of nfs/server1.example.com and copied the SAME file to BOTH ,server1 AND server2 /etc/krb5.keytab location,
      While,
      you’ve generated two different keytabs, one for server1 and the other for server 2 and place the keytab file in each /etc folder…..
      So,
      how server2 will be able to mount server1:/exports if it does not have its principal service embedded in it keytab?
      Thanks in adv.

  2. @tomas so basically what you are saying is the kerberos server should provide DNS services for the client to pick it up? i have a KDC configured but with no DNS service running and it seems i have to configure the clients via authconfig-tui but i have to manually place the server name and ports when it asks for the kdc and the kdc_admin server

    • You need to ensure the hostname resolution is working. It can be a DNS server, or it can be static entries in /etc/hosts, up to you really.

    • I have also noticed that the /etc/sssd/sssd.conf file does get automatically created which makes the whole process complex

    • To use ipa-client-install, you need to know the Kerberos admin password, otherwise you cannot join the domain. I doubt such information will be supplied during the RHCE exam.

  3. Thank you @tomas , i had actually overlooked the fact that there is a password needed to join the IPA server and its mostly like not going to be supplied during the exam and that makes sense to use the other method…

    I luv sssd because its newer so i will stick to sssd.

  4. Interesting information regarding _netdev, I spoke to a Red Hat trainer who advised that it should not be required for SMB/NFS mounting, because when you mount the file system is either specified or automatically detected as either Samba or NFS, and as such it knows that these mounts should come up after the network. Despite this I think I’ll still add it in just to be safe!

  5. @tomas confirm something on this nfs share below

    /srv/nfs_group – a share for group collaboration,

    I am sure we need the devops groups on the client also ,with users in it to be able to create files and directories

  6. Omg! Because of your tutorial I’ve finally got my Kerberized NFS setup to work using CentOS 7.2. I could finally move on with the rest of my labs. Thank you very much!

  7. Hey Tomas, I cannot connect to a Kerberized share in my CentOS 7.2 environment. The service is registered and the key is downloaded from IPA server, LDAP clients are configured correctly. nfs-server service is started on both server and client, but on client, I cannot start nfs-client service:

    [root@server2 /]# systemctl start nfs-client
    Failed to start nfs-client.service: Unit nfs-client.service failed to load: No such file or directory.

    So, getting this:

    [root@server2 /]# mount -o sec=krb5p server1.example.com:/nfs-secureshare /nfssecure
    mount.nfs: an incorrect mount option was specified
    [root@server2 /]# mount -o sec=sys server1.example.com:/nfs-secureshare /nfssecure
    mount.nfs: access denied by server while mounting server1.example.com:/nfs-secureshare

    • The error is self-explanatory, the service file does not exist.

      You need a target, not a service. It’s explained in the blog post.

  8. Yes, thank, Tomas! I have started the target, now I have this:
    [root@server2 ~]# mount -o sec=krb5p -V 4.2 server1.example.com:/nfs-secureshare /nfssecure
    mount from util-linux 2.23.2 (libmount 2.23.0: selinux, debug, assert)
    [root@server2 ~]# echo $?
    0
    [root@server2 ~]# mount | grep nfssecure
    [root@server2 ~]#

    /var/log/messages shows nothing

  9. [root@server2 ~]# mount | grep nfs
    sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
    nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
    [root@server2 ~]# df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/mapper/centos-root 9.4G 1.1G 8.3G 12% /
    devtmpfs 483M 0 483M 0% /dev
    tmpfs 493M 0 493M 0% /dev/shm
    tmpfs 493M 6.7M 486M 2% /run
    tmpfs 493M 0 493M 0% /sys/fs/cgroup
    /dev/sda1 497M 166M 332M 34% /boot
    //server1.example.com/smb 9.4G 1.3G 8.1G 14% /smb
    tmpfs 99M 0 99M 0% /run/user/0

  10. -v option for “mount” did not show anything. And I dont know where to see the logs (/var/log/messages shows nothing) Here is strace output (you may not to post it all here, just have a look):

    [root@server2 ~]# strace -vts1024 mount -o sec=krb5p -V 4.2 server1.example.com:/nfs-secureshare /nfssecure

    • See “man mount” for the -V parameter that you use. You should figure it out.

      Did you read the blog post? There is an example on how to mount a Kerberised NFS share.

  11. yes, I have read it more carefully now:

    [root@server2 ~]# mount -t nfs4 -o sec=krb5p,vers=4.2 server1.example.com:/nfs-secureshare /nfssecure –verbose
    mount.nfs4: timeout set for Thu Nov 24 02:08:11 2016
    mount.nfs4: trying text-based options ‘sec=krb5p,vers=4.2,addr=10.39.92.67,clientaddr=10.39.92.77’
    mount.nfs4: mount(2): Permission denied
    mount.nfs4: access denied by server while mounting server1.example.com:/nfs-secureshare

    I have a few issues here: time is not synced between server and client (chronyd is configured to sync with IPA server, ports are opened and tcpdump shows traffic on port 123, timedatectl set-ntp true)

    [root@server2 ~]# systemctl status nfs-secure
    Б≈▐ rpc-gssd.service – RPC security service for NFS client and server
    Loaded: loaded (/usr/lib/systemd/system/rpc-gssd.service; static; vendor preset: disabled)
    Active: active (running) since Wed 2016-11-23 23:25:31 +07; 50min ago
    ERROR: gssd_refresh_krb5_machine_credential: no usable keytab entry found in keytab /etc/krb5.keytab for connection with host

    Key version is not 4:

    [root@server2 ~]# klist -k
    Keytab name: FILE:/etc/krb5.keytab
    KVNO Principal
    —- ————————————————————————–
    1 NFS/[email protected]
    1 NFS/[email protected]
    1 NFS/[email protected]
    1 NFS/[email protected]

    I will dig into it all….

    • “no usable keytab entry found in keytab /etc/krb5.keytab for connection with host” – what’s the KVNO that is used by the KDC (FreeIPA)?

  12. [root@ipa ~]# klist -k
    Keytab name: FILE:/etc/krb5.keytab
    KVNO Principal
    —- ————————————————————————–
    2 host/[email protected]
    2 host/[email protected]
    2 host/[email protected]
    2 host/[email protected]
    2 host/[email protected]
    2 host/[email protected]

    [root@server1example ~]# klist -k
    Keytab name: FILE:/etc/krb5.keytab
    KVNO Principal
    —- ————————————————————————–
    1 NFS/[email protected]
    1 NFS/[email protected]
    1 NFS/[email protected]
    1 NFS/[email protected]

    [root@server2 ~]# klist -k
    Keytab name: FILE:/etc/krb5.keytab
    KVNO Principal
    —- ————————————————————————–
    1 NFS/[email protected]
    1 NFS/[email protected]
    1 NFS/[email protected]
    1 NFS/[email protected]

  13. kadmin.local
    ktadd NFS/server1.example.com
    ktadd NFS/server2.example.com
    quit

    ipa-getkeytab -s ipa.example.com -p NFS/server1.example.com -k /root/server1.nfs.keytab
    ipa-getkeytab -s ipa.example.com -p NFS/server2.example.com -k /root/server2.nfs.keytab

    copy them to server1 and server2 to /etc/krb5.tab

    Is everything correct?

  14. [root@server2 ~]# mount -o vers=4.2,sec=krb5p server1.example.com:/nfs-secureshare /nfssecure –verbose
    mount.nfs: timeout set for Fri Nov 25 08:31:13 2016
    mount.nfs: trying text-based options ‘vers=4.2,sec=krb5p,addr=192.168.0.102,clientaddr=192.168.0.104’
    mount.nfs: mount(2): Invalid argument
    mount.nfs: an incorrect mount option was specified

    [root@server2 ~]# systemctl status nfs-secure
    ● rpc-gssd.service – RPC security service for NFS client and server
    Loaded: loaded (/usr/lib/systemd/system/rpc-gssd.service; static; vendor preset: disabled)
    Active: active (running) since Fri 2016-11-25 08:26:42 +07; 5s ago
    Process: 3139 ExecStart=/usr/sbin/rpc.gssd $GSSDARGS (code=exited, status=0/SUCCESS)
    Main PID: 3140 (rpc.gssd)
    CGroup: /system.slice/rpc-gssd.service
    └─3140 /usr/sbin/rpc.gssd

    [root@server1example ~]# systemctl status nfs-secure-server
    ● rpc-svcgssd.service – RPC security service for NFS server
    Loaded: loaded (/usr/lib/systemd/system/rpc-svcgssd.service; static; vendor preset: disabled)
    Active: inactive (dead)
    Condition: start condition failed at Fri 2016-11-25 08:28:00 +07; 24s ago
    none of the trigger conditions were met

    This is CentOS 7.2 and nfs-secure-servver should be autostarted by nfs-server, right?

  15. mount.nfs4: trying text-based options ‘sec=krb5p,vers=4.2,addr=192.168.0.102,clientaddr=192.168.0.104’
    mount.nfs4: mount(2): Invalid argument
    mount.nfs4: an incorrect mount option was specified

    [root@server1 ~]# cat /proc/fs/nfsd/versions
    -2 +3 +4 +4.1 +4.2

    rolled-back the whole environment and reinstalled the services and the keys. now getting this.
    knvo shows the same key versions.
    investigating further. from the IPA server installation instructions, I haven’t configured DNS zone transfer
    investigating this further…

  16. the following command can be used during troubleshooting to get the creds manually:

    # kinit -k -t /etc/krb5.keytab nfs/[email protected]

    I tried and got an error, something like cannot contact to KDC

  17. when I used authconfig-tui to configure LDAP client, I set “User DNS to find servers”, something like that
    as I can see, specifying ipa.example.com in /etc/hosts was not enough

    so, to sum-up:

    1. principals were not added to keytab
    2. nfs version was not specified in /etc/sysconfig/nfs
    3. DNS was not configured correctly

    Thank you very much!

    • There are two links mentioned for how to configure LDAP auth, one with SSSD and another with nslcd. I take if you haven’t checked them? :)

  18. in your sample exam a keytab for srv1 is already provided on FTP
    how will it be on the exam? both keytabs for srv1 and srv2 will be on FTP or I will make these files using ipa-getkeytab?

  19. if one keytab is already provided, where is the second one? if I kame it, the kvno versions can differ

  20. Tomas, I have Permission denied error accessing /srv/nfsssec share:

    [root@server1 ~]# ls -Zd /srv/nfssec/
    drwxr-xr-x. vince vince unconfined_u:object_r:nfs_t:s0 /srv/nfssec/

    [root@server1 ~]# exportfs -avr
    exporting 192.168.0.104/32:/srv/nfssec
    [root@server1 ~]# cat /etc/exports
    /srv/nfssec 192.168.0.104/32(rw,sync,sec=krb5p)

    [root@server2 ~]# su – vince
    Last login: Tue Dec 6 15:46:53 +07 2016 on tty1
    su: warning: cannot change directory to /home/vince: No such file or directory
    -sh-4.2$ cd /protected/
    -sh: cd: /protected/: Permission denied
    -sh-4.2$ exit
    logout
    [root@server2 ~]# mount | grep nfs
    sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
    nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
    server1.example.com:/srv/nfssec on /protected type nfs4 (rw,relatime,seclabel,vers=4.2,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=192.168.0.104,local_lock=none,addr=192.168.0.102)

    locally, the /srv/nfssec dir is accessible:

    [root@server1 ~]# su – vince
    Last login: Tue Dec 6 15:47:17 +07 2016 on tty1
    su: warning: cannot change directory to /home/vince: No such file or directory
    -sh-4.2$ cd /srv/nfssec/
    -sh-4.2$ touch vincelocal2
    -sh-4.2$ ll
    total 0
    -rw-r–r–. 1 nfsnobody nfsnobody 0 Dec 3 10:16 dffdg
    -rw-r–r–. 1 nfsnobody nfsnobody 0 Dec 5 19:52 rootfile
    -rw-rw-r–. 1 vince vince 0 Dec 6 15:47 vincelocal
    -rw-rw-r–. 1 vince vince 0 Dec 6 15:55 vincelocal2

    Will check the logs…

  21. Hey Tomas, I am stuck again at this. Please advice:

    mount.nfs4: timeout set for Sat Dec 31 01:29:26 2016
    mount.nfs4: trying text-based options ‘vers=4.2,sec=krb5p,addr=192.168.0.102,clientaddr=192.168.0.104’
    mount.nfs4: mount(2): Operation not permitted
    mount.nfs4: Operation not permitted

    log:
    Dec 31 05:51:04 system2 rpc.gssd[4070]: WARNING: Failed to create krb5 context for user with uid 0 for server [email protected]
    Dec 31 05:51:04 system2 rpc.gssd[4070]: WARNING: Failed to create machine krb5 context with credentials cache FILE:/tmp/krb5ccmachine_EXAMPLE.COM for server system1.example.com
    Dec 31 05:51:04 system2 rpc.gssd[4070]: WARNING: Failed to create machine krb5 context with any credentials cache for server system1.example.com

  22. Hi, Tomas
    Thanks for the sharing.
    I got some problems when trying to start nfs-secure-server
    [root@ipa~]# systemctl status nfs-secure-server
    ● rpc-svcgssd.service – RPC security service for NFS server
    Loaded: loaded (/usr/lib/systemd/system/rpc-svcgssd.service; static; vendor preset: disabled)
    Active: inactive (dead)
    Condition: start condition failed at Fri 2016-11-25 08:28:00 +07; 24s ago
    none of the trigger conditions were met

    I’ve tried several times…. it failed again and again either on ipa server or srv1 ……
    I followed the steps on you posting and everything seems fine until I tried to active nfs-secure-server on both ipa and srv1..
    On ipa server, why my nfs-secure-server status is “static”, it won’t work even when I try to enable it , right?
    What could be the common cause or do you have any suggestion?
    Thanks for the help.

    • On RHEL 7.1 and RHEL 7.2, the nfs-secure-server service becomes a static service and cannot be enabled. It is started by nfs-server service automatically assuming the file /etc/krb5.keytab is present on the system.

    • This seems to be a bug in the status output. I’m seeing the exact same status output on my server however I can mount the share using nfs v4.2 and kerberos authentication.

      mount -o sec=krb5p,nfsvers=4.2,rw srv1.rhce.local:/srv/nfs_secure /nfssecure

      You should be able to then manage the files in the directory using the alice account.

    • It seems to be enough, are you sure that your issue isn’t somewhere else? I tested instructions numerous times, they work. The nfs-secure service does start on reboot on the client, I get no issues with that.

      I don’t have to enable the nfs-server service on the NFS client machine because I don’t need an NFS server on it.

  23. it is enough, but it is not stable as hell: the Kerberized share can have “Failed to create machine krb5 context” error, then it automatically fixes, out of the blue. Then “operation not permitted” error can appear, and be fixed the same way.

    • Although I haven’t encountered the issue you described myself, I agree that there may be bugs in software. If you think you found a bug, you may have more luck getting in touch with Red Hat and perhaps filling a bug report.

  24. I have bad experience with thier support: they redirect the ticket to another team and then colse it in a week without any answer)

  25. Hello friends,

    While I was doing a sample exam I have noticed that the nfs-secure.service required a restart in order to mount the kerberized nfs.
    Please note that this is RHEL 7.0 DVD repo and as I have not met this in higher versions – it’s probably fixed.

    HOW TO DEBUG: As Tomas has mentioned enable “-vvv” flag for
    RPCGSSDARGS=”-vvv”
    RPCSVCGSSDARGS=”-vvv”
    in both CLIENT and SERVER “/etc/sysconfig/nfs”

    QUICK FIX: check the hostname is FQDN (static section) like this:
    ####################
    [root@server2 ~]# hostnamectl
    Static hostname: server2.example.com
    ####################

    The reason , which I found 30 min later is the following:
    1. I’m using hostnames like “ipa” , “server1” and “server2″ , while dns resolution still works (until you dig the short name) :
    #############################
    [root@server2 ~]# hostnamectl
    Static hostname: server2
    #############################
    Of course dns resolves with short and FQDN , but this caused the issue that was logged as this:
    #############################
    [root@server2 ~]# systemctl status nfs-secure
    nfs-secure.service – Secure NFS
    Loaded: loaded (/etc/systemd/system/nfs-secure.service; enabled)
    Active: active (running) since сб 2017-04-15 11:28:48 EEST; 25s ago
    Process: 654 ExecStart=/usr/sbin/rpc.gssd $RPCGSSDARGS (code=exited, status=0/SUCCESS)
    Main PID: 662 (rpc.gssd)
    CGroup: /system.slice/nfs-secure.service
    └─662 /usr/sbin/rpc.gssd -vvv

    apr 15 11:28:48 server2 systemd[1]: Starting Secure NFS…
    apr 15 11:28:48 server2 rpc.gssd[662]: beginning poll
    apr 15 11:28:48 server2 systemd[1]: Started Secure NFS.
    apr 15 11:28:55 server2 rpc.gssd[662]: handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt0)
    apr 15 11:28:55 server2 rpc.gssd[662]: handle_gssd_upcall: ‘mech=krb5 uid=0 service=* enctypes=18,17,16,23,3,1,2 ‘
    apr 15 11:28:55 server2 rpc.gssd[1405]: handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt0)
    apr 15 11:28:55 server2 rpc.gssd[1405]: process_krb5_upcall: service is ‘*’
    apr 15 11:28:55 server2 rpc.gssd[1405]: Name or service not known while getting full hostname for ‘server1.example.com’
    apr 15 11:28:55 server2 rpc.gssd[662]: handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt0)
    apr 15 11:28:55 server2 rpc.gssd[662]: handle_gssd_upcall: ‘mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 ‘
    apr 15 11:28:55 server2 rpc.gssd[1406]: handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt0)
    apr 15 11:28:55 server2 rpc.gssd[1406]: process_krb5_upcall: service is ”
    apr 15 11:28:55 server2 rpc.gssd[662]: Closing ‘gssd’ pipe for /var/lib/nfs/rpc_pipefs/nfs/clnt0
    apr 15 11:28:55 server2 rpc.gssd[662]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt1
    apr 15 11:28:55 server2 rpc.gssd[662]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt0
    [root@server2 ~]# mount -a
    mount.nfs4: access denied by server while mounting server1.example.com:/srv/secure

    [root@server2 ~]# systemctl status nfs-secure
    nfs-secure.service – Secure NFS
    Loaded: loaded (/etc/systemd/system/nfs-secure.service; enabled)
    Active: active (running) since сб 2017-04-15 11:30:04 EEST; 6s ago
    Process: 2515 ExecStart=/usr/sbin/rpc.gssd $RPCGSSDARGS (code=exited, status=0/SUCCESS)
    Main PID: 2516 (rpc.gssd)
    CGroup: /system.slice/nfs-secure.service
    └─2516 /usr/sbin/rpc.gssd -vvv

    apr 15 11:30:06 server2 rpc.gssd[2536]: using FILE:/tmp/krb5ccmachine_EXAMPLE.COM as credentials cache for machine creds
    apr 15 11:30:06 server2 rpc.gssd[2536]: using environment variable to select krb5 ccache FILE:/tmp/krb5ccmachine_EXAMPLE.COM
    apr 15 11:30:06 server2 rpc.gssd[2536]: creating tcp client for server server1.example.com
    apr 15 11:30:06 server2 rpc.gssd[2536]: DEBUG: port already set to 2049
    apr 15 11:30:06 server2 rpc.gssd[2536]: creating context with server [email protected]
    apr 15 11:30:06 server2 rpc.gssd[2536]: DEBUG: serialize_krb5_ctx: lucid version!
    apr 15 11:30:06 server2 rpc.gssd[2536]: prepare_krb5_rfc4121_buffer: protocol 1
    apr 15 11:30:06 server2 rpc.gssd[2536]: prepare_krb5_rfc4121_buffer: serializing key with enctype 18 and size 32
    apr 15 11:30:06 server2 rpc.gssd[2516]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt8
    apr 15 11:30:06 server2 rpc.gssd[2516]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt7
    #############################

  26. hi tomas,
    for nfs client with user, when the system reboot the user must must execute the kinit command again to have access in the shared dir, is this behaviour normal ? since the exam require all the configuration must persist after reboot
    can we make it persistent ?

  27. Hi Tomas,
    Thanks for your great articles.
    Just wanted to contribute with following info.
    Mounting a kerberized share on stock RHEL 7.2 (both server and client) results following output:
    mount.nfs4: access denied by server while mounting srv1.rhce.local:/srv/nfs_secure

    In srv1.rhce.local:/var/log/messages you’ll see:
    sqlite_insert_client: insert statement prepare failed: table clients has 2 columns but 3 values were supplied

    This is a known bug access.redhat.com/solutions/2065873 and has been fixed in:
    – nfs-utils-1.3.0-0.33.el7_3.x86_64.rpm
    – libtirpc-0.2.4-0.8.el7_3.x86_64.rpm
    Of course one need an active subscription to download them separately, or extract them from RHEL7.3 ISO.
    HTH

    • Thanks for your feedback, much appreciated.

      You can get a no-cost RHEL Developer Subscription by registering with the Red Hat Developers Program. It includes RHEL 7, High Availability Add-On etc as well as provides access to Red Hat knowledgebase, forums and updates. I found it useful when studying for RHCE exam.

  28. Hello,
    Thank you very much for this well-explained article. I have one issue and need to know if there is any workaround of it.
    I am using CentOS 7.0 and have been able to set-up complete Kerberised NFS environment using ipa-client without any problem following this article as well as Sander’s videos. Sander recommend ipa-client procedure which assume that IPA admin credentials will provided in the exam. However, this is not sure. Therefore, I have tried using authconfig-tui procedure. In this procedure, there is no problem in nfs-server side set-up. However, on nfs-client side, nfs-secure failed to start in its first attempt and gives below error.

    rpc.gssd[17099]: segfault at 0 ip 00007f9812d63f0a sp 00007fffabe086e8 error 4 in libc-2.17.so[7f9812c32000+1b6000]
    rpc.gssd[747]: WARNING: forked child was killed with signal 11

    However, if I restart nfs-secure after that fail, it works fine afterwards. When I include the mount in /etc/fstab, nfs-secure crash during boot time and give above error message and if I don’t include in /etc/fstab, this error comes at first mount time. After that, if I restart nfs-secure service, everything works fine afterwards.

    Keeping everything same on nfs-server side, if I use ipa-client-install procedure on nfs-client side, again everything goes perfect but I have to have IPA admin credentials.

    This was a bug ( https://bugzilla.redhat.com/show_bug.cgi?id=1032379) which has been resolved in the later release. However in exam, if we’ll have 7.0 release and will not have IPA admin credentials, then what would be the workaround?

  29. Hi Tomas,
    Maybe you can point me to my mistake with NFS? I’ve encountered an error that an LDAP user cannot access mounted krb5p share on the system2. It shows permission denied. And on system1 in /var/log/messages, nslcd service reported something like that the user is not found in validusers for nfs/…..
    this user was an owner of the share on system1 and permissions were 777. the error was the same with disabled firewalld/selinux.

  30. Thank you very much for your quick reply. I also believe that such bugs will at least be fixed in the version of RHEL provided in the exam. I spent one whole day to figure out the exact problem which I summarized in my previous message but this bug has led me a new and very useful learning and got the solution this morning.

    Actually, the IPA admin credentials are only required by ipa-client-install to generate the kytab file and if we already have keytab file, there is an option in ipa-client-install to give that keytab file i.e

    -k KEYTAB, –keytab=KEYTAB path to backed up keytab from previous enrollment

    After giving this option, ipa-client-install don’t ask admin credentials.
    Therefore, if you have keytab file, use below command instead of authconfig-tui
    ipa-client-install -k KEYTABFILE

    I have tested on the same (fresh) RHEL7.0 system without any update or patch fix and it works and also have no issue by putting the two (public and secure krb5p) mounts in /etc/fstab file.

    Conclusion:
    No need to use authconfig-tui utility at all.
    If you have keytab file provided in the exam, use “ipa-client-install -k KEYTABFILE”.
    Otherwise, you should have IPA admin credentials and use ipa-client-install without any option and provide IPA admin credentials when asked by ipa-client-install.

    I have not seen this info anywhere on the internet and if anyone test this on his/her system, please let us know if it works or not.

    • Thanks for your valuable input. If you have a keytab file, it should be sufficient to copy it to /etc/krb5.keytab providing that remote authentication against a Kerberos server is configured.

  31. Hi i am setting up the LAB but when i need to get the keytabs is give me this message:

    [root@server1 ~]# ipa-getkeytab -s ipa.rhce.local -p nfs/server1.rhce.local -k /etc/krb5.keytab
    Unable to initialize STARTTLS session
    Failed to bind to server!
    Retrying with pre-4.0 keytab retrieval method…
    Unable to initialize STARTTLS session
    Failed to bind to server!
    Failed to get keytab
    [root@server1 ~]

    • [root@server1 ~]# cat /etc/sssd/sssd.conf
      [domain/default]

      autofs_provider = ldap
      cache_credentials = True
      ldap_search_base = dc=rhce,dc=local
      id_provider = ldap
      auth_provider = krb5
      chpass_provider = krb5
      ldap_uri = ldap://ipa.rhce.local
      ldap_id_use_start_tls = True
      ldap_tls_cacertdir = /etc/openldap/cacerts
      ldap_tls_reqcert = never
      krb5_realm = RHCE.LOCAL
      krb5_server = ipa.rhce.local
      krb5_store_password_if_offline = True
      [sssd]
      services = nss, pam, autofs

      domains = default
      [nss]
      homedir_substring = /home

      [pam]

      [sudo]

    • Which version of the FreeIPA server?

      Can you capture the exit status code of the ipa-getkeytab command and post it here?

      Is the LDAPS port open on the FreeIPA server and is there a service that listens on it?

      Did you configure the FreeIPA server as per these instructions:

      https://www.lisenet.com/2016/freeipa-server-on-rhel-7-centos-7/

      Make sure that the ipa-server version that you have matches the ones the blog post was written for, as otherwise it may be a configuration issue that’s relevant to your version.

  32. [root@ipa ~]# rpm -qa | grep ipa
    ipa-common-4.5.0-21.el7.centos.1.2.noarch
    ipa-server-dns-4.5.0-21.el7.centos.1.2.noarch
    ipa-client-common-4.5.0-21.el7.centos.1.2.noarch
    python2-ipalib-4.5.0-21.el7.centos.1.2.noarch
    sssd-ipa-1.15.2-50.el7_4.2.x86_64
    python-iniparse-0.4-9.el7.noarch
    python-libipa_hbac-1.15.2-50.el7_4.2.x86_64
    python-ipaddress-1.0.16-2.el7.noarch
    python2-ipaclient-4.5.0-21.el7.centos.1.2.noarch
    ipa-client-4.5.0-21.el7.centos.1.2.x86_64
    python2-ipaserver-4.5.0-21.el7.centos.1.2.noarch
    ipa-server-4.5.0-21.el7.centos.1.2.x86_64
    ipa-server-common-4.5.0-21.el7.centos.1.2.noarch
    [root@ipa ~]#
    [root@server1 cacerts]# ipa-getkeytab -s ipa.rhce.local -p NFS/server1.rhce.local -k /etc/krb5.keytab
    Unable to initialize STARTTLS session
    Failed to bind to server!
    Retrying with pre-4.0 keytab retrieval method…
    Unable to initialize STARTTLS session
    Failed to bind to server!
    Failed to get keytab
    [root@server1 cacerts]# echo $?
    9
    [root@server1 cacerts]#

    i didnt invoke the ipa-client-install

    • I didn’t use ipa-server 4.5.0. It may be a configuration issue that is relevant to this particular version. Let us know if you get this resolved as it may help others.

  33. Hi i redone the client install for the server1 and server2.

    490 ipa-client-install –mkhomedir –enable-dns-updates –force-ntpd
    491 cd /etc/openldap/cacerts/
    492 ls
    493 elinks ftp://ipa.rhce.local/pub and got the ca.crt to /et/openldap/cacerts/
    494 rm cacert.p12
    495 su – alice
    496 kinit admin
    497 klist
    498 klist -l
    499 ipa-getkeytab -s labipa.example.com -p nfs/
    500 ipa-getkeytab -s ipa.rhce.local -p nfs/server1.rhce.local -k /etc/krb5.keytab
    and it got it.

    the issue now is that after exporting the shares as:

    [root@server1 ~]# cat /etc/exports
    /srv/nfs_pub 10.1.1.0/24(sec=sys,rw,sync,root_squash,all_squash)
    /srv/nfs_group 10.1.1.0/24(sec=sys,rw,sync,root_squash)
    /srv/nfs_secure server2.rhce.local(sec=krb5p,rw,sync)

    when i mount it on the server2 like:
    [root@server2 ~]# mount -t nfs4 -o vers=4.2,sec=krb5p server1.rhce.local:/srv/nfs_secure /mnt/nfs_secure/
    mount.nfs4: mounting server1.rhce.local:/srv/nfs_secure failed, reason given by server: No such file or directory
    [root@server2 ~]#

    i am lost now the OS is CentOS 7.3

  34. [root@server2 ~]# mount -o sec=krb5p server1.rhce.local:/srv/nfs_secure /mnt/nfs_secure
    mount.nfs: access denied by server while mounting server1.rhce.local:/srv/nfs_secure
    [root@server2 ~]# mount -t nfs -o sec=krb5p server1.rhce.local:/srv/nfs_secure /mnt/nfs_secure
    mount.nfs: access denied by server while mounting server1.rhce.local:/srv/nfs_secure
    [root@server2 ~]# mount -t nfs4 -o sec=krb5p server1.rhce.local:/srv/nfs_secure /mnt/nfs_secure
    mount.nfs4: mounting server1.rhce.local:/srv/nfs_secure failed, reason given by server: No such file or directory
    [root@server2 ~]# mount -t nfs4 -o v4.2 sec=krb5p server1.rhce.local:/srv/nfs_secure /mnt/nfs_secure

    • Does the NFS mount point exist? I’d suggest to enable verbose logging (-vvv) for the RPCGSSDARGS, then try mounting with the verbose mount option and see what gets recorded.

  35. SERVER1:
    [root@server1 ~]# ls -lZHtas /srv
    total 0
    dr-xr-xr-x. root root system_u:object_r:root_t:s0 ..
    drwxr-xr-x. root root system_u:object_r:var_t:s0 .
    drwxrws—. root devops unconfined_u:object_r:nfs_t:s0 nfs_group
    drwxr-xr-x. nfsnobody nfsnobody unconfined_u:object_r:public_content_rw_t:s0 nfs_pub
    drwxr-xr-x. alice root unconfined_u:object_r:nfs_t:s0 nfs_secure
    [root@server1 ~]#

    SERVER2:
    [root@server2 ~]# ls -lartshZ /mnt/
    total 0
    drwxr-xr-x. root root unconfined_u:object_r:mnt_t:s0 nfs_secure
    drwxr-xr-x. root root system_u:object_r:mnt_t:s0 .
    dr-xr-xr-x. root root system_u:object_r:root_t:s0 ..
    [root@server2 ~]#

    tried: mount -t nfs4 -o vers=4.2,sec=krb5p server1.rhce.local:/srv/nfs_secure /mnt/nfs_secure

    Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: handle_gssd_upcall: ‘mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 ‘ (nfs/clnte)
    Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: krb5_use_machine_creds: uid 0 tgtname (null)
    Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: Full hostname for ‘server1.rhce.local’ is ‘server1.rhce.local’
    Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: Full hostname for ‘server2.rhce.local’ is ‘server2.rhce.local’
    Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: No key table entry found for [email protected] while getting keytab entry for ‘[email protected]
    Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: No key table entry found for [email protected] while getting keytab entry for ‘[email protected]
    Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: No key table entry found for root/[email protected] while getting keytab entry for ‘root/[email protected]
    Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: Success getting keytab entry for ‘nfs/[email protected]
    Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: INFO: Credentials in CC ‘FILE:/tmp/krb5ccmachine_RHCE.LOCAL’ are good until 1508368319
    Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: INFO: Credentials in CC ‘FILE:/tmp/krb5ccmachine_RHCE.LOCAL’ are good until 1508368319
    Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: creating tcp client for server server1.rhce.local
    Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: creating context with server [email protected]
    Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: doing downcall: lifetime_rec=71521 [email protected]
    [root@server2 ~]# date
    Tue Oct 17 19:28:33 EDT 2017
    [root@server2 ~]#
    now i am clueless

    Tim

  36. i nailed it down:

    [root@server2 ~]# mount -t nfs4 -o vers=4.2,sec=krb5p server1.rhce.local:/srv/nfs_secure /mnt/nfs_secure
    mount.nfs4: mounting server1.rhce.local:/srv/nfs_secure failed, reason given by server: No such file or directory
    [root@server2 ~]# man mount.nfs4
    [root@server2 ~]# mount -t nfs4 -v -o vers=4.2,sec=krb5p server1.rhce.local:/srv/nfs_secure /mnt/nfs_secure
    mount.nfs4: timeout set for Tue Oct 17 19:49:46 2017
    mount.nfs4: trying text-based options ‘sec=krb5p,vers=4.1,addr=192.168.4.101,clientaddr=192.168.4.102’
    mount.nfs4: mount(2): No such file or directory
    mount.nfs4: mounting server1.rhce.local:/srv/nfs_secure failed, reason given by server: No such file or directory

    i had setup the IPA with the reverse DNS entries for both my networks 192.168.4.0/24 and the 10.1.1.0 but i exported it only for server2.rhce.local

    and the reverse DNS that its doing is 192.168.4.101 instead of 10.1.1.101 as here:

    [root@server2 ~]# mount -t nfs4 -v -o vers=4.2,sec=krb5p 10.1.1.101:/srv/nfs_secure /mnt/nfs_secure
    mount.nfs4: timeout set for Tue Oct 17 19:50:19 2017
    mount.nfs4: trying text-based options ‘sec=krb5p,vers=4.1,addr=10.1.1.101,clientaddr=10.1.1.102’
    [root@server2 ~]#

    [root@server2 ~]# findmnt | grep secure
    └─/mnt/nfs_secure 10.1.1.101:/srv/nfs_secure nfs4 rw,relatime,vers=4.1,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=10.1.1.102,local_lock=none,addr=10.1.1.101
    [root@server2 ~]#

    i tracked the issue that when i did the IPA from start i had put the internet eth in manual and when installed IPA it regiestered it in DNS.
    so the total point was to use the ca.crt

    Tim

    • Well done! Exactly the reason I suggested to try to mount with the verbose mount option, as now you can see the clientaddr.

  37. I know this question has already been asked on this page, but i guess i need help on this topic.
    I am constantly getting these errors:

    # showmount -e srv01.example.local
    Export list for srv01.example.local:
    /srv/nfs_secure *
    /srv/nfs_group *
    /srv/nfs_pub *

    Error while mounting: ( mouting a root)

    # mount -t nfs4 -o vers=4.2,sec=krb5p srv01.example.local:/srv/nfs_secure /mnt/
    mount.nfs4: access denied by server while mounting srv01.example.local:/srv/nfs_secure

    #tail -f /var/log/messages
    Oct 18 18:27:21 localhost rpc.gssd[3587]: ERROR: No credentials found for connection to server srv01.example.local
    Oct 18 18:27:21 localhost rpc.gssd[3587]: ERROR: gssd_refresh_krb5_machine_credential: no usable keytab entry found in keytab /etc/krb5.keytab for connection with host srv01.example.local
    Oct 18 18:27:21 localhost rpc.gssd[3587]: ERROR: No credentials found for connection to server srv01.example.local
    Oct 18 18:27:21 localhost rpc.gssd[3587]: ERROR: gssd_refresh_krb5_machine_credential: no usable keytab entry found in keytab /etc/krb5.keytab for connection with host srv01.example.local
    Oct 18 18:27:21 localhost rpc.gssd[3587]: ERROR: No credentials found for connection to server srv01.example.local

    #systemctl status nfs-secure -l
    Oct 18 18:28:58 srv02.example.local rpc.gssd[3587]: ERROR: gssd_refresh_krb5_machine_credential: no usable keytab entry found in keytab /etc/krb5.keytab for connection with host srv01.example.local
    Oct 18 18:28:58 srv02.example.local rpc.gssd[3587]: ERROR: No credentials found for connection to server srv01.example.local4

    From srv01.example.local
    # klist -k
    Keytab name: FILE:/etc/krb5.keytab
    KVNO Principal
    —- ————————————————————————–
    2 nfs/[email protected]
    2 nfs/[email protected]

    From srv01.example.local
    #klist -k
    Keytab name: FILE:/etc/krb5.keytab
    KVNO Principal
    —- ————————————————————————–
    2 nfs/[email protected]
    2 nfs/[email protected]

    Both the servers have NOT joined ipa domain. could this be the issue ?
    I have tried 3 times and getting the same error on all machines.

    if you have a youtube video for this topic, that will be helpfull for me.

    • I don’t do youtube videos I’m afraid.

      You may want to check the answers that have been provided on this page as the solution may already be there.

    • join it. with ipa-client i think. check on the web gui on ipa that the hosts are registered. you have to add the hosts to the IPA as on the guide here.

      Tim.

  38. I have setup this ipa and nfs things from scratch. using centos 7.0.
    authentication works fine . when I copy the keytab files, i receive following error.

    From server2,example.local
    [root@server2 ~]# klist -k
    Keytab name: FILE:/etc/krb5.keytab
    klist: Unsupported key table format version number while starting keytab scan

    From server ( ipa.example.local )
    [root@ipa ~]# klist -k
    Keytab name: FILE:/etc/krb5.keytab
    KVNO Principal
    —- ————————————————————————–
    2 host/[email protected]
    2 host/[email protected]
    2 host/[email protected]
    2 host/[email protected]
    1 nfs/[email protected]
    1 nfs/[email protected]
    1 nfs/[email protected]
    1 nfs/[email protected]
    1 nfs/[email protected]
    1 nfs/[email protected]
    1 nfs/[email protected]
    1 nfs/[email protected]
    [root@ipa ~]# kvno nfs/server2.example.local
    nfs/[email protected]: kvno = 3

    how to solve this error ???
    thanks

  39. In the description above we first have to change the file /srv/nfs_secure to alice on srv1:
    chown alice /srv/nfs_secure
    How is that possible if we don’t have first made the Kerberized connection with the ipa server?

    Maybe i’m confused….or I don’t see it anymore!!!

    Properly you have to do first on ipa-server:
    kinit admin
    ipa user-add alice
    ipa passwd alice
    After that on srv1 all commands for “Kerberos Keytab File”.
    Then you are able to chown the /srv/nfs_secure file to alice and continue with “Configure NFS Exports”.

    Thx for all Dennis

    • If you read the article again, you will notice the following line under “The Lab” section:

      “Servers srv1 and srv2 have been configured for LDAP and Kerberos authentication using sssd”.

  40. Hi,
    I’ve followed all the steps and can’t start nfs-secure-server:

    [root@server1 log]# systemctl -l status nfs-secure-server
    ● rpc-svcgssd.service – RPC security service for NFS server
    Loaded: loaded (/usr/lib/systemd/system/rpc-svcgssd.service; static; vendor preset: disabled)
    Active: inactive (dead)
    Condition: start condition failed at Fri 2017-11-10 11:36:20 EST; 8min ago
    none of the trigger conditions were met
    Nov 10 11:18:21 server1.example.com systemd[1]: Started RPC security service for NFS server.
    Nov 10 11:25:21 server1.example.com systemd[1]: Started RPC security service for NFS server.
    Nov 10 11:25:25 server1.example.com systemd[1]: Started RPC security service for NFS server.
    Nov 10 11:27:33 server1.example.com systemd[1]: Started RPC security service for NFS server.
    Nov 10 11:36:20 server1.example.com systemd[1]: Started RPC security service for NFS server.

    I’ve checked the files, permissions, SELinux settings, firewall settings, logs and also googled it and still can’t figure out what’s the problem. What logs can I check?

    • Which version of RHEL are you using? The output on your OS indicates that nfs-secure-server is a static service and therefore cannot be enabled.

  41. Hi, it’s a centos 7.0. I downloaded the IPA server from Sander van Gundt’s website. Should I setup my own?

  42. Hi Tomas, I’m having problem to mount sub directory in client even though shared from the server.Permission is correct.It says no /data/mount fount. Please Here’s the line from /etc/exports:
    /data 192.168.10.200/24(rw,sync,no_root_squash)

  43. My fstab entry is as below:
    server1:/data /mnt/share nfs defaults 0 0

    I tried this way too:
    192.168.10.201:/data/mount /mnt/share nfs defaults 0 0

    • What happens when you run the following command, where server1 is your NFS sever?

      # mount -v -t nfs4 server1:/data /mnt/share

      Change nfs4 to nfs if you use NFSv3.

  44. Hello,
    it will be better if:
    1) in the line “groupadd devops” we define the free GID number, for example “groupadd -g 9999 devops”
    2) on an nfs client we should add a group with the same GID (“groupadd -g 9999 devops”) and add some users there: “usermod -G devops vince”
    In other case, you’ll probably get “Permission denied” when you try to test access to this mount, because gids of the devops group may be different on a server and clients.

  45. Nfs-secure-server is giving me some trouble only with SELinux enabled on NFS Kerberized server. I’m working on a RHEL 7.0.

    If I disabled SELinux everything work smooth.

    Anyone have already got into this? Can be a 7.0 issue?

    [root@server1 ~]# systemctl restart nfs-secure-server.service 
    Job for nfs-secure-server.service failed. See 'systemctl status nfs-secure-server.service' and 'journalctl -xn' for details.
    
    [root@server1 ~]# journalctl -xn
    Jun 02 11:30:51 server1.teo.lab python[2948]: SELinux is preventing /usr/sbin/rpc.svcgssd from read access on the file                                             
                                                  *****  Plugin catchall (100. confidence) suggests   **************************
                                                  
                                                  If you believe that rpc.svcgssd should be allowed read access on the  file by default.
                                                  Then you should report this as a bug.
                                                  You can generate a local policy module to allow this access.
                                                  Do
                                                  allow this access for now by executing:
                                                  # grep rpc.svcgssd /var/log/audit/audit.log | audit2allow -M mypol
                                                  # semodule -i mypol.pp
    • This was the line:
      Jun 02 11:30:51 server1.teo.lab python[2948]: SELinux is preventing /usr/sbin/rpc.svcgssd from read access on the file

      I solved it with this command on both server:
      # restorecon -Rv /etc/krb5.keytab

      Thanks

  46. Hello,

    I have been going over this for some time and I am stuck because I get an error from the nfs/client that says mount.nfs: access denied by serer while mounting srv1.rhce.local:/srv/nfs_secure. I get this when running the following command from the nfs/client as root.
    mount -t nfs4 -o vers=4.2,sec=krb5 srv1.rhce.local:/srv/nfs_secure /mnt/nfs_secure

    RHEL 7.4 server for IPA and another server for NFS
    RHEL 7.4 workstation for nfs/client

    All other steps are completed successfully.

    Thanks

  47. I’ve followed your lab installation (at least I believe I did), and I’m facing an issue with Kerberized NFS that I can’t solve.

    As suggested already, I added -vvv to both client and servers. This is the client behaviour :

    [root@srv2 log]#  mount -t nfs4 -v -o sec=krb5p srv1.rhce.local:/srv/secureshare /srv/securenfs
    mount.nfs4: timeout set for Sat Jul 28 18:10:10 2018
    mount.nfs4: trying text-based options 'sec=krb5p,vers=4.1,addr=192.168.122.71,clientaddr=192.168.122.72'
    mount.nfs4: mount(2): Permission denied
    mount.nfs4: access denied by server while mounting srv1.rhce.local:/srv/secureshare
    [root@srv2 log]# systemctl status nfs-secure
     rpc-gssd.service - RPC security service for NFS client and server
       Loaded: loaded (/usr/lib/systemd/system/rpc-gssd.service; static; vendor preset: disabled)
       Active: active (running) since Sat 2018-07-28 17:59:45 CEST; 1min 35s ago
      Process: 691 ExecStart=/usr/sbin/rpc.gssd $GSSDARGS (code=exited, status=0/SUCCESS)
     Main PID: 698 (rpc.gssd)
       CGroup: /system.slice/rpc-gssd.service
               └─698 /usr/sbin/rpc.gssd -vvv
    
    Jul 28 18:00:22 srv2.RHCE.LOCAL rpc.gssd[698]: No key table entry found for root/[email protected] while getting keytab entry for 'root/[email protected]'
    Jul 28 18:00:22 srv2.RHCE.LOCAL rpc.gssd[698]: Success getting keytab entry for 'nfs/[email protected]'
    Jul 28 18:00:22 srv2.RHCE.LOCAL rpc.gssd[698]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_RHCE.LOCAL' are good until 1532879989
    Jul 28 18:00:22 srv2.RHCE.LOCAL rpc.gssd[698]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_RHCE.LOCAL' are good until 1532879989
    Jul 28 18:00:22 srv2.RHCE.LOCAL rpc.gssd[698]: creating tcp client for server srv1
    Jul 28 18:00:22 srv2.RHCE.LOCAL rpc.gssd[698]: creating context with server nfs@srv1
    Jul 28 18:00:27 srv2.RHCE.LOCAL rpc.gssd[698]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@srv1
    Jul 28 18:00:27 srv2.RHCE.LOCAL rpc.gssd[698]: WARNING: Failed to create machine krb5context with cred cache FILE:/tmp/krb5ccmachine_RHCE.LOCAL for server srv1
    Jul 28 18:00:27 srv2.RHCE.LOCAL rpc.gssd[698]: WARNING: Failed to create machinekrb5 context with any credentialscache for server srv1
    Jul 28 18:00:27 srv2.RHCE.LOCAL rpc.gssd[698]: doing error downcall

    /var/log/messages extract :

    Jul 28 18:10:15 srv2 rpc.gssd[698]: #012handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 ' (nfs/clnt0)
    Jul 28 18:10:15 srv2 rpc.gssd[698]: krb5_use_machine_creds: uid 0 tgtname (null)
    Jul 28 18:10:15 srv2 rpc.gssd[698]: Full hostname for 'srv1' is 'srv1'
    Jul 28 18:10:15 srv2 rpc.gssd[698]: Full hostname for 'srv2.rhce.local' is 'srv2.rhce.local'
    Jul 28 18:10:15 srv2 rpc.gssd[698]: No key table entry found for [email protected] while getting keytab entry for '[email protected]'
    Jul 28 18:10:15 srv2 rpc.gssd[698]: No key table entry found for [email protected] while getting keytab entry for '[email protected]'
    Jul 28 18:10:15 srv2 rpc.gssd[698]: No key table entry found for root/[email protected] while getting keytab entry for 'root/[email protected]'
    Jul 28 18:10:15 srv2 rpc.gssd[698]: Success getting keytab entry for 'nfs/[email protected]'
    Jul 28 18:10:15 srv2 rpc.gssd[698]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_RHCE.LOCAL' are good until 1532879989
    Jul 28 18:10:15 srv2 rpc.gssd[698]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_RHCE.LOCAL' are good until 1532879989
    Jul 28 18:10:15 srv2 rpc.gssd[698]: creating tcp client for server srv1
    Jul 28 18:10:15 srv2 rpc.gssd[698]: creating context with server nfs@srv1
    Jul 28 18:10:20 srv2 rpc.gssd[698]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@srv1
    Jul 28 18:10:20 srv2 rpc.gssd[698]: WARNING: Failed to create machine krb5context with cred cache FILE:/tmp/krb5ccmachine_RHCE.LOCAL for server srv1
    Jul 28 18:10:20 srv2 rpc.gssd[698]: WARNING: Machine cache prematurelyexpired or corrupted trying torecreate cache for server srv1
    Jul 28 18:10:20 srv2 rpc.gssd[698]: Full hostname for 'srv1' is 'srv1'
    Jul 28 18:10:20 srv2 rpc.gssd[698]: Full hostname for 'srv2.rhce.local' is 'srv2.rhce.local'
    Jul 28 18:10:20 srv2 rpc.gssd[698]: No key table entry found for [email protected] while getting keytab entry for '[email protected]'
    Jul 28 18:10:20 srv2 rpc.gssd[698]: No key table entry found for [email protected] while getting keytab entry for '[email protected]'
    Jul 28 18:10:20 srv2 rpc.gssd[698]: No key table entry found for root/[email protected] while getting keytab entry for 'root/[email protected]'
    Jul 28 18:10:20 srv2 rpc.gssd[698]: Success getting keytab entry for 'nfs/[email protected]'
    Jul 28 18:10:20 srv2 rpc.gssd[698]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_RHCE.LOCAL' are good until 1532879989
    Jul 28 18:10:20 srv2 rpc.gssd[698]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_RHCE.LOCAL' are good until 1532879989
    Jul 28 18:10:20 srv2 rpc.gssd[698]: creating tcp client for server srv1
    Jul 28 18:10:20 srv2 rpc.gssd[698]: creating context with server nfs@srv1
    Jul 28 18:10:20 srv2 rpc.gssd[698]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@srv1
    Jul 28 18:10:20 srv2 rpc.gssd[698]: WARNING: Failed to create machine krb5context with cred cache FILE:/tmp/krb5ccmachine_RHCE.LOCAL for server srv1
    Jul 28 18:10:20 srv2 rpc.gssd[698]: WARNING: Failed to create machinekrb5 context with any credentialscache for server srv1
    Jul 28 18:10:20 srv2 rpc.gssd[698]: doing error downcall

    Any help appreciated, thanks.

    • I think that srv1 should use FQDN (looking at /var/log/messages).

      When you set up FreeIPA, did you add nfs/srv1.rhce.local or nfs/srv1?

    • I think so too but I couldn’t find the root cause, and I believe that I followed carefully your instructions. At the end I removed this VM srv1 and installed another one srv3, followed some steps on the IPA server, and I could make it work. Thanks for helping and for your website.

  48. Hello,
    I’m also impressed from your guidance and I really thank you for all of that.
    maybe I’m confused, but what I’ve been learned from Sander (And in my lab), is that the keytab for NFS client, should be copied from the IPA server with SERVER1 nfs principal and not srv2, as you’ve written…
    since ti is the server1 with the nfs krb5 shares, we need the keytab contains nfs/server1.example.com service..

    correct me if I’m wrong,
    and thanks again
    Ami

    • Thanks for your feedback.

      In our case srv1.rhce.local acts as an NFS server, and srv2.rhce.local is an NFS client. It’s mentioned in “The Lab” section of the article.

  49. Hi Tomas Thank you for the blog and all the material you have put together l have been following your tutorials for some time now and l find them very detailed and well clarified.
    l am currently preparing for the RHCE exam will be seating in a month’s time.
    I set up a kerberized NFS virtual lab and at some point l got the error:
    mount.nfs: remote share not in ‘host:dir’ format

    My configurations are as follows
    Server side
    in the /etc/exports file l have
    /public 192.168.8.0(ro)
    /protected 192.168.8.0(rw,sec=krb5p)

    Client Side l have created mount points
    /mnt/public for the /public share
    /mnt/secure for the /protected share

    l was able to export and mount the non-kerberized public share successfully, it was only the protected kerberized share that
    gave me this error.
    Can you please assist l am not quite sure where l am making the mistake.If you need more details l am ready to send them through.

    Thank You

    • Hi, thanks for your feedback!

      The error message says that your remote share format is not correct. Looking at your export statement, you’re trying to export to a network address 192.168.8.0 (I take it that you use 192.168.8.0/24) rather that a host. Try something like 192.168.8.5 (or whatever IP is set on your server).

  50. I am still training a lot.. I created three NFS shares (one normal, two kerberized), and after mounting them on the client I get an inconsistent output with the df command. As you see below, mount and df listings of NFS mounts are different. How is it possible ? It’s on RH 7.0 for both server and client.

    [root@srv3 mnt]# mount -lt nfs4
    srv2:/srv/public-ro on /mnt/public-ro type nfs4 (rw,relatime,vers=4.0,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.122.73,local_lock=none,addr=192.168.122.72,_netdev)
    srv2:/srv/protected on /mnt/protected type nfs4 (rw,relatime,vers=4.0,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=192.168.122.73,local_lock=none,addr=192.168.122.72,_netdev)
    srv2:/srv/project on /mnt/project type nfs4 (rw,relatime,vers=4.0,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=192.168.122.73,local_lock=none,addr=192.168.122.72,_netdev)
    [root@srv3 mnt]# df -hTt nfs4
    Filesystem          Type  Size  Used Avail Use% Mounted on
    srv2:/srv/public-ro nfs4  4.0G  852M  3.2G  21% /mnt/public-ro
    srv2:/srv/project   nfs4  4.0G  852M  3.2G  21% /mnt/project
  51. I have noticed that when I reboot the nfs server the nfs-server and nfs-secure-server take bit to start (like 3 minutes or so) is this normal?

  52. Hi Tomas

    Im having issues here configuring kerberised nfs .

    IPA IP 10.8.8.2
    Serv1 = 10.8.8.50 (NFS Server)
    Serv2 = 10.8.8.51 (NFS Client)

    on NFS server 10.8.8.50 exports file contains

    /srv/nfs_pub 10.8.8.0/24(sec=sys,rw,sync,root_squash,all_squash)
    /srv/nfs_group 10.8.8.0/24(sec=sys,rw,sync,root_squash)
    /srv/nfs_secure serv2.rhce.local(sec=krb5p,rw,sync)

    while mounting on client i get this error

    mount.nfs4: timeout set for Sun Oct 28 14:27:25 2018
    mount.nfs4: trying text-based options ‘sec=krb5p,addr=10.8.8.50,clientaddr=10.8.8.51’
    mount.nfs4: mount(2): Permission denied
    mount.nfs4: access denied by server while mounting serv1.rhce.local:/srv/nfs_secure

    and on NFS Server i see this error message

    Oct 25 11:12:16 serv1 rpc.svcgssd[17636]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more information) – Ticket not yet valid
    Oct 25 11:12:16 serv1 rpc.svcgssd[17636]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more information) – Ticket not yet valid
    Oct 25 11:12:16 serv1 rpc.svcgssd[17636]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more information) – Ticket not yet valid

    Note i didnt add Version 4.2 on NFS server i kept it like original RPCNFSDARGS=””

    • Ticket not yet valid suggests that there is a time synchronisation problem. Make sure that all servers are in sync.

    • Ok Tomas

      i fixed the time sync issue, but this time when im trying to mount kerberised nfs on Client , im getting this error

      [root@serv2 (~)]$ > mount -v -t nfs4 -o sec=krb5p serv1.rhce.local:/srv/nfs_secure /mnt/nfs_secure
      mount.nfs4: timeout set for Mon Oct 29 11:36:08 2018
      mount.nfs4: trying text-based options ‘sec=krb5p,addr=10.8.8.50,clientaddr=10.8.8.51’
      mount.nfs4: mount(2): Operation not permitted
      mount.nfs4: Operation not permitted

      On NFS Server messages file has these errors

      Oct 29 11:34:00 serv1 nslcd[2333]: [2342ec] request denied by validnames option
      Oct 29 11:34:00 serv1 nslcd[2333]: [487cb0] request denied by validnames option

    • From client if i try to mount just using “mount -t nfs -o sec=krb5p ” it will mount successfully but this is mounting using NFSv3 but if i try to mount using “mount -t nfs4 ” im receiving Access denied or Operation not Permitted messages.

  53. mount.nfs: an incorrect mount option was specified
    I can’t get past this error when mounting a kerberized share.
    I have kerberos authentication setup and nfs-secure-server running on both servers.
    Any ideas?

  54. Hey Tomas, I am getting this error from server2

    [root@srv2 ~]# mount -t nfs4 -o vers=4.2,sec=krb5p srv1.rhce.local:/srv/nfs_secure /mnt/nfs_secure
    mount.nfs4: access denied by server while mounting srv1.rhce.local:/srv/nfs_secure

    [root@srv2 ~]# hostnamectl
    Static hostname: srv2.rhce.local

    [root@srv2 ~]# klist -k
    Keytab name: FILE:/etc/krb5.keytab
    KVNO Principal
    —- ————————————————————————–
    2 nfs/[email protected]
    2 nfs/[email protected]
    2 nfs/[email protected]
    2 nfs/[email protected]
    2 nfs/[email protected]
    2 nfs/[email protected]

    [root@srv2 ~]# systemctl status nfs-secure -l
    nfs-secure.service – Secure NFS
    Loaded: loaded (/usr/lib/systemd/system/nfs-secure.service; enabled)
    Active: active (running) since Sat 2018-11-10 20:24:55 ALMT; 24min ago
    Process: 3310 ExecStart=/usr/sbin/rpc.gssd $RPCGSSDARGS (code=exited, status=0/SUCCESS)
    Main PID: 3311 (rpc.gssd)
    CGroup: /system.slice/nfs-secure.service
    └─3311 /usr/sbin/rpc.gssd -vvv

    Nov 10 20:47:13 srv2.rhce.local rpc.gssd[3465]: No key table entry found for root/[email protected] while getting keytab entry for ‘root/[email protected]
    Nov 10 20:47:13 srv2.rhce.local rpc.gssd[3465]: No key table entry found for nfs/[email protected] while getting keytab entry for ‘nfs/[email protected]
    Nov 10 20:47:13 srv2.rhce.local rpc.gssd[3465]: No key table entry found for host/[email protected] while getting keytab entry for ‘host/[email protected]
    Nov 10 20:47:13 srv2.rhce.local rpc.gssd[3465]: No key table entry found for [email protected] while getting keytab entry for ‘SRV2$@’
    Nov 10 20:47:13 srv2.rhce.local rpc.gssd[3465]: No key table entry found for root/[email protected] while getting keytab entry for ‘root/srv2.rhce.local@’
    Nov 10 20:47:13 srv2.rhce.local rpc.gssd[3465]: No key table entry found for nfs/[email protected] while getting keytab entry for ‘nfs/srv2.rhce.local@’
    Nov 10 20:47:13 srv2.rhce.local rpc.gssd[3311]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt17
    Nov 10 20:47:13 srv2.rhce.local rpc.gssd[3311]: Closing ‘gssd’ pipe for /var/lib/nfs/rpc_pipefs/nfs/clnt15
    Nov 10 20:47:13 srv2.rhce.local rpc.gssd[3311]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt16
    Nov 10 20:47:13 srv2.rhce.local rpc.gssd[3311]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt15

    Any idea? Thanks

    • It looks like you have a domain mismatch. Your keytab contains entries for @RHCE.LOCAL, but your client is configured to use @EXAMPLE.COM.

    • Hello again, In client machine, I edited /etc/krb5.conf file and edited EXAMPLE.COM & example.com to RHCE.LOCAL & rhce.local. Then I restarted the nfs-secure service & tried to mount again (mount -t nfs4 -o vers=4.2,sec=krb5p srv1.rhce.local:/srv/nfs_secure /mnt/nfs_secure) and still get access denied error. In systemctl status nfs-secure, I get this error:

      Nov 14 20:45:39 srv2.rhce.local rpc.gssd[3607]: handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt3)
      Nov 14 20:45:39 srv2.rhce.local rpc.gssd[3607]: process_krb5_upcall: service is ”
      Nov 14 20:45:39 srv2.rhce.local rpc.gssd[3607]: Full hostname for ‘srv1.rhce.local’ is ‘srv1.rhce.local’
      Nov 14 20:45:39 srv2.rhce.local rpc.gssd[3607]: Full hostname for ‘srv2.rhce.local’ is ‘srv2.rhce.local’
      Nov 14 20:45:39 srv2.rhce.local rpc.gssd[3607]: No key table entry found for [email protected] while getting keytab entry for ‘[email protected]
      Nov 14 20:45:39 srv2.rhce.local rpc.gssd[3607]: No key table entry found for root/[email protected] while getting keytab entry for ‘root/[email protected]
      Nov 14 20:45:39 srv2.rhce.local rpc.gssd[3607]: Success getting keytab entry for ‘nfs/[email protected]
      Nov 14 20:45:44 srv2.rhce.local rpc.gssd[3598]: Closing ‘gssd’ pipe for /var/lib/nfs/rpc_pipefs/nfs/clnt3
      Nov 14 20:45:44 srv2.rhce.local rpc.gssd[3598]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt4
      Nov 14 20:45:44 srv2.rhce.local rpc.gssd[3598]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt3

      Any Idea. I completed all the RHCE objectives other than this objective :( Need help please

    • Keytab entry for srv1 was not found, are you sure that the entry exists?

      I’d suggest you to start from scratch and use the same domain across all systems when configuring services.

  55. Again, I’m trying to practice with just CentOS 7.0, but when installing ‘nfs-utils’ via yum, it automatically tries to update to 7.5:

    Installing:
    nfs-utils x86_64 1:1.3.0-0.54.el7 Packages 407 k
    Updating:
    centos-release x86_64 7-5.1804.el7.centos Packages 24 k

    I setup a local repository from packages in the CentOS 7.0 Everything iso. Any way of preventing this? It happens when I try and install ‘sssd’ also.

  56. Hi!
    When I export NFS V4.2 shares with mixed security (2 folders with sys and 1 with krb5p), it doesn’t work for ip ranges or wildcards (192.168.3.0/24 nor *.rhce.local). I can only make it work if I specify a single IP address or hostname for all the shares.

    For example, it works with:
    /srv/nfspub 192.168.3.0/24(sec=sys,rw,sync,root_squash,all_squash)
    /srv/nfsgroup 192.168.3.0/24(sec=sys,rw,sync,root_squash)

    It also works with:
    /srv/nfssec srv2.rhce.local(sec=krb5p,rw,sync)

    Everything works with:
    /srv/nfspub srv2.rhce.local(sec=sys,rw,sync,root_squash,all_squash)
    /srv/nfsgroup srv2.rhce.local(sec=sys,rw,sync,root_squash)
    /srv/nfssec srv2.rhce.local(sec=krb5p,rw,sync)

    But nothing works with:
    /srv/nfspub 192.168.3.0/24(sec=sys,rw,sync,root_squash,all_squash)
    /srv/nfsgroup 192.168.3.0/24(sec=sys,rw,sync,root_squash)
    /srv/nfssec srv2.rhce.local(sec=krb5p,rw,sync)

    Any idea ?
    Thanks!

    • Mi Remi. I had a similar issue at some point (when mixed sec used).
      I believe it resolved for me when – while mounting on client side – specified sec=sys. When I raised verbosity on client, I noticed on nfs-secure status there are some log messages containing krb stuff (and we want to share /srv/nfsgroup with sec=sys on srv side).
      Hope this may help :)

  57. Hello,

    I have a question regarding krb5p nfs share. I have exported the share and mounted it on srv2.

    Srv1 — nfs-server

    drwxr-xr-x. alice root unconfined_u:object_r:nfs_t:s0 secur

    /srv/secure srv2.rhce.local(sec=krb5p,rw,sync)

    Srv2 — nfs-client
    drwxr-xr-x. alice root system_u:object_r:nfs_t:s0 secure

    When i switch to user Alice I’m unable to access the share at all. After acquiring the ticket (kinit) I can enter the share and write to it, but after destroying the the kerberos ticket (klist: Credentials cache keyring ‘persistent:1761600001:1761600001’ not found) I still can write to that share with to issues. Is this behaviour correct?

    • Only Kerberos authorized users can access and write to Kerberos-protected shares, you switched to user alice- which user alice, the local one? Local users can’t write to Kerberos protected shares when those shares are mounted, we may expect to be able to write as some user or even as root since it is a locally mounted share- but that does not work. So you switched to the Kerberos authorized user alice with kinit, it’s not the local user alice anymore, THAT user will have full read/write privileges provided via ownership.

  58. Hello,

    I’m having a trouble with making the exported setting persistent. After nfs server reboot I’m unable to access the previously exported share (/var/lib/nfs/etab is empty) and I need to issue exportfs again. NFs is configured with -V 4.2 in /et/sysconfig/nfs . nfs-server is set to autostart. (redhat 7.5)

  59. I see you, along with Sander and CertDepot, rely a lot on kinit on both server and client in the Kerberized NFS scenario. Using kinit requires that an account is opened for the client on the IPA server and that you are given that account’s password. I am really not sure any of that will be available on the RHCE exam. Is there a solution that does the role of the IPA and that would require only possesing keytab files for granting the client access to and mount of, the Kerberized NFS share?

    • Yes, there is (good question BTW). You can alternatively download the keytab file if don’t have Kerberos admin credentials. It’s explained in the blog post.

    • Tomas, I followed your recommendation to practice with RHEL 7.0 and not CentOS 7.0 given as a response to my comment in your FreeIPA setup guide and have practiced steps given in this guide, the part about mounting Kerberized NFS shares when only keytabs are given when both NFS server and client are RHEL 7.0 and:
      – once you install nfs-utils, even if you insist on installing version 1.3.0, you will no longer be working with RHEL 7.0 but with 7.6 as verified by checking /etc/redhat-release. Succesfull installation of nfs-utils will move you to different version.
      – when given only keytab files and not kinit credentials for authentication, you’ll have to manually redact /etc/krb5.conf and replace every mention of “example.com” with your own custom domain. Keep track of case and punctuation in that file but replace the “example.com” otherwise access will be denied and keytabs invalidated and reported as bad in /var/log/messages.

    • You should be using RHEL 7.0 DVD as a local repository to avoid this problem. At least that’s what I always do.

      See here for more info:

      https://www.lisenet.com/2016/create-local-yum-repository-from-systemd-mounted-dvd-on-rhel-7/

      I didn’t do any manual changes to /etc/krb5.conf. Instead, I removed all content, and used authconfig-tui to configure Kerberos. See here for more info: https://www.lisenet.com/2016/ldap-and-kerberos-client-authentication-on-rhel-7-using-sssd/

    • I just wrapped up setting up the FreeIPA server as a repo server as well, with the copy of the DVD as the source for the repo. Brilliant solution for a home made, RHEL- compliant lab, have the DVD as a repo and avoid versioning disparities between CentOS and RHEL and whatnot, just for sake of labbing!
      In your guide on using authconfig-tui for properly setting up NFS client to mount a secure NFS share you rely on downloading cacert file- am not sure at all it would be available on the exam.

    • Yea, you can also upgrade between RHEL versions easily. Simply mount a RHEL 7.1 DVD, and your repo becomes a RHEL 7.1 repo on a RHEL 7.0 server. Similar way you can go from RHEL 7.0 to RHEL 7.6 if you want to test them all.

      With regards to NFS, you have to have a certificate in order to configure the service. Therefore the certificate must provided in one way or another.

  60. With Kerberos-protected NFS shares, only Kerberos-authenticated users can write on those shares when they are mounted on client. Local users on the client can’t write on those shares, not even root. And here lies the catch: with only the keytab files you can mount the secured NFS share- but you can’t write on it. In order to effectively enable Kerberos-authenticated users to write on your mounted share, your client has to join the IPA/Kerberos domain/realm. In order to do that, you need to run ipa-client-install and to succeed at that you’ll need the password of the Kerberos-authenticated user authorized to enroll hosts in the domain. Without authentication detail of that level there can be no writing to protected share.

  61. … or at exam time, their IdM server is specially set up so that downloading keytab files is sufficient to join client machines to the domain and allow domain-only, Kerberos- authenticated users to log in…
    In lieu of that, Kerberos admin credentials HAVE to be listed (and with that run ipa-client-install, by far the easiest way of joining client machines to a domain) otherwise I can’t see how would a client join just by downloading keytab files.

  62. Followed your tutorial exactly except the v4.2 stuff.
    Used the Centos 7.0 download for server1 and server2.
    Having trouble with writing to the kerborized share.
    After doing a kinit for alice , I can read the directory.
    Creating a file results in Permission denied.
    After chmod 777 on /srv/nfs_secure, alice can create files , but the are owned by nfsnobody.

    /etc/exports on srv1:
    /srv/nfs_secure *(rw,sec=krb5p,no_root_squash)

    Any help is greatly appreciated.

Leave a Reply

Your email address will not be published. Required fields are marked *