RuntimeError: No suitable key found in keytab.

[joakim-tjernlund]

Moving ufven#11 to here:
I get 3-4 of these each time I boot

2020-05-19 17:43:25,014 main:ERROR:Traceback (most recent call last):
File "/usr/libexec/certmonger/cepces-submit", line 61, in main
config = Configuration.load()
File "/usr/lib64/python3.6/site-packages/cepces/config.py", line 131, in load
return Configuration.from_parser(config)
File "/usr/lib64/python3.6/site-packages/cepces/config.py", line 169, in from_parser
return Configuration(endpoint, endpoint_type, cas, authn.handle())
File "/usr/lib64/python3.6/site-packages/cepces/auth.py", line 94, in handle
raise RuntimeError('No suitable key found in keytab.')
RuntimeError: No suitable key found in keytab.

Error is raised when there is no network yet.
Should it really do that or just wait a bit and then try again ?

[dmulder]

Honestly, this is probably a bug in the certmonger daemon startup of your distro. Certmonger should not be starting prior to the network being up. File a bug against your distro.

[joakim-tjernlund]

Honestly, this is probably a bug in the certmonger daemon startup of your distro. Certmonger should not be starting prior to the network being up. File a bug against your distro.

hmm, I don't think so. One should be able to inquire certmonger about cert like getcert list without network.
This feels more like missing error handling in cepces, it should not barf like that, just return an error to certmonger.

[dmulder]

Maybe, but the primary problem is your distro should not start certmonger before networking is up.

[dmulder]

And how would we fix this in cepces? Add a early check to make sure the network is reachable, then bail out? No, it's the OS job to ensure the network is up, not ours.

[dmulder]

We could catch the exception somewhere, and print a prettier message, and that might be more desirable for sysadmins, but it is not for those working on the code. I would much rather see the 'barf' so I can determine where things failed.

[dmulder]

One should be able to inquire certmonger about cert like getcert list without network.

OK, when I think about it, this part makes sense. We shouldn't break getcert just because the network is down.
@joakim-tjernlund could you test that out, just to be sure? With certificates already tracked, check if we break getcert if you disconnect the network.

[joakim-tjernlund]

One should be able to inquire certmonger about cert like getcert list without network.

OK, when I think about it, this part makes sense. We shouldn't break getcert just because the network is down. @joakim-tjernlund could you test that out, just to be sure? With certificates already tracked, check if we break getcert if you disconnect the network.

Looking deeper there seems to something else going on, I can see that NetworkManager starts long before certmonger and certmonger starts when NM becomes active later. Something else is bugging SOAPAuth:

 for principal in principals.strip().split('\n'):
            if realm:
                principal = '{}@{}'.format(principal, realm)

            try:
                auth = SOAPAuth.TransportKerberosAuthentication(
                    principal_name=principal,
                    init_ccache=ccache,
                    keytab=keytab,
                )
            except KerberosError:
                # Ignore
                pass

        if auth:
            return auth
        else:
            raise RuntimeError('No suitable key found in keytab.')

[c-erb]

Okay, I did some further testing with older Ubuntu versions and I managed to find two different things that can cause this issue.

1. Make sure the realm is configured correctly. You can configure it in the cepces configuration to be safe side. The difference between the Ubuntu versions is that the package krb5-config is finding the default realm in newer packaged versions and configures it in /etc/krb5.conf which is not the case for older versions.

2. Make sure your hostname is not longer than 15 characters. The issue here is that the configured principal names will not account for the computer object having a cut-off name if it exceeds 15 characters. So either you go with a shorter hostname or you change the principal names in the cepces configuration.

[joakim-tjernlund]

1. I have default_realm set in krb5.conf, should be enough?
2. Our hostname is FQDN, contains the domain as well. Other than that, hostname is less than 16 chars.

[c-erb]

1. I have default_realm set in krb5.conf, should be enough?

Yes that is enough.

2. Our hostname is FQDN, contains the domain as well. Other than that, hostname is less than 16 chars.

So is your output for hostname without domain and hostname -f with domain? Then this should be fine.
What you can do is check /etc/krb5.keytab (or wherever it might be on your system) to see which computerobject is addressed in there and if it fits the principalnames.

Other than that I installed those packages on my test machine (additionally to what is included in the cepces setup:

• libkrb5-dev
• krb5-config
• python3-pip
• python3-requests-kerberos

[joakim-tjernlund]

1. I have default_realm set in krb5.conf, should be enough?

Yes that is enough.

2. Our hostname is FQDN, contains the domain as well. Other than that, hostname is less than 16 chars.

So is your output for hostname without domain and hostname -f with domain? Then this should be fine. What you can do is check /etc/krb5.keytab (or wherever it might be on your system) to see which computerobject is addressed in there and if it fits the principalnames.

No, the other way around. hostname and hostname -f both have domain in it. hostname -s does not.
This I cannot change, something else breaks but dot remember what ATM(maybe NFS?)

Other than that I installed those packages on my test machine (additionally to what is included in the cepces setup:

* libkrb5-dev

* krb5-config

* python3-pip

* python3-requests-kerberos

This is on Gentoo but I have these installed as well except for requests-kerberos, there I have requests-gssapi as later cepces has swithed but The problem was seen before that switch too.

[c-erb]

No, the other way around. hostname and hostname -f both have domain in it. hostname -s does not. This I cannot change, something else breaks but dot remember what ATM(maybe NFS?)

If you checked the keytab file and the hostname is wrong there, than you could try to change the principals directly in the cepces config.

Like so, exchange hostname and domainname ofc.
host/HOSTNAME$$@domainname
host/HOSTNAME$$
host/hostname$$.domainname$$@domainname
HOSTNAME$$

[joakim-tjernlund]

No, the other way around. hostname and hostname -f both have domain in it. hostname -s does not. This I cannot change, something else breaks but dot remember what ATM(maybe NFS?)

If you checked the keytab file and the hostname is wrong there, than you could try to change the principals directly in the cepces config.

Like so, exchange hostname and domainname ofc. host/HOSTNAME$$@domainname host/HOSTNAME$$ host/hostname$$.domainname$$@domainname HOSTNAME$$

Not sure I follow, in keytab I have most variations:

HOST$@REALM
host/host.domain.com@REALM
host/HOST@REALM

and cepces principals are default:

principals=
  ${shortname}$$
  ${SHORTNAME}$$
  host/${SHORTNAME}
  host/${fqdn}

Seems to be match there

[c-erb]

What I am trying to say is that you can set the principals to the actual names to make sure that the problem is not the discovery of the correct host and domain names.

Those are all the issues that I found before I got it working.