On one of my instances (all running Rome), I am seeing the following in the system logs (separate but related log entries):
KMF cryptographic module key for glide encrypter is missing: no thrown error
Couldn't find Crypto Module: no thrown error
I noticed this right after a scheduled upgrade + clone on the target instance; however, I'm not 100% sure the error had existed on the target instance prior to the upgrade and/or clone. I don't see the error in the source instance. With that being said, I opened a case with ServiceNow Support and they weren't any help, unfortunately. They pointed me to a KB article, thinking it's SAML related:
The article I linked above has a similar error, yes, but it's not the same one I'm reporting here. SAML is not enabled and the error I'm seeing doesn't have any mention of SAML, unlike the KB article I linked.
I found another community thread that might have some useful information which applies to this particular error but I think it might speak to a different matter.
I did my own digging and what I found was the errors were being generated during the login/SSO process albeit, again, not using SAML. I don't have any login issues. SSO works fine. It's not like this error is preventing users from signing into ServiceNow. It's more of an annoyance and noise in the logs and the question is why and how is the error generating. I eventually narrowed it down to the following Script Include and its getSecretKey function:
Script Include:
global.MultiSSO_Abstract_Core
Function:
getSecretKey : function (data){
var cred = new String(data);
var e = new GlideEncrypter();
var jsCred = cred + '';
var decryptedData = e.decrypt(jsCred);
return decryptedData ;
}
I threw the call into Scripts - Background and isolated the errors to the following line:
var decryptedData = e.decrypt(jsCred);
// output:
KMF cryptographic module key for glide encrypter is missing: no thrown error Couldn't find Crypto Module: no thrown error
It generates the errors as long as the decrypt function has a non-zero length argument being passed (i.e. empty string will be ignored but '12345' is valid and function will try to decrypt). If it's a valid secret, user logs in successfully. If it's an invalid secret, user auth fails. Which is what we want. But the error is generated every time, regardless if user auth succeeds or fails.
The problem lies in the decrypt function call. It's a black box to me. Where can I view its source code so I may trace the error further? Or is GlideEncrypter API compiled code and, thus, I'm at a dead-end for the moment?
The other question is, OK, if SSO/login auth is completing successfully, then what is this noise all about? Is this a defect with Rome? Does anyone have any insights into what scripts or plugins/libraries the decrypt function may be leveraging so I may continue to track down the root cause of these errors?
Thank you for your help!
I opened up another case with ServiceNow Support on this one. They were able to correct it and resolve the issue.
They determined that the KMF plugin was not activated correctly in some instances and needed to be reinstalled. Before they could do that, however, they found a node in the retired state and restored the node. Restoring the node may or may not have been required in order to reinstall the KMF plugin. From what it sounds like, it was separate but possibly related so no harm in restoring the node in case the KMF plugin reinstall didn't fix the issue. Once those two steps were done, I tested the GlideEncrypter API decrypt function and, boom, no more KMF or crypto errors.
One causality of the node restore, KMF plugin reinstall, or both, is my SSO secret was blown out, breaking SSO. I'm not sure if the field data was NULL, an empty string, random data, or maybe a secret from a different instance. In any case, I re-set the SSO secret to its proper value and SSO worked again.
As an aside, I couldn't see the KMF Cryptographic Modules table in the Navigator or its records. Sys Admin + Security Admin was still not sufficient to see its records. What gives? Well, as it turns out, I needed to add the Sys Admin user to the allowed list in the Key Management Administration module (which is searchable in the navigator) to add the KMF Admin role to the Sys Admin user. Once I did that and logged out and logged back in, Cryptographic Modules appeared in the navigator and I could see all its records.
I am sorry I cannot provide a solution, we are experiencing similar difficulties, however we are unable to use SSO to sign in to any of the apps (Agent or Now).
Are you able to sign in to them?
We have the same error as you but also multiple KMF errors when accessing the app.
Signing in to the instance from browser works, and, everything works okay in Dev including the apps.
We starting getting the issue at the beginning of May (do your errors start a similar time?).
I have an open case with HI being investigated, I will report back if anything useful that could apply to both of us.
Hey. I have traced it to the Encryption Support plugin, at least for my error thrown. Encryption Support plugin interacts with the GlideEncrypter API, it would appear, and, specifically, with the decrypt method. What it's doing is anyone's guess but, based on my sleuthing, it's likely performing a table lookup and the table either doesn't exist or the table exists and there are no records, or the table exists, records exists, but it's performing a particular query and not finding the record it's seeking. Instead of recording it as an info log, it records it as an error thrown, but it's inconsequential to the SSO/Login auth process. If the secret key is valid, it succeeds. Otherwise, it fails. Has nothing to do with the error thrown. Again, my particular error. Your error may be worded slightly differently and is, therefore, a different error thrown.
If I were you, I'd first check the SPEntry Script Include and start tracing from there. Then check your SSO scripts and look for any GlideEncrypter functions, be it encryption or decryption. Most likely, your KMF errors are being generated there. If you're using SAML, then your cert/keys may play a part and there are KB articles on how to supposedly resolve it.
Yes, please, if you hear anything back that's useful, that'd be great if you could update this thread. Thank you!
Correction: Encryption Support plugin may be related. Tested on a fresh personal Rome instance with and without that plugin installed and no errors. Opened up a case with ServiceNow Support. The easiest way to troubleshoot this now is for the ServiceNow Engineers, if they're completely unaware of this issue, is scan the decrypt function and look for the block of code which is triggering these errors. At the moment, I'm not sure what plugins could be tied into the decrypt function triggering these errors. SSO/Login auth still works. Who's to say it couldn't break tomorrow? Leave no stone unturned, I say.
I opened up another case with ServiceNow Support on this one. They were able to correct it and resolve the issue.
They determined that the KMF plugin was not activated correctly in some instances and needed to be reinstalled. Before they could do that, however, they found a node in the retired state and restored the node. Restoring the node may or may not have been required in order to reinstall the KMF plugin. From what it sounds like, it was separate but possibly related so no harm in restoring the node in case the KMF plugin reinstall didn't fix the issue. Once those two steps were done, I tested the GlideEncrypter API decrypt function and, boom, no more KMF or crypto errors.
One causality of the node restore, KMF plugin reinstall, or both, is my SSO secret was blown out, breaking SSO. I'm not sure if the field data was NULL, an empty string, random data, or maybe a secret from a different instance. In any case, I re-set the SSO secret to its proper value and SSO worked again.
As an aside, I couldn't see the KMF Cryptographic Modules table in the Navigator or its records. Sys Admin + Security Admin was still not sufficient to see its records. What gives? Well, as it turns out, I needed to add the Sys Admin user to the allowed list in the Key Management Administration module (which is searchable in the navigator) to add the KMF Admin role to the Sys Admin user. Once I did that and logged out and logged back in, Cryptographic Modules appeared in the navigator and I could see all its records.
