Welcome to Part 5 of the series Intune PKI Made Easy With Joy.
Today’s post discusses the different Intune SCEP HTTP errors we may encounter while working with SCEP certificate deployments from Intune. It explains how to identify probable causes to help with quicker troubleshooting and ensure lower downtime.
In the context of Intune, the Simple Certificate Enrollment Protocol (SCEP) can be leveraged to authenticate connections for your apps and corporate resources. SCEP relies on the Certification Authority (CA) certificate to securely facilitate the Certificate Signing Request (CSR) message exchange.
The exchange of messages for the Certificate Signing Request (CSR) is secured using the Certification Authority (CA) certificate within SCEP. When your infrastructure supports SCEP, you can deploy certificates to your devices by utilizing Intune SCEP certificate profiles, a specific type of device profile in Intune.
Disclaimer: The Intune SCEP HTTP Error causes listed in this article are purely based on experience dealing with a production environment and may not match the order in which MS has listed them in its troubleshooting document.
Previous articles of this series Intune PKI Made Easy With Joy
- Part 1 – Learn The Basic Concepts of PKI
- Part 2 – Knowing SCEP – The General Workflow
- Part 3 – Intune SCEP PKI Implementation Deep Dive
- Part 4 – Intune SCEP Certificate Workflow Analysis
Let’s get started.
Demystifying Intune SCEP HTTP Errors
For an internet-facing device to send the SCEP request to NDES, the request must go via a proxy. Azure AD App Proxy (Microsoft recommended) exposes the internal NDES mscep.dll URL in most setups.
So, let’s begin with the HTTP errors we may likely get due to Azure AD App Proxy.
Intune SCEP HTTP Errors – AAD App Proxy Errors
504 Gateway Timeout
This mainly occurs if the AAD App Proxy connector is not Running or the Server hosting the connector has gone offline.
The Microsoft AAD Application Proxy Connector service’s startup type is set to Automatic (Delayed Start) by default.
As such, if the Server recently underwent a reboot cycle, check whether the connector service has started.
502 Bad Gateway
This mainly occurs due to a typo in the internal URL while creating/configuring the App Proxy application in Azure. As a result, the connector fails to resolve the internal URL, resulting in the following error:
This page cannot be displayed/Can’t reach this page
This occurs due to incorrect App Proxy configuration for publishing the SCEP URL.
If you choose to use your custom verified domain in Azure as the mscep URL domain rather than the default “msappproxy.net” domain, you must create a CNAME record in the DNS provider for the custom domain to map the requests.
By default, Azure App Proxy requires no additional configuration to use the default “-.msappproxy.net” domain. Also, you must provide an SSL cert if you choose to use a custom domain other than the default “-msappproxy.net” domain.
In this case, I went with the default configuration to save myself from extra work, but I still got the error. Generating Diagnostics For the App Proxy application in Azure tells the error source.
While configuring the App Proxy application, I left the Pre Authentication set to Azure Active Directory.
For NDES, the pre-authentication setting within the AAD App Proxy application in Azure needs to be set to Passthrough since no actual AAD user will be accessing the application.
Provided that we are good from the proxy side and AAD App Proxy can resolve the external requests to the internal NDES MSCEP URL, the HTTP errors we will encounter depend on the physical infrastructure settings (NDES and PKI).
Intune SCEP HTTP Errors – Configuration Issues
Intune SCEP HTTP Error 500 – Internal Server Error
There can be several causes for this particular error, and you would need to check several things to narrow them down to the actual cause.
NDES Service Account permission/rights issue
The SCEP pool in IIS runs under the NDES service account identity.
You need to check and ensure that the NDES Service Account
- is not in a Locked state,
- password is not Expired,
- is a member of the local IIS_IURS group on the NDES box (ensure no GPO is modifying the membership of the particular group),
- the IIS_IURS group is assigned the Impersonate a client after authentication user right (by default, the right is present unless modified),
- has Read permission to the MSCEP RA certs private key
If the above conditions are not satisfied, you will have the Intune SCEP HTTP Error 500 – Internal Server Error.
MSCEP RA Certificates are Expired (or Deleted or Revoked)
NDES service startup depends on the MSCEP RA Certificates. [More of the NDES service startup sequence is discussed later in detail in this post]
Suppose the certificates are expired, deleted, or revoked from the issuing CA for any cause. In that case, the NDES service will fail to start, resulting in the Intune SCEP HTTP Error 500 – Internal Server Error.
CRP in IIS has Windows Authentication set to Enabled
Ensure CertificateRegistrationSvc in IIS [CRP] has Windows Authentication set to Disabled.
If it is set to Enabled, as shown in the above snapshot, it may result in Intune SCEP HTTP Error 500 – Internal Server Error.
Note! This particular setting may also result in HTTP Error 503 – Service Unavailable.
Just like the/certsrv/mscep app in the SCEP pool, IIS authentication for CRP also needs to have
- Anonymous Authentication = Enabled
- Windows Authentication = Disabled
Issuing CA is Offline or Unreachable from NDES
*********One of the most notable causes of Intune SCEP HTTP Error 500 – Internal Server Error.*********
Network configuration changes done by the network/firewall/proxy team may result in the Issuing CA becoming unreachable or unavailable from the NDES box, resulting in the error. Error events can be viewed from the events below.
Event Source: Windows Logs > Application > NetworkDeviceEnrollmentService Event ID 8 - The Network Device Enrollment Service cannot retrieve information about the certification authority (0x80004005). Unspecified error Event ID 2 - The Network Device Enrollment Service cannot be started (0x80004005). Unspecified error
You can use CertUtil
Command on the NDES server to verify CA availability. If your Issuing CA is unreachable or unavailable from the NDES box, you should expect an output like the one below.
PS C:\Windows\system32> certutil -config - -ping
WIN-2CQQDB9STE6.joymalya.xyz\joymalya-WIN-2CQQDB9STE6-CA
Connecting to WIN-2CQQDB9STE6.joymalya.xyz\joymalya-WIN-2CQQDB9STE6-CA ...
Server could not be reached: The RPC server is unavailable. 0x800706ba (WIN32: 1722 RPC_S_SERVER_UNAVAILABLE) -- (32ms)
CertUtil : -ping command FAILED: 0x800706ba (WIN32: 1722 RPC_S_SERVER_UNAVAILABLE)
CertUtil : The RPC server is unavailable.
The same can be verified using another tool named pkiview.msc
Sometimes, the issuing CA server is online for a multi-tier PKI setup. Still, Certificate Services of the Issuing CA fail to start after a server restart [legit causes] because the Root CA CRL has expired.
Consider the classic example below: When I do a standard ping, my CA server returns a response that means it’s online and available, but a CertUtil ping to the same CA server says otherwise.
PS C:\Windows\system32> ping WIN-2CQQDB9STE6.joymalya.xyz Pinging WIN-2CQQDB9STE6.joymalya.xyz [10.0.0.20] with 32 bytes of data: Reply from 10.0.0.20: bytes=32 time<1ms TTL=128 Reply from 10.0.0.20: bytes=32 time=1ms TTL=128 Reply from 10.0.0.20: bytes=32 time=1ms TTL=128 Reply from 10.0.0.20: bytes=32 time=1ms TTL=128 Ping statistics for 10.0.0.20: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 1ms, Average = 0ms PS C:\Windows\system32> certutil -ping WIN-2CQQDB9STE6.joymalya.xyz Connecting to WIN-2CQQDB9STE6.joymalya.xyz ... Server could not be reached: The RPC server is unavailable. 0x800706ba (WIN32: 1722 RPC_S_SERVER_UNAVAILABLE) -- (32ms) CertUtil: -ping command FAILED: 0x800706ba (WIN32: 1722 RPC_S_SERVER_UNAVAILABLE) CertUtil: The RPC server is unavailable.
This can happen for multi-tier PKI infra where the Root CA is primarily offline and the Root CA CRL has expired. The Issuing CA went through a reboot due to any cause. After rebooting, the Issuing CA’s certificate services will fail to start, as when it tries to retrieve the Root CA CRL to verify its cert chain, the verification operation fails.
Fixing the issue might require engagement from both the Network and/or ADCS teams, as it does not generally fall within the Intune scope.
NDES unable to retrieve CRL from CDP
******This is the other most prominent cause of Intune SCEP HTTP Error 500 – Internal Server Error******
This occurs because the NDES box fails to retrieve the Certificate Revocation List (CRL from CDP) required for the NDES service to start itself.
Remember that NDES is implemented as an ISAPI extension in IIS; as such, you will not see NDES as a service when you check in services.msc. Instead, the service runs within the w3wp.exe IIS worker process.
The NDES service must start successfully for the SCEP pool in IIS to run. The NDES service startup checklist is available here.
In a nutshell, the NDES Service Startup Sequence is –
- Locate RA cert from machine cert store – X509Objects
- Acquire the Private Keys – CryptAcquireCertificatePrivateKey
- Build cert chain – CertGetCertificateChainStart
- Verify Revocation Status – CertVerifyRevocationStart
If all the steps are completed successfully, the NDES service starts, and you get the success events below
Event Source: Windows Logs > Application > NetworkDeviceEnrollmentService Event ID 47 - The Network Device Enrollment Service loaded the Registration Authority (RA) key exchange certificate with serial number ###### from the "MY" store. Event ID 48 - The Network Device Enrollment Service loaded the Registration Authority (RA) signature certificate with serial number ###### from the "MY" store. Event ID 1 - The Network Device Enrollment Service started successfully.
However, the NDES service fails to start if it fails to verify the MSCEP-RA certificates. Error events are as follows.
Event Source: Windows Logs > Application > NetworkDeviceEnrollmentService Event ID 8 - The Network Device Enrollment Service cannot retrieve information about the certification authority (0x80004005). Unspecified error Event ID 2 - The Network Device Enrollment Service cannot be started (0x80004005). Unspecified error
The event above does not necessarily mean that the MSCEP RA certs are expired or not present (deleted); it can also be because of an Expired CRL or the NDES server unable to retrieve the CRL, though the events above do not expressly state such.
To check the actual reason for the failure to load the RA certs, you need to enable CAPI2 logging-in events [Applications and Services logs > Microsoft > Windows > CAPI2], then restart the Cryptographic Services and IIS [PS command iisreset] and check back in the CAPI2 Operational logs.
CAPI 2 Event Trace for the same is shown below
Event ID 90 - X509 Objects - Loads RA cert Event ID 70 - Acquire Certificate Private Key - Crypt Function CryptAcquireCertificatePrivateKey Event ID 10 - Build Chain - Crypt Function CertGetCertificateChainStart to start Chain Building Event ID 40 - Verify Revocation - Crypt Function CertVerifyRevocationStart to start revocation status Event ID 52 - Retrieve Object from Network - Crypt Function CryptRetrieveObjectByUrlWireStart to retrieve CRL
The error events start here if the CA CRL has expired or the CRL location is offline.
Error Event ID 53 - Retrieve Object from Network - Crypt Function CryptRetrieveObjectByUrlWire which will report error if CRL not retrievable. Error Event ID 42 - Reject Revocation Information - Crypt Function CertRejectedRevocationInfo. Here you can see that if CRL was actually retrieved, the failure reason might be due to Action CheckTimeValidity [retrieved CRL has expired] Error Event ID 41 - Verify Revocation - Crypt Function CertVerifyRevocation will give you the outcome of revocation verification.
In the NDES server, you can use the CertUtil command CertUtil -url cache CRL to view the locally cached CRL URLs and then use the CertUtil command certutil -URL <LDAP or HTTP URL of CRL>
to verify if the NDES server can retrieve the CRL from CDP.
You will see the URL Retrieval Tool GUI window like this: Select CRLs (from CDP) and click on Retrieve. In an ideal scenario, the status will be OK.
Important! If a proxy server facilitates communication between servers, ensure that proxy settings in all the servers involved are configured for both user and machine/system context.
In the production environment, a firewall may be in between or a proxy facilitating the communications. For a firewall, you need to ensure you have the proper exceptions set to allow connections. For a proxy, you must set proxy settings in the SYSTEM context for NDES to work. The best way to do this is to use the PSEXEC tool.
Open a CMD using PSEXEC and confirm the CMD process runs as a SYSTEM using the command whoami. Once confirmed, you can use the netsh winhttp show proxy to see if the proxy settings are correctly configured in the SYSTEM context. If not, use the netsh winhttp to set the proxy and the command to set the same.
Fixing issues with retrieving CRLs or expired CRLs should mostly fall under the scope of ADCS and not the scope of Intune in general.
Once the CRL issue is resolved, restart the Cryptographic Services and IIS on your NDES server and wait to check if NDES services started successfully. [Event ID 1 for the successful startup of NDES]
Intune SCEP Http Error 503 – Service Unavailable
This one is pretty straightforward. In most cases, the IIS SCEP pool crashes (stopped stated). The likely causes are
Untrusted Intermediate Certificates in NDES server cert store
This is very common for multi-tier PKI infrastructure. Hence, if you are facing this issue, the first thing you should check on your NDES box is to open PowerShell and run the following command.
Get-Childitem -Path cert:\LocalMachine\root -Recurse | Where-Object {$_.Issuer -ne $_.Subject}
If the above code gives any output, it means there are untrusted certs present, and you can delete them by running the command below.
Get-Childitem -Path cert:\LocalMachine\root -Recurse | Where-Object {$_.Issuer -ne $_.Subject}| Remove-Object
Note! If you frequently encounter this issue, consider creating a scheduled task to run this command automatically at regular intervals to remove the untrusted certificates.
- IIS SSL binding certificate was renewed, but the NDES Policy module still points to the old SSL certificate thumbprint
Remember, the certificate to bind port 443 of IIS must be selected while installing the Intune NDES certificate connector. This situation arises primarily when the IIS SSL Binding Certificate gets renewed but not updated with the Intune NDES Certificate Connector.
You must ensure that the thumbprint of the certificate held by the NDES Policy Module must match the thumbprint of the SSL binding certificate used in IIS.
The Intune NDES Connector service requires access to the URLs in CRL for proper functioning. As such, after a successful NDES service startup, if, for any reason, the CRL URLs become unreachable again from the NDES box, it may result in HTTP Error 503—Service Unavailable.
Notice that this time, the NDES service has started itself. Otherwise, the same scenario during NDES service startup results in HTTP Error 500—Internal Server Error.
The End
In addition to all the Intune SCEP HTTP Errors as listed above, you may also encounter HTTP Error 414 – Request URI Too Long and HTTP Error 413 – Payload Too Large, which are all due to incorrect IIS Request Filtering settings for the /certsrv/mscep Virtual app.
The causes of the errors presented in this article may not be exhaustive. While troubleshooting, you may encounter an entirely new cause, which is quite possible. I would be very interested to learn if you have discovered any new reason for the above error while troubleshooting in your environment.
Well, that was all I had planned for today. I will be back with some other posts shortly. Until then, continue to stay safe.
We are on WhatsApp. To get the latest step-by-step guides and news updates, Join our Channel. Click here –HTMD WhatsApp.
Author
Joymalya Basu Roy is an experienced IT service professional with almost five years of experience working with Microsft Intune. He is currently working as a Senior Consultant – Architect with Atos India. He is an ex-MSFT, where he worked as a Premiere Support Engineer for Microsoft Intune. He was also associated with Wipro and TCS in the early stages of his career. He was awarded the Microsoft MVP award for Enterprise Mobility in 2021. You can find all his latest posts on his blog site, MDM Tech Space, at https://joymalya.com
Good stuff can you do a blog on HA for NDES? I do not see much content on this subject.
HA for NDES is also on my radar. Hopefully, it will be sooner than later.
Hi, any hint if we encounter event id 29 error in Network Device enrollment ? thanks.
Hi Sebastien,
Though an old post, but have you checked https://docs.microsoft.com/en-us/archive/blogs/tune_in_to_windows_intune/ndes-event-id-29-the-password-in-the-certificate-request-cannot-be-verified
Hello Joy,
I also ran into HTTP 500 error when setting up NDES server, logged a case with MS and didn’t revert back for 2 weeks and then I came across your blog which helped me to look at the CRL location and turns out in our case the CRL location was wrongly published. thanks and keep sharing!!
Hi Surjeet,
Great to hear that the blog helped you.
We have a subject name: false error on our IIS cert. Any tips?
Great extensive post!
I’m still confused about the relation with a third party on-prem forward proxy and Azure Application (reverse) proxy.
I cannot get an Internet connection through our forward on-prem proxy when signing in to Azure via the Intune certificate connector.
However, if I browse to that URL via IE on my NDES server to https://portal.manage.microsoft.com, I get an HTTP 403 ‘forbidden’. This makes sense since we have to authenticate through our on-prem proxy with an account.
But after authenticating and configuring this proxy in the Intune Connector configuration, I still don’t get access.
Any help would be appreciated, thanks!
NDES Disable Password Requirement.
I’ve read a few blogs and articles that say;
“There is no way for Cisco devices to supply the required password to enroll with NDES/MSCEP, so you need to disable the requirement for a password.”
This is NOT TRUE, however the whole point of issuing certificates via your PKI infrastructure, is that it can scale dramatically. If you are creating passwords and embedding those passwords in all your enrollments, it can get a little unwieldy. So it may be sensible to remove the password requirement.
1. Windows Key+R > regedit {Enter} > Navigate to;
HKEY_LOCAL_MACHINE > SOFTWARE > Microsoft > Cryptography > MSCEP > EnforcePassword > EnforcePassword
To disable change the value to 0 (zero).
Disable NDES Password Enforce
Update: 22/10/21: You may also need to recycle the SCEP application pool in IIS (on the Certificate Services Server)
From IIS Manager > CA > Application Pools, SCEP. > From the right hand panel > Advanced Settings. > Set Load User Profile to ‘True‘ > OK.
Again in the right panel > Recycle > From IIS Manager > Sites > Default Web Site. > From the right panel, click Restart.
Wonderful article
Any chance you have an updated screen shot for CRP in IIS has Windows Authentication set to Enabled . It looks like it doesnt create this website anymore with the new connector