Intune SCEP HTTP Errors Troubleshooting Made Easy With Joy – #5

4
Intune SCEP HTTP Error - Intune PKI Made Easy With Joy - #5

Welcome to Part 5 of the series Intune PKI Made Easy With Joy.

Today’s post is all about the different Intune SCEP HTTP errors that we may get while working with SCEP certificate deployments from Intune – how to identify their probable causes to help quicker troubleshooting to ensure lower downtime.

Disclaimer: The Intune SCEP HTTP Error causes as listed in this article are purely based on experience dealing with production environment, and may not match the order in which MS has listed the same in its troubleshooting document.

Previous articles of this series Intune PKI Made Easy With Joy

Let’s get started.

Demystifying Intune SCEP HTTP Errors

In order for an internet-facing device to send the SCEP request to NDES, the request must go via a proxy. In most setup, Azure AD App Proxy (Microsoft recommended) exposes the internal NDES mscep.dll URL.

So let’s begin with the HTTP errors that we may likely get due to Azure AD App Proxy.

Intune SCEP HTTP Errors – AAD App Proxy Errors

504 Gateway Timeout

This mostly occurs if the AAD App Proxy connector is not in Running state or the Server which hosts the connector has gone offline.

Intune SCEP HTTP Errors - AAD App Proxy related HTTP Errors - 504 Gateway Timeout - App Proxy Diagnostic Reports
Intune SCEP HTTP Errors – AAD App Proxy related HTTP Errors – 504 Gateway Timeout – App Proxy Diagnostic Reports

Startup Type of Microsoft AAD Application Proxy Connector service by default is set to Automatic (Delayed Start).

Demystify Intune SCEP HTTP Error - AAD App Proxy related HTTP Error - Connector Service Startup is Delayed Start by Default
Demystify Intune SCEP HTTP Error – AAD App Proxy related HTTP Error – Connector Service Startup is Delayed Start by Default

As such if the server recently went through a reboot cycle, ensure to check if the connector service has started or not.

502 Bad Gateway

This mostly occurs due to typo in the internal URL while creating/configuring the App Proxy application in Azure, as a result of which the connector fails to resolve the internal URL resulting in the error.

Intune SCEP HTTP Error - AAD App Proxy related HTTP Error - 502 Bad Gateway - App Proxy Diagnostic Reports
Intune SCEP HTTP Error – AAD App Proxy related HTTP Error – 502 Bad Gateway – App Proxy Diagnostic Reports

This page cannot be displayed/Can’t reach this page

Occurs due to incorrect App Proxy configuration for publishing the SCEP URL.

If you choose to use your custom verified domain in Azure as the mscep URL domain rather than the default “msappproxy.net” domain, you would require to have a CNAME record created in the DNS provider for the custom domain to map the requests.

By default, Azure App Proxy requires no additional configuration for using the default “-.msappproxy.net” domain. Also, if you choose to use a custom domain other than the default “-msappproxy.net” domain, you would need to provide an SSL cert.

In this case, I went with the default config to save myself from the extra work, but still getting the error. Generating a Diagnostics for the App Proxy application in Azure tells the error source.

Intune SCEP HTTP Error - AAD App Proxy related HTTP Error - This page cannot be displayed - App Proxy Diagnostic Reports
Intune SCEP HTTP Error – AAD App Proxy related HTTP Error – This page cannot be displayed – App Proxy Diagnostic Reports

While configuring the App Proxy application, I left the Pre Authentication set to Azure Active Directory.

Intune SCEP HTTP Errors - AAD App Proxy related HTTP Error - This page cannot be displayed - Pre Authentication to be set to Passthrough instead of Azure Active Directory
Intune SCEP HTTP Errors – AAD App Proxy related HTTP Error – This page cannot be displayed – Pre Authentication to be set to Passthrough instead of Azure Active Directory

For NDES, Pre Authentication setting within AAD App Proxy application in Azure needs to be set to Passthrough since no actual AAD user will be accessing the application.

Provided that we are good from the proxy side and AAD App Proxy is able to resolve the external requests to the internal NDES MSCEP URL, the HTTP errors we will encounter from now onwards are all dependant on the physical infrastructure settings (NDES and PKI).

Intune SCEP HTTP Errors – Configuration Issues

Intune SCEP HTTP Error 500 – Internal Server Error

There can be several causes for this particular error and you would need to check several things to narrow down to the real cause.

NDES Service Account permission/rights issue

The SCEP pool in IIS runs under the NDES service account identity.

Intune SCEP Error - HTTP Error 500 - IIS SCEP Pool runs under NDES Service Account identity - Ensure NDES Service Account password is not expired and account is in unlocked state.
Intune SCEP Error – HTTP Error 500 – IIS SCEP Pool runs under NDES Service Account identity – Ensure NDES Service Account password is not expired and account is in unlocked state.

You need to check and ensure that the NDES Service Account

  • is not in Locked state,
  • password is not Expired,
  • is a member of the local IIS_IURS group on the NDES box (ensure there is no GPO which is modifying the membership of the particular group),
  • the IIS_IURS group is assigned the Impersonate a client after authentication user right (by default the right is present unless modified),
Intune SCEP HTTP Error - IIS_IURS group should have the Imperosnate a client after authentication user rights assigned.
Intune SCEP HTTP Error – IIS_IURS group should have the Imperosnate a client after authentication user rights assigned.
  • has Read permission to the MSCEP RA certs private key
Intune SCEP HTTP Error - NDES Service Account must have Read permission for the MSCEP RA Cert private keys (Both CEP Encryption and Exchange Enrollment Agent)
Intune SCEP HTTP Error – NDES Service Account must have Read permission for the MSCEP RA Cert private keys (Both CEP Encryption and Exchange Enrollment Agent)

If the above conditions are not satisfied, you will have the Intune SCEP HTTP Error 500 – Internal Server Error.

MSCEP RA Certificates are Expired (or Deleted, or Revoked)

NDES service startup depends on the MSCEP RA Certificates. [More of NDES service startup sequence is discussed later in details in this post]

Intune SCEP HTTP Error - Ensure the MSCEP RA Certificates (CEP Encryption and Exchange Enrollment Agent) are not in expired state.
Intune SCEP HTTP Error – Ensure the MSCEP RA Certificates (CEP Encryption and Exchange Enrollment Agent) are not in expired state.

If the certificates are in expired state, or was deleted, or revoked from the issuing CA for any causes, NDES service will fail to start resulting the Intune SCEP HTTP Error 500 – Internal Server Error.

CRP in IIS has Windows Authentication set to Enabled

Ensure CertificateRegistrationSvc in IIS [CRP] has Windows Authentication set to Disabled.

Intune SCEP Error - HTTP Error 500 - CertiticateRegistrationSvc [CRP] in IIS has Windows Authentication Enabled?
Intune SCEP Error – HTTP Error 500 – CertiticateRegistrationSvc [CRP] in IIS has Windows Authentication Enabled?

If it is set to Enabled as shown in the above snapshot, it may result in Intune SCEP HTTP Error 500 – Internal Server Error.

Note! This particular settings may also result in HTTP Error 503 – Service Unavailable.

Just like /certsrv/mscep app in SCEP pool, IIS authentication for CRP also needs to have

  • Anonymous Authentication = Enabled
  • Windows Authentication = Disabled
Issuing CA is Offline or Unreachable from NDES

*********One of the most notable causes of Intune SCEP HTTP Error 500 – Internal Server Error.*********

Network configuration changes done by the network/firewall/proxy team may result in the Issuing CA to become unreachable or unavailable from the NDES box resulting in the error. Error events can be view from events as below.

Event Source: Windows Logs > Application > NetworkDeviceEnrollmentService
Event ID 8 - The Network Device Enrollment Service cannot retrieve information about the certification authority (0x80004005).  Unspecified error
Event ID 2 - The Network Device Enrollment Service cannot be started (0x80004005). Unspecified error

You can use certutil command on NDES server to verify CA availability.You should expect an output as below if your Issuing CA is not reachable or unavailable from the NDES box.

PS C:\Windows\system32> certutil -config - -ping
WIN-2CQQDB9STE6.joymalya.xyz\joymalya-WIN-2CQQDB9STE6-CA
Connecting to WIN-2CQQDB9STE6.joymalya.xyz\joymalya-WIN-2CQQDB9STE6-CA ...
Server could not be reached: The RPC server is unavailable. 0x800706ba (WIN32: 1722 RPC_S_SERVER_UNAVAILABLE) -- (32ms)
CertUtil: -ping command FAILED: 0x800706ba (WIN32: 1722 RPC_S_SERVER_UNAVAILABLE)
CertUtil: The RPC server is unavailable.

Same can be verified using another tool named pkiview.msc

Intune SCEP Error - HTTP Error 500 - pkiview.msc - CA Offline or unavailable
Intune SCEP Error – HTTP Error 500 – pkiview.msc – CA Offline or unavailable

Sometimes, for a multi-tier PKI setup, it happens that Issuing CA Server is online but Certificate Services of Issuing CA fails to start post a server restart [legit causes] due to Root CA CRL has expired.

Consider the below classic example where when I do a normal ping, my CA server returns response means its online and available, but a certutil ping to same CA server says otherwise.

PS C:\Windows\system32> ping WIN-2CQQDB9STE6.joymalya.xyz
 
Pinging WIN-2CQQDB9STE6.joymalya.xyz [10.0.0.20] with 32 bytes of data:
Reply from 10.0.0.20: bytes=32 time<1ms TTL=128
Reply from 10.0.0.20: bytes=32 time=1ms TTL=128
Reply from 10.0.0.20: bytes=32 time=1ms TTL=128
Reply from 10.0.0.20: bytes=32 time=1ms TTL=128
 
Ping statistics for 10.0.0.20:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 1ms, Average = 0ms
 
PS C:\Windows\system32> certutil -ping WIN-2CQQDB9STE6.joymalya.xyz
Connecting to WIN-2CQQDB9STE6.joymalya.xyz ...
Server could not be reached: The RPC server is unavailable. 0x800706ba (WIN32: 1722 RPC_S_SERVER_UNAVAILABLE) -- (32ms)
 
CertUtil: -ping command FAILED: 0x800706ba (WIN32: 1722 RPC_S_SERVER_UNAVAILABLE)
CertUtil: The RPC server is unavailable.

This can happen for multi-tier PKI infra where the Root CA is mostly offline and Root CA CRL has expired. The Issuing CA went through a reboot due to any causes. Post reboot, the Certificate Services of Issuing CA will fail to start as when it tries to retrieve Root CA CRL to verify its own cert chain, the verify operation fails.

Intune SCEP Errors - HTTP Error 500 - pkiview.msc - Root CA CRL expired causing Issuing CA certificate services to stop post server restart
Intune SCEP Errors – HTTP Error 500 – pkiview.msc – Root CA CRL expired causing Issuing CA certificate services to stop post server restart

Fixing the issue might require engagement from both the Network team and/or ADCS team as this does not generally fall within the Intune scope.

NDES unable to retrieve CRL from CDP

******This is the other most prominent cause of Intune SCEP HTTP Error 500 – Internal Server Error******

This occurs due to the NDES box failing to retrieve the Certificate Revocation List (CRL from CDP) which is required for the NDES service to start itself.

Remember NDES is implemented as ISAPI extension in IIS, as such you will not see NDES as a service when you check in services.msc. Instead the service runs within the w3wp.exe process which is an IIS worker process.

Now for the SCEP pool in IIS to run, the NDES service must first start successfully. You can get from here the NDES service startup checklist.

In a nutshell, NDES Service Startup Sequence is –

  • Locate RA cert from machine cert store – X509Objects
  • Acquire the Private Keys – CryptAcquireCertificatePrivateKey
  • Build cert chain – CertGetCertificateChainStart
  • Verify Revocation Status – CertVerifyRevocationStart

If all the steps complete successfully, the NDES service starts and you get the success events as below

Event Source: Windows Logs > Application > NetworkDeviceEnrollmentService
Event ID 47 - The Network Device Enrollment Service loaded the Registration Authority (RA) key exchange certificate with serial number ###### from the "MY" store.
Event ID 48 - The Network Device Enrollment Service loaded the Registration Authority (RA) signature certificate with serial number ###### from the "MY" store.
Event ID 1 - The Network Device Enrollment Service started successfully.

However, NDES service fails to start if it fails to verify the MSCEP-RA certificates. Error events as below.

Event Source: Windows Logs > Application > NetworkDeviceEnrollmentService
Event ID 8 - The Network Device Enrollment Service cannot retrieve information about the certification authority (0x80004005).  Unspecified error
Event ID 2 - The Network Device Enrollment Service cannot be started (0x80004005). Unspecified error

The event as above does not necessarily means that the MSCEP RA certs are expired or not present (deleted), but can also be because of Expired CRL or NDES server unable to retrieve the CRL, though the events as above do not specifically state such.

To check the actual reason for failure to load the RA certs, you need to enable CAPI2 logging in events [Applications and Services logs > Microsoft > Windows > CAPI2], then restart the Cryptographic Services and IIS [PS command iisreset] and check back in the CAPI2 Operational logs.

CAPI 2  Event Trace for same is shown below

Event ID 90 - X509 Objects - Loads RA cert
Event ID 70 - Acquire Certificate Private Key - Crypt Function CryptAcquireCertificatePrivateKey
Event ID 10 - Build Chain - Crypt Function CertGetCertificateChainStart to start Chain Building
Event ID 40 - Verify Revocation - Crypt Function CertVerifyRevocationStart to start revocation status
Event ID 52 - Retrieve Object from Network - Crypt Function CryptRetrieveObjectByUrlWireStart to retrieve CRL

If CA CRL has expired or CRL location is offline, the error events starts from here

Error Event ID 53 - Retrieve Object from Network - Crypt Function CryptRetrieveObjectByUrlWire which will report error if CRL not retrievable.

Error Event ID 42 - Reject Revocation Information - Crypt Function CertRejectedRevocationInfo. Here you can see that if CRL was actually retrieved, the failure reason might be due to Action CheckTimeValidity [retrieved CRL has expired]

Error Event ID 41 - Verify Revocation - Crypt Function CertVerifyRevocation will give you the outcome of revocation verification. 
Intune SCEP Errors - HTTP Error 500 - CAPI2 Looging - Unable to retrieve CRL from CDP - Revocation server was offline
Intune SCEP Errors – HTTP Error 500 – CAPI2 Looging – Unable to retrieve CRL from CDP – Revocation server was offline

In the NDES server, you can use certutil command certutil -urlcache CRL to view the locally cached CRL URLs and then use certutil command certutil -URL <LDAP or HTTP URL of CRL> to verify if the NDES server is able to retrieve the CRL from CDP.

You will see the URL Retrieval Tool GUI window like this. Select CRLs (from CDP) and click on Retrieve. For ideal scenario, the status comes as OK.

Intune SCEP Errors - HTTP Error 500 - Check if CRL is retriveable using CertUtil tool
Intune SCEP Errors – HTTP Error 500 – Check if CRL is retriveable using CertUtil tool

Important! If the communication between servers is facilitated by a proxy server, ensure that proxy settings in all the servers involved are configured for both user and machine/system context.

In the production environment, there may be a firewall in between or there may be a proxy that is facilitating the communications. For firewall, you need to ensure you have the proper exceptions set to allow connections. For proxy, you need to set proxy settings in the SYSTEM context for NDES to work. The best way is to use PSEXEC tool.

Intune SCEP HTTP Error - Use PSEXEC to confirm proxy settings at SYSTEM level.
Intune SCEP HTTP Error – Use PSEXEC to confirm proxy settings at SYSTEM level.

Open a CMD using PSEXEC and confirm the CMD process is running as SYSTEM using command whoami. Once confirmed, you can then use netsh winhttp show proxy to view if the proxy settings are correctly configured as SYSTEM context or if not, use netsh winhttp set proxy command to set the same.

Fixing issues with retrieving CRL or expired CRL mostly should fall under ADCS scope and not Intune scope in general.

Once the CRL issue is resolved, on your NDES server, restart the Cryptographic Services and IIS and wait to check if NDES services started successfully. [Event ID 1 for the successful startup of NDES]

Intune SCEP Http Error 503 – Service Unavailable

This one is pretty straightforward in most cases you will find the IIS SCEP pool has crashed (stopped stated). The likely causes are

Untrusted Intermediate Certificates in NDES server cert store

This is very common for multi-tier PKI infrastructure. Hence, if you are facing this issue, the first thing you should check on your NDES box is to open PowerShell and run the following command

Get-Childitem -Path cert:\LocalMachine\root -Recurse | Where-Object {$_.Issuer -ne $_.Subject}

If the above code gives any output, means there are untrusted certs present and you can simply delete them by running the below command

Get-Childitem -Path cert:\LocalMachine\root -Recurse | Where-Object {$_.Issuer -ne $_.Subject}| Remove-Object

Note! If you face this issue frequently, consider creating a scheduled task to run this command automatically at regular intervals to get rid of the untrusted certificates.

  • IIS SSL binding certificate was renewed but NDES Policy module still points to the old SSL certificate thumbprint

Remember the certificate used to bind port 443 of IIS is the same that needs to be selected while installing the Intune NDES certificate connector. As such, this situation arises mostly when the IIS SSL Binding Certificate gets renewed but the same is not updated with the Intune NDES Certificate Connector.

Intune SCEP HTTP Error 503 - IIS SSL Binding Cert thumbprint must match with the NDES Policy Module Cert thumbprint
Intune SCEP HTTP Error 503 – IIS SSL Binding Cert thumbprint must match with the NDES Policy Module Cert thumbprint

You must ensure that the thumbprint of the certificate held by the NDES Policy Module must match the thumbprint of the SSL binding certificate used in IIS.

Intune SCEP HTTP Error 503 - IIS SSL Binding Cert thumbprint must match with the NDES Policy Module Cert thumbprint
Intune SCEP HTTP Error 503 – IIS SSL Binding Cert thumbprint must match with the NDES Policy Module Cert thumbprint

Intune NDES Connector service requires to access the URLs in CRL for proper functioning. As such, post successful NDES service startup, if for any reason, the CRL URLs becomes unreachable again from the NDES box, may result in HTTP Error 503 – Service Unavailable.

Notice that this time, NDES service itself has started, otherwise, the same scenario during NDES service startup results in HTTP Error 500 – Internal Server Error.

The End

In addition to all the Intune SCEP HTTP Errors as listed above, you may also encounter HTTP Error 414 – Request URI Too Long and HTTP Error 413 – Payload Too Large which are all due to incorrect IIS Request Filtering settings for the /certsrv/mscep virtual app.

The causes listed for the errors presented in this article may not be exhaustive and while troubleshooting you may encounter a completely new cause which is quite possible. I would be very much interested to learn if you have discovered any such new cause for an error listed above while troubleshooting in your environment.

Well that was all in store for today. Will be back with some other post shortly. Till then, continue to stay safe…

4 COMMENTS

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.