Let’s check another SCCM troubleshooting post. This post is related to hardware and software inventory failures and the resolution. What will you do if you get the following message “Error Verify Inventory Action Status Failed 80041003.”
TL;DR
Introduction
The error “Verify Inventory Action Status Failed 80041003” is the most common issue faced where inventory reporting task fails to complete successfully and inventory agent log prints above error. Because of this issue error, SCCM Client doesn’t send the inventory data and marks it unhealthy.
NOTE! – There are several posts related to this issue but today we are going to cover in-depth troubleshooting along with root cause.
Log Entries: More details about SCCM Logs to understand how to read log files efficiently. The following entries are from InventoryAgent.log ?
Failed to Process instances of CCM_System:0 Reporting: (80041003) Reading of reports failed CReportTask::CreateReport() Failed Reporting: Cycle failed:80041003 Inventory: Reporting task failed to completed successfully. No report will be sent
Scope
Let’s check more details about Error Verify Inventory Action Status Failed 80041003.
Scope: This work instruction applies to the ConfigMgr Delivery Team who does the SCCM admin tasks. It doesn’t cover other deployment team tasks.
Symptom: Inventory was failing with error “verify Inventory status failed”
Cause: Multiple suspended instances of WerFault.exe’s causing WmiPrvSE.exe to run out of ProcessLimitAllHosts Quota
Troubleshooting – Error Verify Inventory Action 80041003
Process Monitor
- We found we had 7 WMIPRVSE.EXE Process
- We had 25 WERFAULT.EXE
- We found Werfault.exe was loaded under WMIprvSE.Exe and total count was 32.
- As Job Limit is 32 we are not able to create new Process in Job as a result we get Access Denied
- As per Logs we are not able to IWbemServices::CreateInstanceEnum as limit of 32 is exhausted
More details – https://docs.microsoft.com/en-us/windows/win32/api/wbemcli/nf-wbemcli-iwbemservices-createinstanceenum?redirectedfrom=MSDN
IWbemServices::CreateInstanceEnum method
This is as per Event Logs
Log Name: Microsoft-Windows-WMI-Activity/Trace
Source: Microsoft-Windows-WMI-Activity
Date: 18-05-2016 17:25:43
Event ID: 11
Task Category: None
Level: Information
Keywords:
User: SYSTEM
Computer: DeepakLab.test.com
Description:
CorrelationId = {018F7C40-F800-0000-CBEA-74C68881D101}; GroupOperationId = 9587029; OperationId = 9587030; Operation = Start IWbemServices::CreateClassEnum - root\ccm\invagt : ; ClientMachine = DeepakLab.test.com ; User = Deepak\DeepakRai; ClientProcessId = 11112; NamespaceName = \\.\root\ccm\invagt
Event Xml:
Log Name: Microsoft-Windows-WMI-Activity/Trace
Source: Microsoft-Windows-WMI-Activity
Date: 18-05-2016 17:25:44
Event ID: 13
Task Category: None
Level: Information
Keywords:
User: SYSTEM
Computer: DeepakLab.test.com
Description:
Stop OperationId = 9587030; ResultCode = 0x0
Log Name: Microsoft-Windows-WMI-Activity/Trace
Source: Microsoft-Windows-WMI-Activity
Date: 18-05-2016 17:25:51
Event ID: 11
Task Category: None
Level: Information
Keywords:
User: SYSTEM
Computer: DeepakLab.test.com
Description:
CorrelationId = {018F7C40-F800-0000-CBEA-74C68881D101}; GroupOperationId = 9587029; OperationId = 9587032; Operation = Start IWbemServices::CreateInstanceEnum - root\ccm\invagt : InventoryActionStatus; ClientMachine = DeepakLab.test.com ; User = Deepak\DeepakRai ; ClientProcessId = 11112; NamespaceName = \\.\root\ccm\invagt
Log Name: Microsoft-Windows-WMI-Activity/Trace
Source: Microsoft-Windows-WMI-Activity
Date: 18-05-2016 17:25:51
Event ID: 13
Task Category: None
Level: Information
Keywords:
User: SYSTEM
Computer: DeepakLab.test.com
Description:
Stop OperationId = 9587032; ResultCode = 0x0
Log Name: Microsoft-Windows-WMI-Activity/Trace
Source: Microsoft-Windows-WMI-Activity
Date: 18-05-2016 17:25:51
Event ID: 12
Task Category: None
Level: Information
Keywords:
User: S-1-5-21-1315882459-817801392-1359842108-846669
Computer: DeepakLab.test.com
Description:
ProviderInfo for GroupOperationId = 9587029; Operation = Provider::CreateInstanceEnum - InventoryProvider : InventoryActionStatus; HostID = 5404; ProviderName = InventoryProvider; ProviderGuid = {EF9B9B0B-009A-41F5-AF21-6AD433868C4F}; Path = C:\WINDOWS\CCM\InventoryProvider.dll
Windows Error Reporting Service is Already Disabled via GPO
As a workaround>>We disabled WER settings as Windows Error Reporting Service is already disabled via GPO
https://msdn.microsoft.com/en-us/library/windows/desktop/bb513638(v=vs.85).aspx
WER Settings
- HKEY_CURRENT_USER\Software\Microsoft\Windows\Windows Error Reporting
- HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\Windows Error Reporting
- Disabled–>REG_DWORD–>1
- DontShowUI–>DontShowUI–>1
- https://technet.microsoft.com/en-us/library/cc754364(v=ws.11).aspx
Windows Error Reporting
- If needed we will use Image File Executions Options to cause Werfault.exe to be executed under a non-existent debugger(NUL). Since NUL can’t be found, the process fails to launch.
Reg add “HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\Werfault.exe” /v Debugger /t REG_SZ /d NUL /f
https://msdn.microsoft.com/en-us/library/a329t4ed(VS.71).aspx
Launching the Debugger Automatically
- If you can willing Reproduce the Issue, We can try capture data(iDNA, Full Memory Dump) and can try find Faulting process additionally we can try Clean boot, try uninstall 3rd Party Services, capture Procmon and Xfer with Queuing Dqueing flags.
Note–It seems like main problematic App is 32 Bit and not 64 Bit.
We can increase Values in WMI (This will require Reboot of machine)
Memory and Handle Quotas in the WMI Provider Service
- Windows 2008 R2 – WMI Quota issue caused by either a WMI leak or an application over allocating WMI memory
- We can try to address the underlying crash of wmiprvse.exe to prevent werfault from launching. It looks like we’ll need to monitor just 32-bit instances since werfault.exe is 32-bit when this happens.
- We need to Capture Complete Memory Dump of the machine at the time of an Issue. Currently machine exhibiting the problem will not be of much use as parent process is already terminated and child Process is in Suspended mode.
- Architecture changes in Windows 2012 R2 operating System seems to handle this issue differently.
Log Snippet for ETL File
[1] 0344.554C::05/18/2016-17:02:20.274 [xfiles] a51rep_cpp1183 CNamespaceHandle::ComputeKeyFromPath() - Path = \\DeepakLab\ROOT\CCM\InvAgt:__Win32Provider.Name="InventoryProvider" [1] 0344.554C::05/18/2016-17:02:20.274 [xfiles] a51rep_cpp1442 CNamespaceHandle::ParseKey() - Key = "InventoryProvider" 6224::05/18/2016-17:02:25.179 [coreprox] callsec_cpp1998 CWbemCallSecurity::SetThreadSecurity() - HR_Return_Failure == 0 [0] 1030.55E4::05/18/2016-17:02:25.179 [dll] ProvHost_cpp2059 CServerObject_Host::GetProcessIdentifier() - HR_Return_Failure == 0 [1] 0344.6224::05/18/2016-17:02:25.179 [dll] ProvHost_cpp1746 CServerObject_HostInterceptor::GetProcessIdentifier() - HR_Return_Failure == 0 [0] 0344.6224::05/18/2016-17:02:25.180 [dll] Globals_cpp963 CServerObject_HostInterceptor::FindHost() - HR_Return_Failure == 80041003 HRESULT ProviderSubSystem_Globals::AssignProcessToJobObject ( HANDLE a_Handle ) BOOL t_Status =:: AssignProcessToJobObject ( s_HostJobObject , a_Handle ) ; if ( t_Status == FALSE ) { t_Result = WBEM_E_ACCESS_DENIED ;---------------------------------------------------------> } [0] 0344.6224::05/18/2016-17:02:25.181 [dll] ProvHost_cpp1230 CServerObject_HostInterceptor::FindHost() - HR_Return_Failure == 80041003 [0] 0344.6224::05/18/2016-17:02:25.181 [dll] ProvHost_cpp346 CServerObject_HostInterceptor::CreateUsingToken() - HR_Return_Failure == 80041003 [0] 0344.6224::05/18/2016-17:02:25.181 [dll] ProvFact_cpp786 CServerObject_BindingFactory::Create() - HR_Return_Failure == 80041003 [0] 0344.6224::05/18/2016-17:02:25.181 [dll] ProvFact_cpp1924 CServerObject_BindingFactory::InternalGetProviderViaProxyRoute() - - CServerObject_BindingFactory::InternalGetProviderViaProxyRoute hr = 80041003 [0] 0344.6224::05/18/2016-17:02:25.181 [dll] ProvFact_cpp1926 CServerObject_BindingFactory::InternalGetProviderViaProxyRoute() - HR_Return_Failure == 80041003 [0] 0344.6224::05/18/2016-17:02:25.181 [dll] provwsv_cpp7318 CInterceptor_IWbemProvider::Shutdown() - LONG a_Flags = 0 ULONG a_MaxMilliSeconds = 0 IWbemContext *a_Context = 0000000000000000 [0] 0344.6224::05/18/2016-17:02:25.181 [dll] provwsv_cpp7396 CInterceptor_IWbemProvider::Shutdown() - HR_Return_Failure == 0 [0] 0344.1210::05/18/2016-17:02:25.181 [winmgmt2] winmgmt_cpp795 WaitingFunction() - Got a provider can unload event [0] 0344.6224::05/18/2016-17:02:25.181 [dll] ProvFact_cpp2249 CServerObject_BindingFactory::InternalFindProvider() - HR_Return_Failure == 80041003 [0] 0344.6224::05/18/2016-17:02:25.181 [coredll] wbemname_cpp648 CWbemNamespace::~CWbemNamespace() - [0] 0344.6224::05/18/2016-17:02:25.181 [xfiles] a51rep_cpp861 CNamespaceHandle::`scalar() - [0] 0344.6224::05/18/2016-17:02:25.181 [coredll] wbemname_cpp648 CWbemNamespace::~CWbemNamespace() - [0] 0344.6224::05/18/2016-17:02:25.181 [xfiles] a51rep_cpp861 CNamespaceHandle::`scalar() - [0] 0344.6224::05/18/2016-17:02:25.181 [dll] ProvFact_cpp2614 CServerObject_BindingFactory::GetProvider() - - CServerObject_BindingFactory::GetProvider hr = 80041003 [0] 0344.6224::05/18/2016-17:02:25.181 [dll] ProvFact_cpp2615 CServerObject_BindingFactory::GetProvider() - HR_Return_Failure == 80041003 [0] 0344.6224::05/18/2016-17:02:25.181 [coredll] wbemname_cpp6449 CWbemNamespace::Exec_DynAux_GetInstances() - HR_Failure(80041003) [0] 0344.6224::05/18/2016-17:02:25.181 [coredll] internalmerger_cpp2162 CMergerTargetSink::SetStatus() - [0] 0344.6224::05/18/2016-17:02:25.181 [coredll] sinks_cpp327 CForwardingSink::SetStatus() - long a_Flags = 0x0 HRESULT a_Result = 0x80041003 Sink = 0000000004BA7CD0 BSTR a_StringParam = NULL IWbemClassObject *a_ObjectParam = 0000000000000000 [0] 0344.6224::05/18/2016-17:02:25.181 [coredll] sinks_cpp382 CCombiningSink::SetStatus() - long a_Flags = 0x0 HRESULT a_Result = 0x80041003 Sink = 00000000056D9AB0 BSTR a_StringParam = NULL IWbemClassObject *a_ObjectParam = 0000000000000000 [0] 0344.6224::05/18/2016-17:02:25.181 [coredll] mergerreq_cpp123 CMergerDynReq_DynAux_GetInstances::Execute() - HR_Return_Failure == 80041003 [0] 0344.6224::05/18/2016-17:02:25.181 [coreprox] FASTCLS_CPP5386 CWbemClass::GetProperty() - HR_Return_Failure == 0 [0] 0344.6224::05/18/2016-17:02:25.181 [coredll] coreq_cpp835 CCoreQueue::LogError() - Error 80041003 occured executing request for InventoryActionStatus
NOTE! – It looks like error is returned from the Windows API AssignProcessToJobObject
AssignProcessToJobObject function
Windows 7, Windows Server 2008 R2, Windows XP with SP3, Windows Server 2008, Windows Vista, and Windows Server 2003:
The process must not already be assigned to a job; if it is, the function fails with ERROR_ACCESS_DENIED. This behavior changed starting in Windows 8 and Windows Server 2012.
Resolution
All the logs and troubleshooting performed on Windows Server 2008 R2 SP1. Because of server OS you will not be able to recreate WMI Repository so we will process all the MOF’s available in “C:\Windows\System32\wbem”.
EXECUTE THIS SCRIPT AT YOUR OWN RISK. WE ARE NOT RESPONSIBLE IF ANYTHING BREAKS IN THIS SERVER AFTER EXECUTION.
Save it as BAT. Open CMD as Administrator and Navigate to the path where you saved it. Execute it and monitor what’s going on in the CMD Window. After few minutes when it’s done, trigger inventory and see if successful. Reboot the server once if still fails.
@echo off
sc config winmgmt start= disabled
net stop winmgmt /y
%systemdrive%
cd %windir%\system32\wbem
for /f %%s in ('dir /b *.dll') do regsvr32 /s %%s
wmiprvse /regserver
winmgmt /regserver
sc config winmgmt start= auto
net start winmgmt
for /f %%s in ('dir /s /b *.mof *.mfl') do mofcomp %%s
net start ccmexec