Troubleshoot SCCM Fast Channel Push Notification Issues Configuration Manager ConfigMgr

Troubleshoot SCCM Fast Channel Push Notification Issues Configuration Manager ConfigMgr? The fast channel notification feature is there in SCCM products since 2012 SP1. SCCM fast channel notification was mainly used to notify clients about vital policies, collect inventories, etc.

You can refer to SCCM CB Run PowerShell Script Directly from the Collection post to get more details about the run PowerShell script option.

SCCM CB 1706 introduced the “RUN Script” option through the fast channel push notification. In this post, we will go through a video guide to troubleshoot SCCM CB fast channel push notification issues. Video tutorial about SCCM CB fast channel push notification is here.

What is SCCM Fast Channel Push Notification?

Let’s understand Fast channel notifications for clients. SCCM Fast channel notification is a “PUSH” method of notifying clients about the new policies. This communication channel used for SCCM client fast notification is via TCP (port 10123) or HTTP (port 80).

SCCM client communicates to its MP every 15 minutes to confirm it’s still online. In simple words, when your client is NOT showing as ONLINE in the SCCM console then, we may have a problem with the fast notification communication channel.

SCCM Push Vs Pull

Historically, SCCM uses the PULL method, and it expects the client to ask for new policies at regular intervals. But, the fast channel uses the PUSH method. What is BGB in SCCM? BGB = Fast Channel Notification related components. I don’t know whether this notification channel was codenamed as “Big Green Button” or not 😉

What are the components of SCCM CB Fast Channel Notification?

There are three components in SCCM CB fast channel notification. Notification manager will be located along with site servers (Primary/Secondary). It generates “push messages” for clients and sends notification to BGB server (MP) and stores the results.  Notification manager initiates push notifications from site server. BGBmgr.log is the log file which will give more details about notification manager. Notification files (*.BOS files) will be stored in INBOX/BGB.box folder. You can see the BOS file getting created in the video tutorial here.

As you can see in the following fast channel notification architecture diagram, when primary server has MP component then, notification manager and notification server will be there in that primary server.

Notification server will be located along with Management Point (MP) and in secondary sites. Notification server will have TCP and HTTP listeners. This will help listen to PUSH notifications from notification manager (DB), and it also confirms the client ONLINE status. Notification manager pushes result files (*.BTS) from clients. BGBServer.log is the log file on the MP setup directory or site server setup directory.

Notification Agent is fast channel notification component at SCCM client end. It’s part of SMS agent (CCMEXEC). Fast channel notification agent in SCCM client establishes a persistent connection with its notification server, and this will receive the PUSH messages from MP. CcmNotificationAgent.log is the log file on the SCCM client device. MP/Notification server communication errors can be noted in the log.

What is the architecture flow of SCCM CB Fast channel push notification?

Why is SCCM CB client NOT showing as ONLINE?

The problem statement here is that SCCM CB client is not showing as ONLINE in the console. Rather, it always stays OFFLINE. The problem is ONLY with FAST notification channel communication, and normal deployments + policies are working fine.

Troubleshooting of SCCM CB Fast Channel Notification

First of all, you need to make sure all the notification components are installed correctly on the server and client side. This can be confirmed using the following log files.

For installation issues troubleshooting :- SCCM CB Notification Server/Manager
BGBServer.log
BgbHttpProxy.log
BgbSetup.log
BGBisapiMSI.log

Fast Channel Notification – Server side troubleshooting

Checked the log files on my primary and MP (both are on the same server) and BGBServer.log shows a warning all the time “WARNING: Notification Server (%systemroot%\system32\dllhost.exe) with TCP port 10123 is NOT allowed by Windows Firewall on all interfaces”. But, I thought it should work with port 80 HTTP channel. It was not working as expected.

Following are the extracts of troublesome logs on BGB notification server.  BGBServer.log :-

Starting SMS Notification Server...~~

lt;SMS_NOTIFICATION_SERVER><08-15-2017 01:16:02.005-330><thread=3088 (0xC10)> Server GC is OFF~~

lt;SMS_NOTIFICATION_SERVER><08-15-2017 01:16:02.006-330><thread=3088 (0xC10)> Trigger to start TCP listener~~

lt;SMS_NOTIFICATION_SERVER><08-15-2017 01:16:02.022-330><thread=3088 (0xC10)> The HTTP listener is started~~

lt;SMS_NOTIFICATION_SERVER><08-15-2017 01:16:02.030-330><thread=3088 (0xC10)> Listening connections on port 10123. Waiting for clients to connect...~~

lt;SMS_NOTIFICATION_SERVER><08-15-2017 01:16:02.051-330><thread=1968 (0x7B0)> STATMSG: ID=9807 SEV=I LEV=M SOURCE="SMS Server" COMP="SMS_NOTIFICATION_SERVER" SYS=SCCMTP1.INTUNE.COM SITE=TP1 PID=3280 TID=1968 GMTDATE=Mon Aug 14 19:46:02.059 2017 ISTR0="SCCMTP1.INTUNE.COM" ISTR1="10123" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=0

lt;SMS_NOTIFICATION_SERVER><08-15-2017 01:16:02.061-330><thread=1968 (0x7B0)> WARNING: Notification Server (%systemroot%\system32\dllhost.exe) with TCP port 10123 is NOT allowed by Windows Firewall on all interfaces.~~

lt;SMS_NOTIFICATION_SERVER><08-15-2017 01:16:02.062-330><thread=5144 (0x1418)> Total online clients: 0 (TCP: 0 HTTP: 0)~~

lt;SMS_NOTIFICATION_SERVER><08-15-2017 01:21:02.039-330><thread=2924 (0xB6C)> Generated BGB online status FULL report C:\Program Files\Microsoft Configuration Manager\inboxes\bgb.box\Bgb72ul2.BOS (version: 0) at 08/15/2017 01:21:02~~

lt;SMS_NOTIFICATION_SERVER><08-15-2017 01:21:02.055-330><thread=2924 (0xB6C)> WARNING: Notification Server (%systemroot%\system32\dllhost.exe) with TCP port 10123 is NOT allowed by Windows Firewall on all interfaces.~~

lt;SMS_NOTIFICATION_SERVER><08-15-2017 01:21:02.067-330><thread=3540 (0xDD4)> Wait 300 seconds for notifications...

lt;SMS_NOTIFICATION_SERVER><08-15-2017 01:21:02.276-330><thread=7144 (0x1BE8)>

Client side troubleshooting Fast Channel Notification component

Notification agent was running. But, the  CcmNotificationAgent.log log showed loads of errors. One of the error was giving a very clear indication that there could be a communication issue between server and client.

Error 10060 means ==> A connection attempt failed because the connected party did not properly respond after a period, or established connection failed because connected host has failed to respond. BGBAgent component log :-

<![LOG[Bgb client agent is starting...]LOG]!><time="01:23:55.212-330" date="08-15-2017" component="BgbAgent" context="" type="1" thread="6372" file="agentendpoint.cpp:238">
<![LOG[BgbController main thread is started with settings: {bgb enable = 1}, {tcp enabled = 1}, {tcp port = 10123} and {http enabled = 1}.]LOG]!><time="01:23:55.259-330" date="08-15-2017" component="BgbAgent" context="" type="1" thread="6372" file="bgbcontroller.cpp:126">
<![LOG[Startup random sleep for 1 seconds.]LOG]!><time="01:23:55.290-330" date="08-15-2017" component="BgbAgent" context="" type="1" thread="5200" file="bgbcontroller.cpp:416">
<![LOG[Critical Battery: [FALSE]]LOG]!><time="01:23:56.306-330" date="08-15-2017" component="BgbAgent" context="" type="1" thread="5200" file="bgbcommon.cpp:60">
<![LOG[Connection Standy: [FALSE]]LOG]!><time="01:23:56.306-330" date="08-15-2017" component="BgbAgent" context="" type="1" thread="5200" file="bgbcommon.cpp:61">
<![LOG[Network allowed to use: [TRUE]]LOG]!><time="01:23:56.306-330" date="08-15-2017" component="BgbAgent" context="" type="1" thread="5200" file="bgbcommon.cpp:62">
<![LOG[Access point is SCCMTP1.INTUNE.COM. (SSLEnabled = 0)]LOG]!><time="01:23:56.415-330" date="08-15-2017" component="BgbAgent" context="" type="1" thread="5200" file="bgbcontroller.cpp:276">
<![LOG[CRL Checking is Enabled.]LOG]!><time="01:23:56.431-330" date="08-15-2017" component="BgbAgent" context="" type="1" thread="5200" file="bgbcontroller.cpp:284">
<![LOG[Both TCP and http are enabled, let's try TCP connection first.]LOG]!><time="01:23:56.431-330" date="08-15-2017" component="BgbAgent" context="" type="1" thread="5200" file="bgbcontroller.cpp:792">
<![LOG[Connecting to server with IP: 20.20.20.22 Port: 10123 
]LOG]!><time="01:23:56.447-330" date="08-15-2017" component="BgbAgent" context="" type="1" thread="5200" file="bgbtcpclient.cpp:699">
<![LOG[Failed to connect to server with IP v4 address with error 10060. Try next IP...
]LOG]!><time="01:24:17.468-330" date="08-15-2017" component="BgbAgent" context="" type="1" thread="5200" file="bgbtcpclient.cpp:703">
<![LOG[Failed to signin bgb client with error = 80004005.]LOG]!><time="01:24:17.468-330" date="08-15-2017" component="BgbAgent" context="" type="3" thread="5200" file="bgbcontroller.cpp:635">
<![LOG[Connecting to server with IP: 20.20.20.22 Port: 10123 
]LOG]!><time="01:25:17.482-330" date="08-15-2017" component="BgbAgent" context="" type="1" thread="5200" file="bgbtcpclient.cpp:699">
<![LOG[Failed to connect to server with IP v4 address with error 10060. Try next IP...
]LOG]!><time="01:25:38.501-330" date="08-15-2017" component="BgbAgent" context="" type="1" thread="5200" file="bgbtcpclient.cpp:703">
<![LOG[Failed to signin bgb client with error = 80004005.]LOG]!><time="01:25:38.501-330" date="08-15-2017" component="BgbAgent" context="" type="3" thread="5200" file="bgbcontroller.cpp:635">
<![LOG[Fallback to HTTP connection.]LOG]!><time="01:25:38.501-330" date="08-15-2017" component="BgbAgent" context="" type="1" thread="5200" file="bgbcontroller.cpp:828">
[CCMHTTP] ERROR: URL=http://SCCMTP1.Intune.com/bgb/handler.ashx?RequestType=Continue, Port=80, Options=224, Code=0, Text=CCM_E_BAD_HTTP_STATUS_CODE
Successfully queued event on HTTP/HTTPS failure for server 'SCCMTP1.Intune.com'.
Failed to post continue request with error code 87d0027e.

Fix for SCCM CB Fast Channel Notification Issues

The Firewall port 10123 port was not opened between SCCM client and BGB server primary server. I ran the following command from client “Telnet 10123” and it didn’t work (the port was not opened). I checked the software and hardware firewalls on the server side and came to know that Windows Firewall was blocking the port communication 10123.

Disabled the Windows Firewall on notification server for testing and restarted the client agent services (SMS Agent) on the client machine. This helped to resolve the fast channel notification issue with SCCM CB environment.

In an ideal world, you should exclude/exempt the port 10123/80 from the hardware and software firewall between fast channel notification server and agent. This will help to resolve the issue.

More details are available in the video tutorial here

Server Side Logs – After successful Actions on Fast Channel Notification

Finished sending push task (PushID: 1 TaskID: 3) to 1 clients and Generated BGB online status DELTA report are two important lines of SCCM CB fast notification channel server log BGBServer.log.

C:\Program Files\Microsoft Configuration Manager\inboxes\bgb.box\Bgb7cbzg.BOS

Receiving message from queue timeout.~~

lt;SMS_NOTIFICATION_SERVER><08-24-2017 12:07:50.411-330><thread=6060 (0x17AC)> Retrieving push tasks from database...~~ 

lt;SMS_NOTIFICATION_SERVER><08-24-2017 12:07:50.412-330><thread=6060 (0x17AC)> Retrieving online resync flag from database...~~

lt;SMS_NOTIFICATION_SERVER><08-24-2017 12:11:50.914-330><thread=6040 (0x1798)> Total online clients: 1 (TCP: 1 HTTP: 0)~~

lt;SMS_NOTIFICATION_SERVER><08-24-2017 12:11:50.916-330><thread=6040 (0x1798)> Online/Offline clients since last successful report: 1~~

lt;SMS_NOTIFICATION_SERVER><08-24-2017 12:11:50.919-330><thread=6040 (0x1798)> Generated BGB online status DELTA report C:\Program Files\Microsoft Configuration Manager\inboxes\bgb.box\Bgb7cbzg.BOS (version: 18) at 08/24/2017 12:11:50~~

lt;SMS_NOTIFICATION_SERVER><08-24-2017 12:11:50.940-330><thread=6040 (0x1798)> Get one push message from database.~~ 

lt;SMS_NOTIFICATION_SERVER><08-24-2017 12:11:52.184-330><thread=6060 (0x17AC)> Starting to send push task (PushID: 1 TaskID: 3 TaskGUID: 3700E17A-4BDC-45C7-990E-EA26FF92E5BF TaskType: 4 TaskParam: ) to 1 clients with throttling (strategy: 1 param: 42)~~

lt;SMS_NOTIFICATION_SERVER><08-24-2017 12:11:52.209-330><thread=6060 (0x17AC)> Finished sending push task (PushID: 1 TaskID: 3) to 1 clients~~

lt;SMS_NOTIFICATION_SERVER><08-24-2017 12:11:52.214-330><thread=6060 (0x17AC)> Starting to send push task (PushID: 1 TaskID: 4 TaskGUID: FD80647D-9748-4C96-AFC8-8BC71E00C235 TaskType: 1 TaskParam: ) to 1 clients with throttling (strategy: 1 param: 42)~~

lt;SMS_NOTIFICATION_SERVER><08-24-2017 12:11:52.216-330><thread=6060 (0x17AC)> Finished sending push task (PushID: 1 TaskID: 4) to 1 clients~~

lt;SMS_NOTIFICATION_SERVER><08-24-2017 12:11:52.237-330><thread=6060 (0x17AC)> Starting to send push task (PushID: 1 TaskID: 5 TaskGUID: 9D2B274B-F6E3-452F-A1C3-C1C166523EC8 TaskType: 1 TaskParam: ) to 1 clients with throttling (strategy: 1 param: 42)~~

lt;SMS_NOTIFICATION_SERVER><08-24-2017 12:11:52.238-330><thread=6060 (0x17AC)> Finished sending push task (PushID: 1 TaskID: 5) to 1 clients~~

lt;SMS_NOTIFICATION_SERVER><08-24-2017 12:11:52.261-330><thread=6060 (0x17AC)> Starting to send push task (PushID: 1 TaskID: 6 TaskGUID: CFB76FC4-BCC5-4525-AA53-80BCD4393E46 TaskType: 1 TaskParam: ) to 1 clients with throttling (strategy: 1 param: 42)~~

lt;SMS_NOTIFICATION_SERVER><08-24-2017 12:11:52.262-330><thread=6060 (0x17AC)> Finished sending push task (PushID: 1 TaskID: 6) to 1 clients~~

lt;SMS_NOTIFICATION_SERVER><08-24-2017 12:11:52.285-330><thread=6060 (0x17AC)> Starting to send push task (PushID: 1 TaskID: 7 TaskGUID: 1ED88E07-5E52-44FD-AF07-73769EDD7FA6 TaskType: 1 TaskParam: ) to 1 clients with throttling (strategy: 1 param: 42)~~

lt;SMS_NOTIFICATION_SERVER><08-24-2017 12:11:52.286-330><thread=6060 (0x17AC)> Finished sending push task (PushID: 1 TaskID: 7) to 1 clients~~

lt;SMS_NOTIFICATION_SERVER><08-24-2017 12:11:52.309-330><thread=6060 (0x17AC)>

Client BGB Agent Fast Notification:- Log files details of successful operations:-

After the Firewall ports changes the client notification agent started working fine. Following are some of the important log file snippets.

<![LOG[Bgb client agent is starting...]LOG]!><time="12:07:25.115-330" date="08-24-2017" component="BgbAgent" context="" type="1" thread="5368" file="agentendpoint.cpp:238">
<![LOG[BgbController main thread is started with settings: {bgb enable = 1}, {tcp enabled = 1}, {tcp port = 10123} and {http enabled = 1}.]LOG]!><time="12:07:25.162-330" date="08-24-2017" component="BgbAgent" context="" type="1" thread="5368" file="bgbcontroller.cpp:126">
<![LOG[Startup random sleep for 23 seconds.]LOG]!><time="12:07:25.178-330" date="08-24-2017" component="BgbAgent" context="" type="1" thread="3864" file="bgbcontroller.cpp:416">
<![LOG[Critical Battery: [FALSE]]LOG]!><time="12:07:48.178-330" date="08-24-2017" component="BgbAgent" context="" type="1" thread="3864" file="bgbcommon.cpp:60">
<![LOG[Connection Standy: [FALSE]]LOG]!><time="12:07:48.178-330" date="08-24-2017" component="BgbAgent" context="" type="1" thread="3864" file="bgbcommon.cpp:61">
<![LOG[Network allowed to use: [TRUE]]LOG]!><time="12:07:48.178-330" date="08-24-2017" component="BgbAgent" context="" type="1" thread="3864" file="bgbcommon.cpp:62">
<![LOG[Access point is SCCMTP1.Intune.com. (SSLEnabled = 0)]LOG]!><time="12:07:48.178-330" date="08-24-2017" component="BgbAgent" context="" type="1" thread="3864" file="bgbcontroller.cpp:276">
<![LOG[CRL Checking is Enabled.]LOG]!><time="12:07:48.178-330" date="08-24-2017" component="BgbAgent" context="" type="1" thread="3864" file="bgbcontroller.cpp:284">
<![LOG[Both TCP and http are enabled, let's try TCP connection first.]LOG]!><time="12:07:48.178-330" date="08-24-2017" component="BgbAgent" context="" type="1" thread="3864" file="bgbcontroller.cpp:792">
<![LOG[Connecting to server with IP: 20.20.20.22 Port: 10123 
]LOG]!><time="12:07:48.194-330" date="08-24-2017" component="BgbAgent" context="" type="1" thread="3864" file="bgbtcpclient.cpp:699">
<![LOG[Handshake was successful
]LOG]!><time="12:07:49.270-330" date="08-24-2017" component="BgbAgent" context="" type="1" thread="3864" file="bgbtcpclient.cpp:495">
<![LOG[Pass verification on server certificate.]LOG]!><time="12:07:49.329-330" date="08-24-2017" component="BgbAgent" context="" type="1" thread="3864" file="bgbtcpclient.cpp:823">
<![LOG[NetworkInfo: IPAddress 20.20.20.23,fe80::b09e:95a3:172a:4212]LOG]!><time="12:07:49.438-330" date="08-24-2017" component="BgbAgent" context="" type="1" thread="3864" file="bgbconnector.cpp:124">
<![LOG[NetworkInfo: IPSubnet 255.0.0.0,64]LOG]!><time="12:07:49.438-330" date="08-24-2017" component="BgbAgent" context="" type="1" thread="3864" file="bgbconnector.cpp:147">
<![LOG[NetworkInfo: AccessMP SCCMTP1.Intune.com]LOG]!><time="12:07:49.471-330" date="08-24-2017" component="BgbAgent" context="" type="1" thread="3864" file="bgbconnector.cpp:155">
<![LOG[NetworkInfo: IsClientOnInternet 0]LOG]!><time="12:07:49.471-330" date="08-24-2017" component="BgbAgent" context="" type="1" thread="3864" file="bgbconnector.cpp:159">
<![LOG[Update the timeout to 900 second(s)]LOG]!><time="12:07:49.471-330" date="08-24-2017" component="BgbAgent" context="" type="1" thread="3864" file="bgbtcpclient.cpp:916">
<![LOG[Receive signin confirmation message from server, client is signed in.]LOG]!><time="12:08:01.062-330" date="08-24-2017" component="BgbAgent" context="" type="1" thread="3864" file="bgbconnector.cpp:221">
<![LOG[Receive task from server with pushid=1, taskid=3, taskguid=3700E17A-4BDC-45C7-990E-EA26FF92E5BF, tasktype=4 and taskParam=]LOG]!><time="12:11:52.227-330" date="08-24-2017" component="BgbAgent" context="" type="1" thread="3864" file="bgbconnector.cpp:312">
<![LOG[Receive task from server with pushid=1, taskid=4, taskguid=FD80647D-9748-4C96-AFC8-8BC71E00C235, tasktype=1 and taskParam=]LOG]!><time="12:11:52.248-330" date="08-24-2017" component="BgbAgent" context="" type="1" thread="3864" file="bgbconnector.cpp:312">
<![LOG[Receive task from server with pushid=1, taskid=5, taskguid=9D2B274B-F6E3-452F-A1C3-C1C166523EC8, tasktype=1 and taskParam=]LOG]!><time="12:11:52.264-330" date="08-24-2017" component="BgbAgent" context="" type="1" thread="3864" file="bgbconnector.cpp:312">
<![LOG[Successfully sent keep-alive message.]LOG]!><time="12:35:51.339-330" date="08-24-2017" component="BgbAgent" context="" type="1" thread="3864" file="bgbconnector.cpp:290">
<![LOG[Successfully sent keep-alive message.]LOG]!><time="12:50:51.356-330" date="08-24-2017" component="BgbAgent" context="" type="1" thread="3864" file="bgbconnector.cpp:290">

References :-

• Fast Channel Client Notification in SCCM  – Here
• Fast channel notification and MP replica issues – Here
• What’s New With ConfigMgr’s Client Notification Feature – Here