SCCM Possible MP Rotation Selection Forest Trust Related Bug

SCCM Possible MP Rotation Selection Forest Trust Related Bug. When you’ve ConfigMgr SCCM 2012 deployed in a very complex environment with more than 15 untrusted domains or forests and a 150K client base, you can help Microsoft find out more bugs :(.

A few days back, only Microsoft helped us fix Application contents duplicated in stand-alone media in SCCM ConfigMgr 2012 R2 – kb/2928122.

This post will walk through a possible bug in SCCM ConfigMgr 2012 R2 client’s MP selection criteria or preference. Every time, the CM12 client cannot identify its network location correctly (trusted AD forest details).

The latest update from Microsoft Support Person “The MPs we get from AD do not have the forest trust hint (ForestTrust’ N’). The forest trust hint is provided only when MPs are obtained from MP (ForestTrust ‘Y’).

Patch My PC

We would then randomize and persist these based on affinity rules after getting from both the AD and MP. So this explains some of the inconsistencies we see in the logs and the perceived instability.” 

I’ve also posted Workaround for Untrusted Forest SCCM 2012 MP Rotation Issue Workaround For Untrusted Forest SCCM MP Rotation Issue HTMD Blog (anoopcnair.com).

Scenario :- 

SCCM hierarchy (CAS + Primary servers) – ACNCMCAS (configmgr.com) & ACNCMPRI.ConfigMgr.com (ABC.ConfigMgr.com)

3 Remote MPs under Primary Server (located in 3 untrusted forest DMZ) – ACNCMMP1.XYZ.DMZ.com, ACNCMMP2.ABC.DMZ.com & ACNCMMP3. 

Adaptiva

Clients – Clients are separated within untrusted as mentioned above. The clients have access to respective MPs in their local forest. 

For example, clients in the forest one can access only one remote MP in its local forest  ACNCMMP1. The same applies to all the other clients in different forests.

Infra Diagram Fine MP Rotation SCCM Possible MP Rotation Selection Forest Trust Related Bug
MP Rotation SCCM Possible MP Rotation Selection Forest Trust Related Bug

AD publishing is enabled on the primary site, so all the MP details are published to all the 3 untrusted forests. When a client (location services) queries AD to get MP details, it will get all the 3 MPs details available in the Active Directory System Management container.

So SCCM ConfigMgr client 2012 client needs to select any one of the MPs from the list of 3 MPs. The MP selection in SCCM 2012 is different and complex compared to SCCM 2007.

SCCM ConfigMgr 2012 Client MP selection is random if a client has multiple MPs in the MP list. When we’ve numerous management points, then the client will automatically select one MP based on the following preferences:-

  1. Based on network location (Local AD Forest): We can check locationservices.log to check whether the client is able to identify the MP as local forest MP or not.

The client thinks MP is not in the same forest when we see  ForestTrust: ‘N’ in locationservices.log.

When we see   ForestTrust: ‘Y’ in locationservices.log, the client thinks the MP is in the same forest.

2. Communication (HTTPS or HTTP)

3. Random Selection of MP

In our scenario, When everything is OK, the local MP is reported as ForestTrust=’Y’, with the others as ForestTrust=’N.’ However, randomly, we see that all MPs are reported as ForestTrust=’N.’  

In this scenario, the client then tries to rotate around the MPs. SCCM Possible MP Rotation Selection Forest Trust Related Bug.

SCCM Possible MP Rotation Selection Forest Trust Related Bug. Sample Locationservices.log file for reference:-

Attempting to retrieve lookup MP(s) from AD LocationServices 2/5/2014 5:07:57 PM
Lookup Management Points from AD: LocationServices 2/5/2014 5:07:57 PM
Name: 'ACNCMMP1.XYZ.DMZ.com' HTTPS: 'N' ForestTrust: 'N' LocationServices 2/5/2014 5:07:57
Name: 'ACNCMMP2.ABC.DMZ.com' HTTPS: 'N' ForestTrust: 'N' LocationServices 2/5/2014 5:07:57
Retrieved lookup MP(s) from AD LocationServices 2/5/2014 5:07:57 PM
Refreshed security settings over AD LocationServices 2/5/2014 5:07:57 PM
Default Management Points from AD: LocationServices 2/5/2014 5:07:57 PM 3308 (0x0CEC)
Name: 'ACNCMMP1.XYZ.DMZ.com' HTTPS: 'N' ForestTrust: 'N' LocationServices 2/5/2014 5:07:57
Name: 'ACNCMMP2.ABC.DMZ.com' HTTPS: 'N' ForestTrust: 'N' LocationServices 2/5/2014 5:07:57
Persisting the default management points in WMI LocationServices 2/5/2014 5:07:57 PM
Current AD site of machine is Default-First-Site-Name LocationServices 2/5/2014 5:07:57 PM
Current AD site of machine is Default-First-Site-Name LocationServices 2/5/2014 5:07:57 PM
No security settings update detected. LocationServices 2/5/2014 5:07:57 PM 1472 (0x05C0)
Default Management Points from MP: LocationServices 2/5/2014 5:07:57 PM 3308 (0x0CEC)
Name: 'ACNCMMP1.XYZ.DMZ.com' HTTPS: 'N' ForestTrust: 'N' LocationServices 2/5/2014 5:07:57
Name: 'ACNCMMP2.ABC.DMZ.com' HTTPS: 'N' ForestTrust: 'Y' LocationServices 2/5/2014 5:07:57
Persisted Default Management Point Locations locally LocationServices 2/5/2014 5:07:57 PM
Current AD site of machine is Default-First-Site-Name LocationServices 2/5/2014 5:07:57 PM
Attempting to retrieve local MPs from the assigned MP LocationServices 2/5/2014 5:07:57 PM
Current AD site of machine is Default-First-Site-Name LocationServices 2/5/2014 5:07:57 PM
Refreshing the Management Point List for site PR1 LocationServices 2/5/2014 5:07:57 PM
Retrieved management point encryption info from AD. LocationServices 2/5/2014 5:07:57 PM

Extracts from ClientLocation.log:-

Rotating assigned management point, new management point [1] is:ACNCMMP2.ABC.DMZ.com (7958) with capabilities: <Capabilities SchemaVersion="1.0"><Property Name="SSLState" Value="0"/></Capabilities>
Rotating assigned management point, new management point [2] is: ACNCMMP1.XYZ.DMZ.com (7958) with capabilities: <Capabilities SchemaVersion="1.0"><Property Name="SSLState" Value="0"/></Capabilities>
Assigned MP changed from <ACNCMMP2.ABC.DMZ.com> to <ACNCMMP1.XYZ.DMZ.com>.

SCCM ConfigMgr 2012 client is getting forest trust information (local network or AD forest), which we already saw in the location services log file, is based on the background check client does with AD or some other component. 

Randomly we can see that all MPs are reported as ForestTrust=’N’ in locationservices.log. Once all the MPs are reported as  ForestTrust=’N’, clients start rotating the MPs. So basically, the problem is every time client is NOT able to identify its network location correctly (trusted AD forest).

Because of this issue, TS will get timed out or fail. Also, some of the client machines cannot find the proper MP and cannot get the policies from MP; hence couldn’t run/deploy applications software center. SCCM Possible MP Rotation Selection Forest Trust Related Bug.  

Microsoft engineer who works, in this case, can reproduce the issue in his ab. The latest news from the Microsoft support engineer is this is expected behavior or by design (I would say this is bad design), and there won’t be any hotfix to f x this. Probably, we may need to wait until the next release 🙁

Author

Anoop C Nair is Microsoft MVP! He is a Device Management Admin with more than 20 years of experience (calculation done in 2021) in IT. He is Blogger, Speaker, and Local User Group HTMD Community leader. His main focus is on Device Management technologies like SCCM 2012, Current Branch, and Intune. He writes about ConfigMgr, Windows 11, Windows 10, Azure AD, Microsoft Intune, Windows 365, AVD, etc.

10 thoughts on “SCCM Possible MP Rotation Selection Forest Trust Related Bug”

  1. Hi Anoop, thanks for the info. Can you please tell me if you filed a bug with MS on this and they are working on it or there is a workaround to overcome a situation like this.

    Reply
    • Hi Sravanth ! – MS is not confirmed officially this as a bug till now. We’ve (not MS) already found some Workarounds (for the scenario which I explained in the post). Removing the published details from AD and publishing it in DNS etc.. I’ll blog about this at some point of time.

      Regards
      Anoop

      Reply
  2. Hi,

    I’m seeing this same problem.

    I never published details to AD.

    I have added the _mssms_mp_ entry to DNS which is correctly picked up during the client install.

    However it can then take a week for the client to finally stabilise on the local management point and then correctly download content.

    Are there any further steps required ? A minimum patch level ?

    Many thanks !

    Reply
    • Hi ! – This is interesting. after one week, the client is settling down with correct MP and it’s not rotating MPs? Is my understanding correct?

      Regards
      Anoop

      Reply
      • Hi Anoop,

        Thanks for getting back to me and thanks for documenting this problem. I’ve found very little on MP selection, only that first it shortlists a MP in the local forest, then prioritises HTTPS and then makes a selection. I haven’t found anything on what mechanism you have to set up so it can short list the local forest in the first place.

        When I install an agent it correctly connects to its local management point (in the same DMZ and forest) and obtains a list of all management points within the primary site. I’m specifying the local MP and DNS suffix as installer options and have put a service record for that MP in DNS as well.

        The agent then starts rotating through management points and can take a week to stabilise on its local MP and start downloading content. All MP’s are showing as ForestTrust: ‘N’ in the agent location services log.

        The agent has no network connection to MP’s outside the DMZ and cannot even resolve their names. I’m guessing the variable time delay is caused by a mixture of randomized MP selection and SCCM connection timeouts to external MP’s.

        I have not expanded the schema, have not published into AD, have not set up any kind of SCCM discovery for example Forest or System Discovery.

        In a previous reply you suggested a DNS publishing workaround for SCCM occasionally reporting ForestTrust:’N’. Am I missing any steps in what I’ve done so far ? Any high level steps on other approaches would also be really helpful.

        Alternatively do you know the steps required for the agent to be able to short list the local forest ? Do I need SCCM Discovery methods/expanded schema ? That may give better results even with the bug highlighted by your article.

        Many thanks,
        Fodder

      • Hi Fodder ! – It’s good that you’ve already published MP detailed into DNS and not published the details into DMZ AD. You’ll have one advantage when you publish MP details into DMZ AD. When your client receives MP list (all the MPs under the primary site) from the local forest MP, it may give preference to local forest MP because of ForestTrust: ‘Y’ entry. So (it seems to me) when you extend the schema and publish MP into DMZ AD then only the CM client can understand one MP is from it’s local forest.

        However, I never recommend to publish MP details to DMZ AD in your case. When you publish MP details then you will fall into the exact scenario which I mentioned in the post.

        Your current situation is better than mine because the client initially contacts to get MP list and it fails to get the list from AD. Next, client will check DNS for MP details and it will provide only one MP which is your local forest MP.

        In our case, we’ve published MP details into AD, so all the MP/s (under a primary server) details is already available in all the forests. SO whenever client contacts AD to fetch the MP details it will get all the MPs and the client will get confused which MP should I connect because when a client gets MP details from AD it won’t get ForestTrust: ‘Y’ information.

        The MPs we get from AD do not have the forest trust hint (ForestTrust ‘N’). The forest trust hint is provided only when MPs are obtained from MP (ForestTrust ‘Y’). We would then randomize and persist these based on affinity rules after getting from both the AD and then MP. So this explains some of the inconsistencies we see in the logs and the perceived instability.

        We’re planning to remove AD publishing and use DNS publishing in our scenario for better MP selection.

        Another few options we’ve is to use local host file entries for the MPs which are in remote forest. Probably, assign loop back IP for those MPs (this method is used in Robert Marshall tool SwitchMP more details https://www.anoopcnair.com/2014/01/28/sccm-tools-robert-marshall-mvp-custom-ddr-client-push-manager-component-manager-inbox-monitor/). By doing this the client may immediately reject that MP. Otherwise make host file entry to MAP wrong MPs to local forest MPs. 🙂

        I’ve not tested these options till now. Hopefully, we will get better workarounds from Microsoft.

        Regards
        Anoop

  3. Hi Anoop,

    DNS publishing does not seem to have helped the situation for me. It only appears to be one of many mechanisms to discover an initial MP.

    Once the agent has connected to the MP it still gets a list of all MP’s (including outside of the DMZ) even though I haven’t extended the schema or published into AD. Similarly (and possibly using the very same mechanism) http://localhost/sms_mp/.sms_aut?mplist returns all MP’s including outside the DMZ.

    The local MP is coming back ForestTrust: “N” suggesting I need to extend the schema and publish into AD to get it to come back ForestTrust: “Y”. However at that point we arrive at the bug this article highlighted with it occasionally returning incorrectly.

    I’m going to assign remote MP’s to the loopback address in the hosts file and see how that goes.

    Thanks for the leads.

    Regards,
    Fodder

    Reply
    • Hi Fodder ! – Got your point that DNS publishing alone won’t fix our problem 🙁 We were thinking to implement 1. Remove AD publishing 2.Publish DNS as workaround to this problem. From your experience that doesn’t make much sense either.

      Regards
      Anoop

      Reply
  4. Hello all, Same issue here with 5 untrusted domains , with an Mp in only one of these domains and 2 MPs in internal domain where sccm site is located .Firewall doesn’t allow the untrusted domains clients to talk with internal MPs, but only the one in untrusted domain. Ad affinity doesn’t work for the 4 domains withut any MP.

    Rotating to internal MPs issue was resolved by
    1 – not publish in AD
    2 – publish only the MP in untrusted domains
    3 – host file change on each client to redirect the internal MPs to the one in untrusted domain
    4 – manually approve the new clients in sccm consoles (auto approve only for trusted domains)

    Regards
    Eric

    this is a pain.

    Reply
    • Hi Eric ! – Thank you for the comment. It’s a pain. I can understand 🙁
      3. Host file change is also lot of work. isn’t it? How do you achieve this? Better to have some duplicate DNS records of other untrusted MPs pointing to local domain MP (IP) on local DNS server.
      4. This is loads of work. Would suggest to auto approve all clients.

      Regards
      Anoop

      Reply

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.