Advertisement

SCCM 2012 Possible MP Rotation and Selection Forest Trust Related Bug

When you’ve ConfigMgr SCCM 2012 deployed in very complex environment with more than 15 untrusted domains or forests and 150K client base, you can help Microsoft to find out more bugs :(. Few days back only Microsoft helped us to fix Application contents are duplicated in stand-alone media in SCCM ConfigMgr 2012 R2 – kb/2928122.

In this post, we are going walk through a possible bug in SCCM ConfigMgr 2012 R2 client’s MP selection criteria or preference. Every time, the CM12 client is NOT able to identify it’s network location correctly (trusted AD forest details).

The latest update from Microsoft Support Person “The MPs we get from AD do not have the forest trust hint (ForestTrust ‘N’). The forest trust hint is provided only when MPs are obtained from MP (ForestTrust ‘Y’). We would then randomize and persist these based on affinity rules after getting from both the AD and then MP. So this explains some of the inconsistencies we see in the logs and the perceived instability.” 

I’ve also posted Workaround for Untrusted Forest SCCM 2012 MP Rotation Issue http://anoopcnair.com/2014/04/11/workaround-sccm-2012-clients-mp-selection-rotation-issue-untrusted-dmz-forests/

Scenario :- 

SCCM hierarchy (CAS + Primary servers) – ACNCMCAS (configmgr.com)  & ACNCMPRI.ConfigMgr.com (ABC.ConfigMgr.com)

3 Remote MPs under Primary Server (located in 3 untrusted forest DMZ) – ACNCMMP1.XYZ.DMZ.com, ACNCMMP2.ABC.DMZ.com & ACNCMMP3

Clients – Clients are separated within untrusted as mentioned above. The clients are having access to respective MPs in its local forest. For example, clients in forest 1 can access only one remote MP in it’s local forest  ACNCMMP1. Same applicable for all the other clients in different forests.

Infra Diagram Fine

Deep dive into Issue 

AD publishing is enabled on the primary site, so all the MP details are published to all the 3 untrusted forest.  When a client (location services) queries AD to get MP details, it will get all the 3 MPs details available in the Active Directory System Management container.

So SCCM ConfigMgr client 2012 client needs to select anyone of the MPs from the list of 3 MPs. The MP selection in SCCM 2012 is bit different and complex when you compare it with SCCM 2007.

SCCM ConfigMgr 2012 Client MP selection is random if a client has multiple MPs in MP list. When we’ve multiple management points then the client will automatically select one MP based on following preferences :-

1. Based on network location (Local AD Forest): We can check locationservices.log to check whether the client is able to identify the MP as local forest MP or not.

When we see  ForestTrust: ‘N’  in locationservices.log then the client thinks MP is not in the same forest.

When we see   ForestTrust: ‘Y’ in locationservices.log then the client thinks that the MP is in the same forest.

2. Communication (HTTPS or HTTP)

3. Random Selection of MP

Following TechNet article  will give you more details – here (expand site system roles).

In our scenario, When everything is OK the local MP is reported as ForestTrust=’Y’, with the others as ForestTrust=’N’.  However randomly we see that all MPs are reported as ForestTrust=’N’.  In this scenario the client then tries to rotate around the MPs.

Sample Locationservices.log file for reference :-

Attempting to retrieve lookup MP(s) from AD LocationServices 2/5/2014 5:07:57 PM
Lookup Management Points from AD: LocationServices 2/5/2014 5:07:57 PM
Name: ‘ACNCMMP1.XYZ.DMZ.com’ HTTPS: ‘N’ ForestTrust: ‘N’ LocationServices 2/5/2014 5:07:57
Name: ‘ACNCMMP2.ABC.DMZ.com’ HTTPS: ‘N’ ForestTrust: ‘N’ LocationServices 2/5/2014 5:07:57
Retrieved lookup MP(s) from AD LocationServices 2/5/2014 5:07:57 PM
Refreshed security settings over AD LocationServices 2/5/2014 5:07:57 PM
Default Management Points from AD: LocationServices 2/5/2014 5:07:57 PM 3308 (0x0CEC)
Name: ‘ACNCMMP1.XYZ.DMZ.com’ HTTPS: ‘N’ ForestTrust: ‘N’ LocationServices 2/5/2014 5:07:57
Name: ‘ACNCMMP2.ABC.DMZ.com‘ HTTPS: ‘N’ ForestTrust: ‘N’ LocationServices 2/5/2014 5:07:57
Persisting the default management points in WMI LocationServices 2/5/2014 5:07:57 PM
Current AD site of machine is Default-First-Site-Name LocationServices 2/5/2014 5:07:57 PM
Current AD site of machine is Default-First-Site-Name LocationServices 2/5/2014 5:07:57 PM
No security settings update detected. LocationServices 2/5/2014 5:07:57 PM 1472 (0x05C0)
Default Management Points from MP: LocationServices 2/5/2014 5:07:57 PM 3308 (0x0CEC)
Name: ‘ACNCMMP1.XYZ.DMZ.com’ HTTPS: ‘N’ ForestTrust: ‘N’ LocationServices 2/5/2014 5:07:57
Name: ‘ACNCMMP2.ABC.DMZ.com‘ HTTPS: ‘N’ ForestTrust: ‘Y’ LocationServices 2/5/2014 5:07:57
Persisted Default Management Point Locations locally LocationServices 2/5/2014 5:07:57 PM
Current AD site of machine is Default-First-Site-Name LocationServices 2/5/2014 5:07:57 PM
Attempting to retrieve local MPs from the assigned MP LocationServices 2/5/2014 5:07:57 PM
Current AD site of machine is Default-First-Site-Name LocationServices 2/5/2014 5:07:57 PM
Refreshing the Management Point List for site PR1 LocationServices 2/5/2014 5:07:57 PM
Retrieved management point encryption info from AD. LocationServices 2/5/2014 5:07:57 PM

Extracts from ClientLocation.log :-

Rotating assigned management point, new management point [1] is:ACNCMMP2.ABC.DMZ.com (7958) with capabilities: <Capabilities SchemaVersion="1.0"><Property Name="SSLState" Value="0"/></Capabilities>
Rotating assigned management point, new management point [2] is: ACNCMMP1.XYZ.DMZ.com (7958) with capabilities: <Capabilities SchemaVersion="1.0"><Property Name="SSLState" Value="0"/></Capabilities>
Assigned MP changed from <ACNCMMP2.ABC.DMZ.com> to <ACNCMMP1.XYZ.DMZ.com>.

Bug

SCCM ConfigMgr 2012 client is getting forest trust information (local network or AD forest), which we already seen in the location services log file, is based on the background check client does with AD or some other component. Randomly we can see that all MPs are reported as ForestTrust=’Nin locationservices.log. Once all the MPs are reported as  ForestTrust=’N’ then clients start rotating the MPs. So basically, the problem is every time client is NOT able to identify it’s network location correctly (trusted AD forest). Because of this issue, TS will get timed out or failed. Also, some of client machines are not able find the proper MP and not able to get the policies from MP hence couldn’t run/deploy applications software center.  Microsoft engineer who works in this case is able reproduce the issue in his lab. The latest news from Microsoft support engineer is this is an expected behavior or by design (I would say this is bad design) and there won’t any hotfix to fix this. Probably, we may need to wait until next release 🙁

About Author 

Anoop is Microsoft MVP and Veeam Vanguard ! He is a Solution Architect on enterprise client management with more than 13 years of experience (calculation done on the year 2014) in IT. He is Blogger, Speaker and Local User Group Community leader. His main focus is on Device Management technologies like SCCM 2012,Current Branch, Intune. He writes about the technologies like SCCM, SCOM, Windows 10, Azure AD, Microsoft Intune, RMS, Hyper-V etc...

    Find more about me on:
  • googleplus
  • twitter
  • facebook
  • linkedin
  • youtube
Posted in: CM2012, ConfigMgr (SCCM), Configmgr2012

12 Comments

  1. sravanth says:

    Hi Anoop, thanks for the info. Can you please tell me if you filed a bug with MS on this and they are working on it or there is a workaround to overcome a situation like this.

    • Anoop says:

      Hi Sravanth ! – MS is not confirmed officially this as a bug till now. We’ve (not MS) already found some Workarounds (for the scenario which I explained in the post). Removing the published details from AD and publishing it in DNS etc.. I’ll blog about this at some point of time.

      Regards
      Anoop

  2. Fodder says:

    Hi,

    I’m seeing this same problem.

    I never published details to AD.

    I have added the _mssms_mp_ entry to DNS which is correctly picked up during the client install.

    However it can then take a week for the client to finally stabilise on the local management point and then correctly download content.

    Are there any further steps required ? A minimum patch level ?

    Many thanks !

    • Anoop says:

      Hi ! – This is interesting. after one week, the client is settling down with correct MP and it’s not rotating MPs? Is my understanding correct?

      Regards
      Anoop

      • Fodder says:

        Hi Anoop,

        Thanks for getting back to me and thanks for documenting this problem. I’ve found very little on MP selection, only that first it shortlists a MP in the local forest, then prioritises HTTPS and then makes a selection. I haven’t found anything on what mechanism you have to set up so it can short list the local forest in the first place.

        When I install an agent it correctly connects to its local management point (in the same DMZ and forest) and obtains a list of all management points within the primary site. I’m specifying the local MP and DNS suffix as installer options and have put a service record for that MP in DNS as well.

        The agent then starts rotating through management points and can take a week to stabilise on its local MP and start downloading content. All MP’s are showing as ForestTrust: ‘N’ in the agent location services log.

        The agent has no network connection to MP’s outside the DMZ and cannot even resolve their names. I’m guessing the variable time delay is caused by a mixture of randomized MP selection and SCCM connection timeouts to external MP’s.

        I have not expanded the schema, have not published into AD, have not set up any kind of SCCM discovery for example Forest or System Discovery.

        In a previous reply you suggested a DNS publishing workaround for SCCM occasionally reporting ForestTrust:’N’. Am I missing any steps in what I’ve done so far ? Any high level steps on other approaches would also be really helpful.

        Alternatively do you know the steps required for the agent to be able to short list the local forest ? Do I need SCCM Discovery methods/expanded schema ? That may give better results even with the bug highlighted by your article.

        Many thanks,
        Fodder

      • Anoop says:

        Hi Fodder ! – It’s good that you’ve already published MP detailed into DNS and not published the details into DMZ AD. You’ll have one advantage when you publish MP details into DMZ AD. When your client receives MP list (all the MPs under the primary site) from the local forest MP, it may give preference to local forest MP because of ForestTrust: ‘Y’ entry. So (it seems to me) when you extend the schema and publish MP into DMZ AD then only the CM client can understand one MP is from it’s local forest.

        However, I never recommend to publish MP details to DMZ AD in your case. When you publish MP details then you will fall into the exact scenario which I mentioned in the post.

        Your current situation is better than mine because the client initially contacts to get MP list and it fails to get the list from AD. Next, client will check DNS for MP details and it will provide only one MP which is your local forest MP.

        In our case, we’ve published MP details into AD, so all the MP/s (under a primary server) details is already available in all the forests. SO whenever client contacts AD to fetch the MP details it will get all the MPs and the client will get confused which MP should I connect because when a client gets MP details from AD it won’t get ForestTrust: ‘Y’ information.

        The MPs we get from AD do not have the forest trust hint (ForestTrust ‘N’). The forest trust hint is provided only when MPs are obtained from MP (ForestTrust ‘Y’). We would then randomize and persist these based on affinity rules after getting from both the AD and then MP. So this explains some of the inconsistencies we see in the logs and the perceived instability.

        We’re planning to remove AD publishing and use DNS publishing in our scenario for better MP selection.

        Another few options we’ve is to use local host file entries for the MPs which are in remote forest. Probably, assign loop back IP for those MPs (this method is used in Robert Marshall tool SwitchMP more details http://anoopcnair.com/2014/01/28/sccm-tools-robert-marshall-mvp-custom-ddr-client-push-manager-component-manager-inbox-monitor/). By doing this the client may immediately reject that MP. Otherwise make host file entry to MAP wrong MPs to local forest MPs. 🙂

        I’ve not tested these options till now. Hopefully, we will get better workarounds from Microsoft.

        Regards
        Anoop

  3. Fodder says:

    Hi Anoop,

    DNS publishing does not seem to have helped the situation for me. It only appears to be one of many mechanisms to discover an initial MP.

    Once the agent has connected to the MP it still gets a list of all MP’s (including outside of the DMZ) even though I haven’t extended the schema or published into AD. Similarly (and possibly using the very same mechanism) http://localhost/sms_mp/.sms_aut?mplist returns all MP’s including outside the DMZ.

    The local MP is coming back ForestTrust: “N” suggesting I need to extend the schema and publish into AD to get it to come back ForestTrust: “Y”. However at that point we arrive at the bug this article highlighted with it occasionally returning incorrectly.

    I’m going to assign remote MP’s to the loopback address in the hosts file and see how that goes.

    Thanks for the leads.

    Regards,
    Fodder

    • Anoop says:

      Hi Fodder ! – Got your point that DNS publishing alone won’t fix our problem 🙁 We were thinking to implement 1. Remove AD publishing 2.Publish DNS as workaround to this problem. From your experience that doesn’t make much sense either.

      Regards
      Anoop

  4. EricDelmotte says:

    Hello all, Same issue here with 5 untrusted domains , with an Mp in only one of these domains and 2 MPs in internal domain where sccm site is located .Firewall doesn’t allow the untrusted domains clients to talk with internal MPs, but only the one in untrusted domain. Ad affinity doesn’t work for the 4 domains withut any MP.

    Rotating to internal MPs issue was resolved by
    1 – not publish in AD
    2 – publish only the MP in untrusted domains
    3 – host file change on each client to redirect the internal MPs to the one in untrusted domain
    4 – manually approve the new clients in sccm consoles (auto approve only for trusted domains)

    Regards
    Eric

    this is a pain.

    • Anoop says:

      Hi Eric ! – Thank you for the comment. It’s a pain. I can understand 🙁
      3. Host file change is also lot of work. isn’t it? How do you achieve this? Better to have some duplicate DNS records of other untrusted MPs pointing to local domain MP (IP) on local DNS server.
      4. This is loads of work. Would suggest to auto approve all clients.

      Regards
      Anoop

Leave a Comment and Contact Anoop