Among the trickier things for IT administrators to nail down are the causes of slow logons and how to fix those problems quickly. That’s why ControlUp created the “Analyze Logon Duration” (ALD) script. And now, we’ve made five key changes to the ALD script that will make spotting problems—and remedying them quickly—simple. In this series, we’ll go over the enhancements and what additional information we can glean from them.

The five changes:

• Citrix Resultant Set of Policies (RSoP) analysis

• Resultant Set of Policies Logging (RSoP Logging)

• Windows Management Instrumentation (WMI) Filter duration analysis

• Individual AppX Package Load Times

• Loopback Processing Mode state

About ControlUp Analyze Logon Duration

Out of the box, ControlUp offers some amazing technology for reporting on logon durations for users. Overall Logon Duration, Profile Load Time, Group Policy Load Time, Desktop Load Time and a column for unknown logon duration contributors are all calculated and displayed—in real time—right after a user has logged on.

This is extraordinarily helpful for identifying low-hanging fruit optimization opportunities.

Originally introduced in 2015, ControlUp (along with help from our community members—particularly Guy Leech) has been continually improving and increasing the functionality of the Analyze Logon Duration script, which can break logons down into granular phases, based on technology or logon processes.

But ALD can only report back on the things it understands.

See those two large gaps of time on the far right of the image? 33.9 seconds and 20.9 seconds?

When ControlUp customers have unexplained logon duration gaps, they can reach out to us to help them assist in troubleshooting the root cause of these gaps. From here, ControlUp Professional Services can assist in identifying the technology or reason for the gaps. This article will go into the technical details of the gap in this screenshot, what it is we discovered, and how ALD reports it after we added this technology into the script.

About Loopback Processing

In this installment of our five-part series on ControlUp Analyze Logon Duration, we’ll examine Loopback Processing Mode state, what it means, and how it can impact your environment.

The confusingly named feature “Loopback Processing Mode” has three different values: “Replace,” “Merge,” and “no value.” What impact do each of these have on logons?

Literature from Microsoft leads you to explanations of Loopback Processing Mode, but honestly? I don’t find these articles helpful in understanding how loopback processing actually works.

The Problem

Active Directory is logically organized. That means the structure of the organizational units (OUs) and where user and computer objects are located is defined by you or someone in your organization. It can be organized by department, physical location, type of resource, or whatever you want. Each of these categories can be nested inside Organizational Units (OUs) or flat or anything else.

When a user logs on to their desktop or laptop, Group Policy settings attached to the OU (and up the tree) for the user and machine apply in a standard, predictable way. The tree for the user object gets enumerated, all Group Policy Objects (GPOs) assigned to OUs up the tree get to apply their values for their user policy and the machine policy sets values for the machine.

This works pretty well for a purely physical environment! The logic is simple and straightforward: users get user policies; machines get machine policies.

Desktops and laptops are full of dedicated resources (e.g., CPUs, GPUs, memory, disk) for the user logging onto that machine, so you define your policies for maximum user experience. Full transparency, animations, full-search indexing. The works!

But…

Let’s add end-user computing (EUC) environments [Citrix Virtual Apps & Desktops (CVAD]), VMware Horizon, or Azure Virtual Desktop (AVD)] into the mix now.

Your EUC servers are virtual machines. They do not have dedicated graphics, and their CPU, disk, and memory is shared with other users or virtual machines. We need to reduce features within Windows to maximize processes that do productive work. Turning off transparency, disabling indexing, reducing animation durations—all of these can save considerable resources. Each of these “saved” resources that apply to each user can quickly add up. Optimizing your server can give it the ability to service more users or increase application performance on a per-user basis.

However, this is where you run into a problem.

You need the user to have a full, maximized digital experience when they are logged onto desktops or laptop machines. But you also need a maximally efficient digital experience when they are logged onto these shared servers. Since these digital experience tweaks are user settings, we can configure them in group policy. But how can we apply them where we want? We can’t apply both Group Policy objects against the same user OU—they conflict!

Enter Loopback Processing

What if we want to apply specific user settings, depending on the machine someone happens to log on to?

Microsoft has come up with a solution: Loopback Processing.

Loopback Processing has three modes. Disabled / Not Configured, Merge Mode, and Replace Mode.

Disabled / Not Configured: When set in this mode, Group Policy is applied with users getting all user policy settings within their scope, and machines get machine policies within their scope. Simple.

Merge Mode: Microsoft describes Merge Mode as:

When the user logs on, the user’s list of GPOs is typically gathered by using the GetGPOList function. The GetGPOList function is then called again by using the computer’s location in Active Directory. The list of GPOs for the computer is then added to the end of the GPOs for the user. It causes the computer’s GPOs to have higher precedence than the user’s GPOs. In this example, the list of GPOs for the computer is added to the user’s list.

Did you catch that? Clear as mud, right?

Replace Mode: Microsoft describes Replace Mode as:

The user’s list of GPOs isn’t gathered. Only the list of GPOs, based on the computer object, is used.

I’m going to demonstrate how each of these modes works and review the ways they can impact your logon time.

Test Setup

I have three machine objects in three different OUs. Each OU will have a GPO setting and a different loopback policy setting.

Each OU will have an additional simple group policy object applied that we can use to track how the settings are applied. This is a total of four group policy objects, with two assigned to the machine OU and one assigned to the user’s OU. The “simple” GPO will look similar to this, with minor tweaks for each GPO to help identify the winner:

Here’s a table for each Group Policy Object and their differences:

With all this configured, this is what we end up with:

I blocked inheritance for my “Loopback Processing” OU to avoid contaminating my test with my functional GPO structure. This should not impact whether the user policy for the front desk users is applied or not.

The machines I logged into were set up like this:

Test Results

Not Configured – LAB01

What’s supposed to happen here should be the easiest to predict with “Not Configured.” The “Configure Desktop Experience” GPO from the “Front Desk” OU should apply the user policies. User policies in the “Loopback Processing / Not Configured” OU should be ignored.

This means we should start with a notepad.exe window and our HKCU values should identify themselves as having come from the “Configure Desktop Experience” GPO.

Indeed, this is what I see. What does Windows report and do?

Using Process Monitor and filtering for PATH “contains” SYSVOL we can see the group policy engine read the group policy object(s)!

And the only Group Policy object it read was “5FC27DF9…”. This is our “Configure Desktop Experience” GPO assigned to the same OU as our user.

The logon generated event ID 5311, which details the loopback policy processing mode:

And Event 5312 showing the list of applicable GPOs to apply:

Simple enough! “Not configured” was exactly as predicted!

Replace – LAB03

You may have noticed that the “servers” are out of order. I am doing LAB03 ahead of LAB02 because LAB02 has a more interesting story. But first things first.

“Replace” is quite simple to understand. The user GPO in the user object’s OU is ignored and user policies assigned to the machine object’s OU will apply. With replace, you configure all your policies in the machine’s OU—user and computer— and they will apply to all users that logon to machines in that OU. GPOs outside that OU will be ignored (ignoring enforced GPOs).

I expect this will launch a command prompt window and the registry values should contain “Desktop Experience – Replace”.

Prediction, correct!

Procmon shows us accessing a single GPO: “01E7B0FE…”  This Group Policy Object is our “Desktop Experience – Replace” GPO.

The Loopback Processing mode comes back as “Replace.”

And the list of group policy objects is only the user policy residing in the machine object’s OU:

BOOM! 💥 Bang on the prediction! Loopback processing modes “Replace” and “Not Configured” are simple to understand.

Merge – LAB02

This is the one that I have struggled with understanding, so I hope this mini adventure helps you like it helped me.

Based on what Microsoft says in it’s documentation for Merge Mode, it appears that Group Policy will grab all the GPOs assigned to the user object, then all the GPOs with user settings assigned to the machine object. The user GPOs will be processed first then the user policy of the machine GPO’s will be processed second. With procmon, we’ll see the conflicting values get written and then we can see them get overwritten (exclamation marks are only a hint at my excitement about this)!

My prediction: we’ll see both the “Configure Desktop Experience” and “Desktop Experience – Merge” GPOs get read and any conflicting value will show the values in the “Desktop Experience – Merge” GPO as it is read and applied last it is the GPO in the machine’s OU.

Look at that! We get the calculator, and we can see the conflicting value is “Desktop Experience – Merge,” which came from the “Desktop Experience – Merge” GPO. But we can also see the “Trentent1 ” registry key. This was generated by the “Configure Desktop Experience” GPO that resided in the users OU.

PROCMON shows some interesting things with respect to the way the order of operations is applied.

What are we seeing here?

We can see clearly that there are two GPOs being read.  5FC27DF9… and A94758DB… are the two GPOs being read and applied. Those GPOs are:

The order they are applied is User OU first → Machine OU next.

We can see each “stage” of the GPO process in the procmon, too. The reading of the GPT.INI files is the check for each GPO on the OU tree to see if they have user policies to apply. The lines that end in “Registry.pol” are the native group policy entries being applied (the ones set with ADM / ADMX). Again, we see the user object OU GPO applying first, followed by the machine object GPO. Lastly, the lines that end in “Registry.xml” are the Group Policy Preferences being applied. And again, the order is user object OU GPO and then machine object OU GPO.

Loopback policy processing mode is “Merge” as expected.

And here, we get our list of group policy objects and the order they will apply. In this list, the bottom most have the highest priority. And this appears to be because it’s a last-write-wins scenario.

Conflicting values

This is pretty neat. With PROCMON you can see each group policy value, as they are written to the registry. We can see the Shell value being set to notepad.exe (as it is in the user OU GPO object) and then being changed to “calc.exe” from the machine OU GPO object. And then the same continues for the other values underneath. The group policy preferences of the user object OU GPO is applied first (“\Trentent1\Configure Desktop Experience” and “\TrententShared\Conflicting Value”) and then we see the machine object OU GPO apply (“\Trentent\Desktop Experience – Merge” and “\TrententShared\Conflicting Value”).

How does this impact logon duration?

The answer to this question is, “it depends.” It can impact a little, not at all, or significantly. This is pretty much the same answer for every technology that touches the logon process. 🙂

ControlUp and ALD makes it incredibly easy to analyze it, though, and determine the impact and its impactor. To analyze sessions in your environment, right-click on a session, select “Script Actions,” “Analyze Logon Duration” and then “OK.”

Alternatively, you can use the ControlUp Virtual Expert™️, click the “menu” icon next to “Logon Duration,” and select “Analyze Logon Duration.”

Easy, right?

Merge Mode can add uncertainty and complexity as different users can have different GPOs applied depending on where they are in your OU structure. Some users might end up with one or two GPOs applied and some might have 50.

More GPO objects that have to be processed means more time spent on the network means the more sensitive to latency you become. In my testing of 100 GPOs, going from sub-millisecond to ten milliseconds on the network caused my logons to be increased by 4 seconds simply in time spent waiting on the network pulling files and querying directories. When latency increased to 25 milliseconds the amount of time spent on the network added 8 seconds to the processing time. And looking at the procmon trace, you can see there were dozens of calls to SYSVOL, each one adding time due to latency. With multiple GPOs you can easily be talking about hundreds or thousands of calls to the network.

Don’t get me wrong, though! Merge mode has interesting benefits that you can leverage (as this article observed). It’s really up to you to sort out what drawbacks you might have using Replace or no loopback processing in place of Merge. But I hope this article has pulled back the curtain a little bit on this interesting Windows feature.

Analyze Your Environment

To get insights like these in your environment, start a trial of ControlUp today (it’s FREE; there’s no sales call required!). Just connect to one of your machines and analyze one of your recent logons. If you need further assistance getting the proper data, reach out to sales@controlup.com and one of our pre-sales engineers can help get your environment set up.

For additional information on what can impact logons check out the ControlUp blogs and check out the rest of our Analyze Logon Duration deep dive series:

• Better Than Ever: 5 Key Updates to ControlUp Analyze Logon Duration

• 5 Key Updates to ControlUp Analyze Logon Duration: AppX Package Load Times

• 5 Key Updates to ControlUp Analyze Logon Duration: RSoP Logging

• 5 Key Updates to ControlUp Analyze Logon Duration: WMI Filter Duration Analysis

• 5 Key Updates to ControlUp Analyze Logon Duration: Citrix RSoP

Trentent Tye

Trentent Tye, a Tech Person of Interest, is based out of Canada and its many, many feet of snow. FUN FACT: Trentent came to ControlUp because, as a former customer, the product impacted his life in so many positive ways—from reducing stress, time to remediation, increased job satisfaction, and more—he had to be our evangelist. Now an integral part of ControlUp’s Product Marketing Team, he educates our customers, pours his heart and soul into the product, and generally makes ControlUp a better place. Trentent recently moved to be closer to family. He does not recommend moving during a pandemic.