Logon Duration Research: AppX Packages

In the last installment in this series, we explored the way Microsoft has designed Windows to have its AppX packages operate in different ways, even if the user context—logging on to the system and the differing logon durations amongst the operating systems—is the same.

The wildly different configurations of Windows have led to some intriguing questions for Citrix, VMware, and Microsoft EUC administrators, chiefly, “Do I need what this feature is offering?”

Multi-session support is new to Windows 10, but has been on Server operating systems for a long time. Are your multi-session users happy with the experience they get on the Windows Server OS? If they are happy or productive, why would a full-fledged Windows Desktop need all the other stuff? What would the drawbacks be if you removed all the superfluous features from the Windows Desktop OSes?

Before we get out over the tips of our skis, though, let’s start by establishing a baseline. 

As an experiment, I installed six different versions of Windows:

• Windows 10 1809 Enterprise
• Windows 10 1809 Enterprise—Long Term Service Channel
• Windows 10 2004 Enterprise
• Windows 10 2004 Enterprise N
• Windows 10 2004 Enterprise for Virtual Desktops
• Windows 2019 Server Standard

I chose these operating systems for some specific reasons.

Windows 10 1809 Enterprise: Long-Term Service Channel

The LTSC version of Windows 10 (supposedly) comes with reduced AppX packages. I’ll validate if that is true, plus the differences between Win10 1809 LTSC and the other OSes.

Windows 10 1809 Enterprise

If the LTSC version has reduced AppX packages, we should be able to measure the impact by comparing it to the “SAC” 1809 version to see what additional packages it has and what they are contributing.

Windows 10 2004 Enterprise

Arguably the latest and greatest version of Windows (as of this writing). I profiled the AppX packages, but we can take a look at the additional packages it contains, compared to its peers.

Windows 10 2004 Enterprise N

This is the “Media Player Free” version of Windows 10. This version automatically comes with at least one less piece of software (compared to the non-N versions). Specifically, I’m curious to see if other AppX packages are removed here, as well, and about how Microsoft chose to handle the file-type associations.

Windows 10 2004 Enterprise for Virtual Desktops

As examined in my last post, Windows 10 Enterprise for Virtual Desktops makes some major changes to the way many AppX packages and file associations are made available. Microsoft significantly reduces the count on both when compared to its Enterprise peer. 

Windows 2019 Server Standard with RDSH enabled

The gold standard: server operating systems contain much-reduced AppX packages, since most are not useful in server environments. 

Since loading AppX packages is a CPU-intensive task, I decided to test with 1vCPU, 2vCPU, and 4vCPU configurations. Observations showed that AppX packages load asynchronously, so the more CPU you have, the less contention and the faster loading happens. Reducing the number of CPUs should increase the duration, as contention is higher; not all packages will get a time slice of the CPU while the operation is processing.

Finally, since CPU is such an important part of this test, it’s important to disclose the types of processors on which I ran the tests. In this case, I used 12-core, 24-thread Ryzen 9 3900X 3.8Ghz processors.

On each of the operating systems, I measured the following data:

• Number of AppX packages
• Number of AppX Provisioned Packages
• Number of File Associations
• Number of File Associations to AppX packages

For this test, I installed each operating system directly from its release ISO and applied Windows Updates. Each was joined to the domain in a OU with no group policy objects applied, since we just want to test the AppX package impact. Power settings were set to “High Performance” or “Ultimate Performance” both in Windows and on the Hypervisor. The Windows 2019 server operating system had Remote Desktop Session Host role installed.

Before I get into all that, though, I want to share some things I found intriguing during the setup phase.

Windows 10 EVD doesn’t have “Ultimate Performance”—only “High Performance”

Windows 10 EVD will prompt you for a reason as to why you want to shut down or restart the machine.

Memory Compression is enabled on all Windows 10 machines (including EVD), but is disabled on Server machines.

I observed that the Windows 10—non-EVD—machines generated CPU load when idling. This was the result of WinSAT.exe running and appeared ~25 mins after boot-up. As such, I waited an hour to run any/all tests to ensure no outside processes were interfering.

Additionally, machines set up for multi-session (Windows 10 EVD and Server 2019) had a process execute [at the ten uptime mark] that consumed significant CPU for an extended period. For Server 2019, it consumed CPU for seven minutes, and on Windows 10 EVD it consumed CPU for one minute.  The CPU consumption for that was mscorsw.exe, which is .NET Framework optimization. Optimizer utilities, such as Citrix Optimizer, VMware Operating System Optimization Tool (OSOT), and Base Image Script Framework (BISF), should remove this load. Since my setup has minimally configured operating systems, this was something I needed to consider, so it wouldn’t impact the performance of my test logons. I captured the early uptime process with ControlUp, so I could see this in real time. See this video:

In Windows 10 1809 Enterprise, the OneDriveSetup.exe utility launched on logon and consumed huge amounts of CPU for significant periods of time; the LTSC version of 1809 did not.

1vCPU Logon Duration:

Averaged out over three runs:

2vCPU Logon Duration:

Averaged out over 3 runs:

4vCPU Logon Duration:

Averaged out over 3 runs:

Intriguingly, there appears to be an overhead that increasing the number of CPUs can’t push past. 

To gather more data, I logged in and queried for the number of provisioned packages, as well as the number of “AppX” packages, then looked at my AppX registry key AND the event viewer to see how many of each package was reported.

Looking at the counts of each operating system:

NOW we’re getting some interesting oddities! Windows 10 1809 E LTSC has close to the same number of packages as Server 2019 standard, but has significantly longer AppX package load times (each of these packages has their load time tracked in the Microsoft-Windows-AppReadiness/Admin log).

The list of packages [added to your “task list”] to execute is ID 240. Generating the list of tasks on my system spanned around ~0.5ms per task. For the 68 packages in 2004 Enterprise N, it took 28 ms to generate the list of packages to be installed (execution is measured by event 213). I created a script action to output my logon events to measure how long they took.

My logon time was 11:00:43; installation of the packages started ~0.6 seconds after logon and finished ~7.4 seconds after that. Total CPU duration was 11.07 seconds for loading of the packages. Sorting by duration revealed some outliers that should be considered. A far cry from the ~68 packages that get loaded, the output here showed just 20 packages. That’s because Microsoft has deemed that some packages aren’t required to be processed at logon, but can be processed later or at app launch. Some (all?) of these packages are processed asynchronously, but I don’t know if that’s all at once or throttled (like 4 at a time).

The reported start/stop for this event was 21 seconds. We’ve now accounted for ~8 seconds, so where are the other 13?

Before we get into that, let’s update our chart with the number of packages that were processed during logon.

Let’s also update this chart with the package durations:

Finally, let’s update the packages evaluated at logon for each OS

From just eye-balling it, Windows 1809 LTSC should have the same (or extremely close to) performance as Server 2019. The actual processing time differs by only 100 or so milliseconds, but the actual logon differs by 18 seconds!

So, now we have an idea what some out of the box Windows operating systems have for a baseline performance.

In our next article, we’ll dive further into the gaps of time that AppX reports and start on optimizations.

Trentent Tye

Trentent Tye, a Tech Person of Interest, is based out of Canada and its many, many feet of snow. FUN FACT: Trentent came to ControlUp because, as a former customer, the product impacted his life in so many positive ways—from reducing stress, time to remediation, increased job satisfaction, and more—he had to be our evangelist. Now an integral part of ControlUp’s Product Marketing Team, he educates our customers, pours his heart and soul into the product, and generally makes ControlUp a better place. Trentent recently moved to be closer to family. He does not recommend moving during a pandemic.