AppX Packages Slowing You Down?

Do you know what AppX packages are? I don’t think most IT people do. 

AppX packages are applications installed on Windows, but the AppX (or MSIX) format is unique. It’s been designed as a container instead of the traditional MSI or EXE installers you might be used to. AppX has advantages, too, like being able to make your app available in the Microsoft Store, and having additional security, since the apps run as per user, instead of as per machine.

And this, cool as it is, appears to be where we run into problems.

Microsoft appears to enumerate all the AppX packages upon a new-user logon. Then, in an effort to facilitate a good user experience, each AppX package is evaluated for things like DefaultAssociations (e.g., http == MSEdge) and FileTypeAssociations (e.g., “.pdf” == MSEdge). To make this process faster, it seems Microsoft tries to operate this asynchronously. This is both good and bad. It’s good, because the whole phase operates super quickly. It’s bad because CPU contention from multiple threads can slow the whole process down.

The challenge with all this enumerating, associating, and configuring is that it is time-consuming. The more AppX packages you have, the longer it takes. Windows Registry operations are slow to begin with, and get slower the larger your hive. 

I highlighted this problem here, and the impact is more severe when you account for Speculative Execution patches. Add to that, the more AppX packages you have, the longer it takes to read/modify the hive, and so on. And this process happens TWICE.  Once during the “User Profile” phase, and then again during the “Preparing Windows” phase.

There is a definite lack of documentation on this whole process, so almost everything appears to be via user discovery (like this document).

The stress of work amid the COVID-19 pandemic is quite enough on its own, but these AppX difficulties aren’t making things any easier for people.  I can hear the frustration in the voices of our customers—and honestly in my own—when I don’t have an answer for their logons being slow because of AppX Packages. These people are onboarding thousands of users a week and this logon performance is, to say the least, not helping.

“Could it really be as bad as all that,” you ask? Let’s take a look.

The example above shows a 54.7-second logon, 38.0 seconds of which is the shell starting. Of this, 36.7 seconds is AppX packages doing its thing. In short, with AppX, the time for this logon could literally be cut in half (or less!)

At ControlUp, we’re all about making our customers’ lives simpler. So, I set out to see if I could come up with a way to help our customers fix this.

For persistent machines, initial logons are usually slow, and subsequent logons are faster. During logon, the AppReadiness and AppXDeploymentServer processes seem to recognize that these packages are already staged and will skip the whole evaluation process above. When this process is skipped, you see the following in EventViewer:

Whereas the first time you logon (or if a new AppX package was loaded) while you were logged off, you’ll get packages actually populated in this event. And it will look like this:

If your profile solution is a profile container, then it should capture everything and subsequent logons to the same machine should be short. However, I have seen it get reevaluated and every logon has the AppX package delay. 

The reason for this appears to be non-persistent machines. There appears to be some AppX packages notifying the server that a user has been here before so check for changes and initialize those, otherwise just skip.

You can view users and who are set up on the server by looking at this registry path: HKLMSoftwareMicrosoftWindowsCurrentVersionAppReadiness

You’ll find this key contains SIDs of the users who’ve logged on to that box.

If you have Windows Virtual Desktop (WVD) and publish applications across different servers, you probably know this pain first hand, as each application has this delay on app launch.

One of things I was interested in was comparing the operating systems themselves. Server 2019 has AppX packages, too, but its logons are considerably faster than those on Windows 10 Enterprise or Windows 10 Enterprise for Virtual Desktops.

This made me wonder: what is missing from the Server 2019 experience? Or is it good enough that we can remove all the additional packages from the Windows 10 operating systems? If we remove all additional packages, would the performance be similar?

My Journey of AppX Discovery Had Begun

I did all the “plain Jane” Windows Operating System installs. I ran the following commands on all the operating systems:

Get-AppxProvisionedPackage -online

Get-AppxPackage

The Results:

Interesting results. Windows 10 Enterprise for Virtual Desktops did not list anything for AppXProvisionedPackages whereas Windows 10 Enterprise showed an extensive list. Both Windows 10 operating systems have far more AppX packages overall compared to Server 2019 at 58 for Win10EVD and 62 for Win10E. Server 2019 has two packages that the desktop OS’s do not have: InputApp and Cortana.

But there was something else I wanted to check.  

I know from looking at procmon logs of Windows logging in that it seems to set file type associations and default program associations. From looking at the procmon log, it appears to iterate through every AppX package and if it finds these associations configured, it sets them up in the user’s profile. This is another process that is quite long, and Windows offers a way to pre-configure default associations that get processed for every user. I was curious about the differences between the operating systems.

The default association can be exported by executing the following command:

Dism /Online /Export-DefaultAppAssociations:"F:AppAssociations.xml"

If exports an XML file in this format

<?xml version="1.0" encoding="UTF-8"?>

<DefaultAssociations>

  <Association Identifier=".3g2" ProgId="WMP11.AssocFile.3G2" ApplicationName="Windows Media Player" />

</DefaultAssociations>

I took this XML file and created a list to make it easier to parse.

The Windows 10EVD list was pretty short, Win10E was quite long, and Server 2019 looked like it might be able to do without the Media Player associations (if Windows 10EVD can do without them).

The differences between the different -types- of operating systems are notable.

This journey is just beginning.

Next I’m going to explore, in more detail, the differences between Windows 10 1809, which should be analogous to Server 2019 and the newest Windows 10 2004—including EVD. Stay tuned for the next installment in this series as I explore AppX packages further.

Related Links:

• Download APPX Add-AppxPackage
• How to Install .Appx or .AppxBundle Software on Windows 10
• Microsoft Replacing MSI, AppX with New Packaging Tool

Trentent Tye

Trentent Tye, a Tech Person of Interest, is based out of Canada and its many, many feet of snow. FUN FACT: Trentent came to ControlUp because, as a former customer, the product impacted his life in so many positive ways—from reducing stress, time to remediation, increased job satisfaction, and more—he had to be our evangelist. Now an integral part of ControlUp’s Product Marketing Team, he educates our customers, pours his heart and soul into the product, and generally makes ControlUp a better place. Trentent recently moved to be closer to family. He does not recommend moving during a pandemic.