AppX Packages Slowing You Down?

Do you know what AppX packages are? I don’t think most IT people do.

AppX packages are applications installed on Windows, but the AppX (or MSIX) format is unique. It’s been designed as a container instead of the traditional MSI or EXE installers you might be used to. AppX has advantages, too, like being able to make your app available in the Microsoft Store, and having additional security, since the apps run as per user, instead of as per machine. And this, cool as it is, appears to be where we run into problems. Microsoft appears to enumerate all the AppX packages upon a new-user logon. Then, in an effort to facilitate a good user experience, each AppX package is evaluated for things like DefaultAssociations (e.g., http == MSEdge) and FileTypeAssociations (e.g., “.pdf” == MSEdge). To make this process faster, it seems Microsoft tries to operate this asynchronously. This is both good and bad. It’s good, because the whole phase operates super quickly. It’s bad because CPU contention from multiple threads can slow the whole process down. The challenge with all this enumerating, associating, and configuring is that it is time-consuming. The more AppX packages you have, the longer it takes. Windows Registry operations are slow to begin with, and get slower the larger your hive. I highlighted this problem here, and the impact is more severe when you account for Speculative Execution patches. Add to that, the more AppX packages you have, the longer it takes to read/modify the hive, and so on. And this process happens TWICE. Once during the “User Profile” phase, and then again during the “Preparing Windows” phase. There is a definite lack of documentation on this whole process, so almost everything appears to be via user discovery (like this document). The stress of work amid the COVID-19 pandemic is quite enough on its own, but these AppX difficulties aren’t making things any easier for people. I can hear the frustration in the voices of our customers—and honestly in my own—when I don’t have an answer for their logons being slow because of AppX Packages. These people are onboarding thousands of users a week and this logon performance is, to say the least, not helping.

“Could it really be as bad as all that,” you ask? Let’s take a look.

The example above shows a 54.7-second logon, 38.0 seconds of which is the shell starting. Of this, 36.7 seconds is AppX packages doing its thing. In short, with AppX, the time for this logon could literally be cut in half (or less!)

At ControlUp, we’re all about making our customers’ lives simpler. So, I set out to see if I could come up with a way to help our customers fix this.

For persistent machines, initial logons are usually slow, and subsequent logons are faster. During logon, the AppReadiness and AppXDeploymentServer processes seem to recognize that these packages are already staged and will skip the whole evaluation process above. When this process is skipped, you see the following in EventViewer:

Whereas the first time you logon (or if a new AppX package was loaded) while you were logged off, you’ll get packages actually populated in this event. And it will look like this: If your AppX profile solution is a profile container, then it should capture everything and subsequent logons to the same machine should be short.

If your profile solution is a profile container, then it should capture everything and subsequent logons to the same machine should be short. However, I have seen it get reevaluated and every logon has the AppX package delay. The reason for this appears to be non-persistent machines. There appears to be some AppX packages notifying the server that a user has been here before so check for changes and initialize those, otherwise just skip. You can view users and who are set up on the server by looking at this registry path: HKLMSoftwareMicrosoftWindowsCurrentVersionAppReadiness You’ll find this key contains SIDs of the users who’ve logged on to that box. If you have Windows Virtual Desktop (WVD) and publish applications across different servers, you probably know this pain first hand, as each application has this delay on app launch. One of things I was interested in was comparing the operating systems themselves. Server 2019 has AppX packages, too, but its logons are considerably faster than those on Windows 10 Enterprise or Windows 10 Enterprise for Virtual Desktops. This made me wonder: what is missing from the Server 2019 experience? Or is it good enough that we can remove all the additional packages from the Windows 10 operating systems? If we remove all additional packages, would the performance be similar?

My Journey of AppX Discovery Had Begun

I did all the “plain Jane” Windows Operating System installs. I ran the following commands on all the operating systems:

Get-AppxProvisionedPackage -online

Get-AppxPackage

The Results:

Command	Server 2019	Windows 10 EVD 2004	Windows 10 E 2004
Get-AppxProvisionedPackage -online			Microsoft.549981C3F5F10
			Microsoft.BingWeather
			Microsoft.DesktopAppInstaller
			Microsoft.GetHelp
			Microsoft.Getstarted
			Microsoft.HEIFImageExtension
			Microsoft.Microsoft3DViewer
			Microsoft.MicrosoftEdge.Stable
			Microsoft.MicrosoftOfficeHub
			Microsoft.MicrosoftSolitaireCollection
			Microsoft.MicrosoftStickyNotes
			Microsoft.MixedReality.Portal
			Microsoft.MSPaint
			Microsoft.Office.OneNote
			Microsoft.People
			Microsoft.ScreenSketch
			Microsoft.SkypeApp
			Microsoft.StorePurchaseApp
			Microsoft.VCLibs.140.00
			Microsoft.VP9VideoExtensions
			Microsoft.Wallet
			Microsoft.WebMediaExtensions
			Microsoft.WebpImageExtension
			Microsoft.Windows.Photos
			Microsoft.WindowsAlarms
			Microsoft.WindowsCalculator
			Microsoft.WindowsCamera
			microsoft.windowscommunicationsapps
			Microsoft.WindowsFeedbackHub
			Microsoft.WindowsMaps
			Microsoft.WindowsSoundRecorder
			Microsoft.WindowsStore
			Microsoft.WindowsTerminal
			Microsoft.Xbox.TCUI
			Microsoft.XboxApp
			Microsoft.XboxGameOverlay
			Microsoft.XboxGamingOverlay
			Microsoft.XboxIdentityProvider
			Microsoft.XboxSpeechToTextOverlay
			Microsoft.YourPhone
			Microsoft.ZuneMusic
			Microsoft.ZuneVideo
Get-AppxPackage	1527c705-839a-4832-9118-54d4Bd6a0c89	1527c705-839a-4832-9118-54d4Bd6a0c89	1527c705-839a-4832-9118-54d4Bd6a0c89
	c5e2524a-ea46-4f67-841f-6a9465d9d515	c5e2524a-ea46-4f67-841f-6a9465d9d515	c5e2524a-ea46-4f67-841f-6a9465d9d515
	E2A4F912-2574-4A75-9BB0-0D023378592B	E2A4F912-2574-4A75-9BB0-0D023378592B	E2A4F912-2574-4A75-9BB0-0D023378592B
	F46D4000-FD22-4DB4-AC8E-4E1DDDE828FE	F46D4000-FD22-4DB4-AC8E-4E1DDDE828FE	F46D4000-FD22-4DB4-AC8E-4E1DDDE828FE
	Microsoft.AAD.BrokerPlugin	Microsoft.AAD.BrokerPlugin	Microsoft.AAD.BrokerPlugin
	Microsoft.AccountsControl	Microsoft.AccountsControl	Microsoft.AccountsControl
	Microsoft.AsyncTextService	Microsoft.AsyncTextService	Microsoft.AsyncTextService
	Microsoft.BioEnrollment	Microsoft.BioEnrollment	Microsoft.BioEnrollment
	Microsoft.CredDialogHost	Microsoft.CredDialogHost	Microsoft.CredDialogHost
	Microsoft.ECApp	Microsoft.ECApp	Microsoft.ECApp
	Microsoft.LockApp	Microsoft.LockApp	Microsoft.LockApp
	Microsoft.Win32WebViewHost	Microsoft.Win32WebViewHost	Microsoft.Win32WebViewHost
	Microsoft.Windows.Apprep.ChxApp	Microsoft.Windows.Apprep.ChxApp	Microsoft.Windows.Apprep.ChxApp
	Microsoft.Windows.CapturePicker	Microsoft.Windows.CapturePicker	Microsoft.Windows.CapturePicker
	Microsoft.Windows.CloudExperienceHost	Microsoft.Windows.CloudExperienceHost	Microsoft.Windows.CloudExperienceHost
	Microsoft.Windows.NarratorQuickStart	Microsoft.Windows.NarratorQuickStart	Microsoft.Windows.NarratorQuickStart
	Microsoft.Windows.OOBENetworkCaptivePortal	Microsoft.Windows.OOBENetworkCaptivePortal	Microsoft.Windows.OOBENetworkCaptivePortal
	Microsoft.Windows.OOBENetworkConnectionFlow	Microsoft.Windows.OOBENetworkConnectionFlow	Microsoft.Windows.OOBENetworkConnectionFlow
	Microsoft.Windows.PeopleExperienceHost	Microsoft.Windows.PeopleExperienceHost	Microsoft.Windows.PeopleExperienceHost
	Microsoft.Windows.PinningConfirmationDialog	Microsoft.Windows.PinningConfirmationDialog	Microsoft.Windows.PinningConfirmationDialog
	Microsoft.Windows.SecHealthUI	Microsoft.Windows.SecHealthUI	Microsoft.Windows.SecHealthUI
	Microsoft.Windows.ShellExperienceHost	Microsoft.Windows.ShellExperienceHost	Microsoft.Windows.ShellExperienceHost
	Microsoft.Windows.XGpuEjectDialog	Microsoft.Windows.XGpuEjectDialog	Microsoft.Windows.XGpuEjectDialog
	Windows.CBSPreview	Windows.CBSPreview	Windows.CBSPreview
	windows.immersivecontrolpanel	windows.immersivecontrolpanel	windows.immersivecontrolpanel
	Windows.PrintDialog	Windows.PrintDialog	Windows.PrintDialog
	InputApp
	Microsoft.Windows.Cortana
		Microsoft.549981C3F5F10	Microsoft.549981C3F5F10
		Microsoft.Advertising.Xaml	Microsoft.Advertising.Xaml
		Microsoft.BingWeather	Microsoft.BingWeather
		Microsoft.DesktopAppInstaller	Microsoft.DesktopAppInstaller
		Microsoft.GetHelp	Microsoft.GetHelp
		Microsoft.Getstarted	Microsoft.Getstarted
		Microsoft.HEIFImageExtension	Microsoft.HEIFImageExtension
		Microsoft.Microsoft3DViewer	Microsoft.Microsoft3DViewer
		Microsoft.MicrosoftEdge	Microsoft.MicrosoftEdge
		Microsoft.MicrosoftEdgeDevToolsClient	Microsoft.MicrosoftEdge.Stable
		Microsoft.MicrosoftOfficeHub	Microsoft.MicrosoftEdgeDevToolsClient
		Microsoft.MicrosoftSolitaireCollection	Microsoft.MicrosoftOfficeHub
		Microsoft.MicrosoftStickyNotes	Microsoft.MicrosoftSolitaireCollection
		Microsoft.MixedReality.Portal	Microsoft.MicrosoftStickyNotes
		Microsoft.MsixPackagingTool	Microsoft.MixedReality.Portal
		Microsoft.MSPaint	Microsoft.MSPaint
		Microsoft.NET.Native.Framework.2.2	Microsoft.NET.Native.Framework.2.2
		Microsoft.NET.Native.Runtime.2.2	Microsoft.NET.Native.Runtime.2.2
		Microsoft.Office.OneNote	Microsoft.Office.OneNote
		Microsoft.People	Microsoft.People
		Microsoft.ScreenSketch	Microsoft.ScreenSketch
		Microsoft.SkypeApp	Microsoft.SkypeApp
		Microsoft.StorePurchaseApp	Microsoft.StorePurchaseApp
		Microsoft.VCLibs.140.00	Microsoft.VCLibs.140.00
		Microsoft.VCLibs.140.00.UWPDesktop	Microsoft.VCLibs.140.00.UWPDesktop
		Microsoft.VP9VideoExtensions	Microsoft.VP9VideoExtensions
		Microsoft.Wallet	Microsoft.Wallet
		Microsoft.WebMediaExtensions	Microsoft.WebMediaExtensions
		Microsoft.WebpImageExtension	Microsoft.WebpImageExtension
		Microsoft.Windows.AssignedAccessLockApp	Microsoft.Windows.AssignedAccessLockApp
		Microsoft.Windows.CallingShellApp	Microsoft.Windows.CallingShellApp
		Microsoft.Windows.ContentDeliveryManager	Microsoft.Windows.ContentDeliveryManager
		Microsoft.Windows.ParentalControls	Microsoft.Windows.ParentalControls
		Microsoft.Windows.Photos	Microsoft.Windows.Photos
		Microsoft.Windows.Search	Microsoft.Windows.Search
		Microsoft.Windows.SecureAssessmentBrowser	Microsoft.Windows.SecureAssessmentBrowser
		Microsoft.Windows.StartMenuExperienceHost	Microsoft.Windows.StartMenuExperienceHost
		Microsoft.WindowsAlarms	Microsoft.WindowsAlarms
		Microsoft.WindowsCalculator	Microsoft.WindowsCalculator
		Microsoft.WindowsCamera	Microsoft.WindowsCamera
		microsoft.windowscommunicationsapps	microsoft.windowscommunicationsapps
		Microsoft.WindowsFeedbackHub	Microsoft.WindowsFeedbackHub
		Microsoft.WindowsMaps	Microsoft.WindowsMaps
		Microsoft.WindowsNotepad
		Microsoft.WindowsSoundRecorder	Microsoft.WindowsSoundRecorder
		Microsoft.WindowsStore	Microsoft.WindowsStore
		Microsoft.WindowsTerminal	Microsoft.WindowsTerminal
		Microsoft.Xbox.TCUI	Microsoft.Xbox.TCUI
		Microsoft.XboxApp	Microsoft.XboxApp
		Microsoft.XboxGameCallableUI	Microsoft.XboxGameCallableUI
		Microsoft.XboxGameOverlay	Microsoft.XboxGameOverlay
		Microsoft.XboxGamingOverlay	Microsoft.XboxGamingOverlay
		Microsoft.XboxIdentityProvider	Microsoft.XboxIdentityProvider
		Microsoft.XboxSpeechToTextOverlay	Microsoft.XboxSpeechToTextOverlay
		Microsoft.YourPhone	Microsoft.YourPhone
		Microsoft.ZuneMusic	Microsoft.ZuneMusic
		Microsoft.ZuneVideo	Microsoft.ZuneVideo
		MicrosoftWindows.Client.CBS	MicrosoftWindows.Client.CBS
		MicrosoftWindows.UndockedDevKit	MicrosoftWindows.UndockedDevKit
		NcsiUwpApp	NcsiUwpApp
			Microsoft.NET.Native.Framework.1.7
			Microsoft.NET.Native.Runtime.1.7
			Microsoft.Services.Store.Engagement
			Microsoft.Services.Store.Engagement
			Microsoft.UI.Xaml.2.0

Interesting results. Windows 10 Enterprise for Virtual Desktops did not list anything for AppXProvisionedPackages whereas Windows 10 Enterprise showed an extensive list. Both Windows 10 operating systems have far more AppX packages overall compared to Server 2019 at 58 for Win10EVD and 62 for Win10E. Server 2019 has two packages that the desktop OS’s do not have: InputApp and Cortana. But there was something else I wanted to check. I know from looking at procmon logs of Windows logging in that it seems to set file type associations and default program associations. From looking at the procmon log, it appears to iterate through every AppX package and if it finds these associations configured, it sets them up in the user’s profile. This is another process that is quite long, and Windows offers a way to pre-configure default associations that get processed for every user. I was curious about the differences between the operating systems.

The default association can be exported by executing the following command:

Dism /Online /Export-DefaultAppAssociations:"F:AppAssociations.xml"

If exports an XML file in this format

<?xml version="1.0" encoding="UTF-8"?>

<DefaultAssociations>

  <Association Identifier=".3g2" ProgId="WMP11.AssocFile.3G2" ApplicationName="Windows Media Player" />

</DefaultAssociations>

I took this XML file and created a list to make it easier to parse. The Windows 10EVD list was pretty short, Win10E was quite long, and Server 2019 looked like it might be able to do without the Media Player associations (if Windows 10EVD can do without them). Windows 10EVD, Win10E, and Server 2019 lists with Media Player associations

Windows 10EVD, Win10E, and Server 2019 lists with Media Player associations

The differences between the different -types- of operating systems are notable. This journey is just beginning. Next I’m going to explore, in more detail, the differences between Windows 10 1809, which should be analogous to Server 2019 and the newest Windows 10 2004—including EVD. Stay tuned for the next installment in this series as I explore AppX packages further.

Trentent Tye

Trentent Tye, a Tech Person of Interest, is based out of Canada and its many, many feet of snow. FUN FACT: Trentent came to ControlUp because, as a former customer, the product impacted his life in so many positive ways—from reducing stress, time to remediation, increased job satisfaction, and more—he had to be our evangelist. Now an integral part of ControlUp’s Product Marketing Team, he educates our customers, pours his heart and soul into the product, and generally makes ControlUp a better place. Trentent recently moved to be closer to family. He does not recommend moving during a pandemic.