Sometimes, when things seem to be going catastrophically wrong, the root cause can be simple. You just have to find it.
After a recent batch of updates were applied to a SharePoint 2016 farm, a large number of items quit working on the application server. Outgoing emails stopped sending, workflows stopped working, and scheduled tasks that used the server object model stopped working.
It was a mess. Worse yet, troubleshooting this proved to be a challenge because the SharePoint tracing service wouldn’t start, and the SharePoint management shell couldn’t connect to the farm.
Down for the Count?
It seemed like this server was down for the count and we were bracing ourselves to rebuild it.
Here are a few details on some of the many symptoms we were seeing:
- The SharePoint Tracing Service would not run, meaning there were no log files to look at. When trying to start it from the Services control panel, we got the message “Error 1053: The service did not respond to the start or control request in a timely fashion.”
- The SharePoint management shell could not connect to the farm and also displayed the following on startup:
Could not read the XML Configuration file in the folder 16\CONFIG\PowerShell\Registration\.
Could not find a part of th e path 'C:\Windows\System32\16\CONFIG\PowerShell\Registration'.
No xml configuration files loaded.
Unable to register core product cmdlets.
Could not read the Types files in the folder 16\CONFIG\PowerShell\types\.
Could not find a part of the path 'C:\Windows\System32\16\CONFIG\PowerShell\types'.
"No Types files Found."
Could not read the Format file in the folder 16\CONFIG\PowerShell\format\.
Could not find a part of the path 'C:\Windows\System32\16\CONFIG\PowerShell\format'.
No Format files Found.
- We saw many errors in the Windows logs that the farm is unavailable. For example:
WebHost fai led to process a request.
Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/62476613
Exception: System.ServiceModel.ServiceActivationException: The service '/ddb6d31923f94358821025a906ed543e/FeedCacheService.svc' cannot be activated due to an exception during compilation. The exception message is: The farm is unavailable.. ---> System.InvalidOperationException: The farm is unavailable.
- The SharePoint Products Configuration Wizard reported that it “Failed to detect if this server is joined to a server farm.”
Procmon to the Rescue
All these issues began happening after the server reboot that followed applying updates. While it seemed like the server was in a very bad state, I felt there was a chance that something simple was causing all SharePoint components to fail, like a configuration file or registry setting somewhere. However, with no SharePoint logs, it was difficult to narrow down what that setting was. Before going down the path of rebuilding the server, I decided to give my favorite last resort troubleshooting tool a try: Process Monitor, or procmon.
I downloaded procmon to the application server, fired it up, and configured it to filter events from the wsstracing.exe process only. I then tried starting up the SharePoint Tracing Service again and looked for clues in procmon.
Shortly before aborting, wsstracing.exe tried to read the registry key:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\16.0\Location
A quick check revealed that that key was empty on the server. I changed it back to the SharePoint install location:
C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\
Everything came back to life!
We are not sure which of the updates that were applied actually caused this, but everything has been working correctly since fixing the registry key.
Quick summary
If it looks like everything on a SharePoint server suddenly stops working, make sure the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\16.0\Location points to the right place.
Learn more about DMC SharePoint Consulting Services.