SharePoint 2013 uses the AppFabric Caching Service to centralize all caches that are used by SharePoint. The distributed cache is used by many features including Authentication, Newsfeed, Security, Page Loading, Workflows, and more. While Microsoft claims that SharePoint’s use of the Distributed Cache greatly increases performance, the service is also quite unreliable and difficult to troubleshoot. Hopefully they will write more documentation and improve the product with the release of SP1.
There is a good TechNet guide with Powershell commands that you might find useful if you ever need to change any of the AppFabric configuration. The article does not cover increasing Request Timeouts so I will review that in this blog post.
A few weeks ago we ran into a problem where one of our web parts stopped working when users were not on the local network. After investigating ULS logs we tracked down the issue to Distributed Cache timeouts.
The web part, a timer application, was developed with heavy use of the View State. As a result, as the View State Cache continued to grow, we began seeing intermittent Timeout errors in the logs.
Unexpected error occurred in method 'Put' , usage 'SPViewStateCache' - Exception 'Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode:SubStatus:The request timed out.. Additional Information : The client was trying to communicate with the server : net.tcp://SP2013Dev.DMC.local:22233 at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody, RequestBody reqBody) at Microsoft.ApplicationServer.Caching.DataCache.InternalPut(String key, Object value, DataCacheItemVersion oldVersion, TimeSpan timeout, DataCacheTag[] tags, String region, IMonitoringListener listener) at Microsoft.ApplicationServer.Caching.DataCache.<>c__DisplayClass25. b__24() at Microsoft.ApplicationServer.Caching.DataCache.Put(String key, Object value, TimeSpan timeout) at Microsoft.SharePoint.DistributedCaching.SPDistributedCache.Put(String key, Object value)'. Unexpected error occurred in method 'Put' , usage 'SPViewStateCache' - Exception 'Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode:SubStatus:The connection was terminated, possibly due to server or network problems or serialized Object size is greater than MaxBufferSize on server. Result of the request is unknown. ---> System.TimeoutException: The socket was aborted because an asynchronous receive from the socket did not complete within the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout. ---> System.IO.IOException: The read operation failed, see inner exception. ---> System.TimeoutException: The socket was aborted because an asynchronous receive from the socket did not complete within the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout. ---> System.Net.Sockets.SocketException: The I/O operation has been aborted because of either a thread exit or an application request at System.ServiceModel.Channels.SocketConnection.HandleReceiveAsyncCompleted() at System.ServiceModel.Channels.SocketConnection.OnReceiveAsync(Object sender, SocketAsyncEventArgs eventArgs) -
We also had a problem with starting SP 2013 workflows which we were able to trace down to Distributed Logon Token Cache timeouts.
Unexpected error occurred in method 'GetObject' , usage 'Distributed Logon Token Cache' - Exception 'Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode:SubStatus:The request timed out.. Additional Information : The client was trying to communicate with the server : net.tcp://SP2013Dev.DMC.local:22233 at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody, RequestBody reqBody) at Microsoft.ApplicationServer.Caching.DataCache.InternalGet(String key, DataCacheItemVersion& version, String region, IMonitoringListener listener) at Microsoft.ApplicationServer.Caching.DataCache.<>c__DisplayClass49.b__48() at Microsoft.SharePoint.DistributedCaching.SPDistributedCache.GetObject(String key)'.
To fix these errors, you need to increase the Request Timeout limit for the relevant cache. You can do so using the following commands.
$LogonTokenCache = Get-SPDistributedCacheClientSetting -ContainerType DistributedLogonTokenCache $LogonTokenCache.RequestTimeout = 300 Set-SPDistributedCacheClientSetting -ContainerType DistributedLogonTokenCache -DistributedCacheClientSettings $LogonTokenCache
$ViewStateCache = Get-SPDistributedCacheClientSetting -ContainerType DistributedViewStateCache $ViewStateCache.RequestTimeout = 300 Set-SPDistributedCacheClientSetting -ContainerType DistributedViewStateCache -DistributedCacheClientSettings $ViewStateCache
By running Get-SPDistributedCacheClientSetting -ContainerType CacheName you are able to see what the current request timeout is set to. By default, Microsoft sets the value of the RequestTimeout to 20 milliseconds. As a result, if the server is under any significant stress you will start seeing a lot of misses on calls to the cache.
PS C:\Users\user> $viewstatecache ChannelInitializationTimeout : 60000 ConnectionBufferSize : 131072 MaxBufferPoolSize : 268435456 MaxBufferSize : 8388608 MaxOutputDelay : 2 ReceiveTimeout : 60000 ChannelOpenTimeOut : 20 RequestTimeout : 300 MaxConnectionsToServer : 99
Learn more about DMC's SharePoint Consulting Services and Digital Workplace Solutions team. Contact us for your next custom SharePoint project.