This issue has been previously discussed within this post First Post on DCache Issues and Fix. After being buried within that project on Azure for a while, I forgot to post some updates to the fix discussed on that post. Basically it didn’t fix the whole problem.

Currently in the process of building a similar SharePoint 2013 platform on Azure and I noticed that Distributed Cache has risen from the ashes to give me some sleepless nights again.

Symptoms

  1. Authenticated users are challenged to re-authenticate themselves while browsing through the site.
  2. Different types of ULS log errors

    1
    2
    3
    security token '0e.t|xxxx_Default' is found in the local cache, but it is expired. Returing Null..
    Unexpected error occurred in method 'Put' , usage 'SPViewStateCache' - Exception 'Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0018>:SubStatus<ES0001>:The request timed out.. Additional Information : The client was trying to communicate with the server : net.tcp://xxxxx:22233
    Unexpected error occurred in method 'GetObject' , usage 'Distributed Logon Token Cache' - Exception 'Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0016>:SubStatus<ES0001>:The connection was terminated, possibly due to server or network problems or serialized Object size is greater than MaxBufferSize on server. Result of the request is unknown.

    Default Distributed Cache Settings:

    The default settings for Distributed Cache is shown below. This has been taken from a SharePoint server that has not been modified.. Yet!

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    119
    120
    121
    122
    DistributedDefaultCache
    ChannelInitializationTimeout : 60000
    ConnectionBufferSize : 131072
    MaxBufferPoolSize : 268435456
    MaxBufferSize : 8388608
    MaxOutputDelay : 2
    ReceiveTimeout : 60000
    ChannelOpenTimeOut : 3000
    RequestTimeout : 3000
    MaxConnectionsToServer : 8

    DistributedAccessCache
    ChannelInitializationTimeout : 60000
    ConnectionBufferSize : 131072
    MaxBufferPoolSize : 268435456
    MaxBufferSize : 8388608
    MaxOutputDelay : 2
    ReceiveTimeout : 60000
    ChannelOpenTimeOut : 3000
    RequestTimeout : 3000
    MaxConnectionsToServer : 8

    DistributedActivityFeedCache
    ChannelInitializationTimeout : 60000
    ConnectionBufferSize : 131072
    MaxBufferPoolSize : 268435456
    MaxBufferSize : 8388608
    MaxOutputDelay : 2
    ReceiveTimeout : 60000
    ChannelOpenTimeOut : 3000
    RequestTimeout : 3000
    MaxConnectionsToServer : 8

    DistributedBouncerCache
    ChannelInitializationTimeout : 60000
    ConnectionBufferSize : 131072
    MaxBufferPoolSize : 268435456
    MaxBufferSize : 8388608
    MaxOutputDelay : 2
    ReceiveTimeout : 60000
    ChannelOpenTimeOut : 3000
    RequestTimeout : 3000
    MaxConnectionsToServer : 8

    DistributedLogonTokenCache
    ChannelInitializationTimeout : 60000
    ConnectionBufferSize : 131072
    MaxBufferPoolSize : 268435456
    MaxBufferSize : 8388608
    MaxOutputDelay : 2
    ReceiveTimeout : 60000
    ChannelOpenTimeOut : 20
    RequestTimeout : 20
    MaxConnectionsToServer : 8

    DistributedServerToAppServerAccessTokenCache
    ChannelInitializationTimeout : 60000
    ConnectionBufferSize : 131072
    MaxBufferPoolSize : 268435456
    MaxBufferSize : 8388608
    MaxOutputDelay : 2
    ReceiveTimeout : 60000
    ChannelOpenTimeOut : 3000
    RequestTimeout : 3000
    MaxConnectionsToServer : 8

    DistributedSearchCache
    ChannelInitializationTimeout : 60000
    ConnectionBufferSize : 131072
    MaxBufferPoolSize : 268435456
    MaxBufferSize : 8388608
    MaxOutputDelay : 2
    ReceiveTimeout : 60000
    ChannelOpenTimeOut : 3000
    RequestTimeout : 3000
    MaxConnectionsToServer : 8

    DistributedSecurityTrimmingCache
    ChannelInitializationTimeout : 60000
    ConnectionBufferSize : 131072
    MaxBufferPoolSize : 268435456
    MaxBufferSize : 8388608
    MaxOutputDelay : 2
    ReceiveTimeout : 60000
    ChannelOpenTimeOut : 3000
    RequestTimeout : 3000
    MaxConnectionsToServer : 8

    DistributedActivityFeedLMTCache
    ChannelInitializationTimeout : 60000
    ConnectionBufferSize : 131072
    MaxBufferPoolSize : 268435456
    MaxBufferSize : 8388608
    MaxOutputDelay : 2
    ReceiveTimeout : 60000
    ChannelOpenTimeOut : 3000
    RequestTimeout : 3000
    MaxConnectionsToServer : 8

    DistributedViewStateCache
    ChannelInitializationTimeout : 60000
    ConnectionBufferSize : 131072
    MaxBufferPoolSize : 268435456
    MaxBufferSize : 8388608
    MaxOutputDelay : 2
    ReceiveTimeout : 60000
    ChannelOpenTimeOut : 20
    RequestTimeout : 20
    MaxConnectionsToServer : 8
    <h1>Resolution:</h1>
    Let’s start to tweek with the following
    <code>
    DistributedLogonTokenCache
    ChannelInitializationTimeout : 60000
    ConnectionBufferSize : 131072
    MaxBufferPoolSize : 268435456
    MaxBufferSize : 8388608
    MaxOutputDelay : 2
    ReceiveTimeout : 60000
    ChannelOpenTimeOut : 20
    RequestTimeout : 20
    MaxConnectionsToServer : 8

    From the above there are three settings that need modification

    1
    2
    3
    ReceiveTimeout : 60000
    ChannelOpenTimeOut : 20
    RequestTimeout : 20

    These need to reflect the following:

    1
    2
    3
    ReceiveTimeout : 82800000 (This makes the receive timeout value to be 23 hours)
    ChannelOpenTimeOut : 3000 (This increases the value from 20ms to 3 seconds)
    RequestTimeout : 3000

    This can be achieved by doing the following:

    1
    2
    3
    4
    5
    6
    7
    8
    $set = Get-SPDistributedCacheClientSetting -ContainerType DistributedLogonTokenCache
    $set.MaxBufferPoolSize = 1073741824
    $set.MaxBufferSize = 33554432
    $set.ReceiveTimeout = 82800000
    $set.ChannelOpenTimeOut = 3000
    $set.RequestTimeout = 3000
    $set.MaxConnectionsToServer = 16
    Set-SPDistributedCacheClientSetting -ContainerType DistributedLogonTokenCache -DistributedCacheClientSettings $set

    Note:

    Though there are no adverse effect of making the changes reflected below to any new farm, I recommend doing this only for farms that have had issues that are similar to the symptoms spoken above. Also make these changes while keeping an eye out on the ULS logs as well, can get a good idea of what’s happening after these changes are happening.

    Also try to pull logs from all the servers at one spot. Use the following code to do the same

    1
    Merge-SPLogFile -Path 'LOG_LOCATION.log' -Overwrite -StartTime '01/13/2015 18:10' -EndTime '01/13/2015 18:22'

    Working Distributed Cache Settings

    I have resolved the issue within the farm that I am building and below are the current Distributed Cache settings.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    DistributedDefaultCache
    ChannelInitializationTimeout : 60000
    ConnectionBufferSize : 131072
    MaxBufferPoolSize : 268435456
    MaxBufferSize : 8388608
    MaxOutputDelay : 2
    ReceiveTimeout : 82800000
    ChannelOpenTimeOut : 3000
    RequestTimeout : 3000
    MaxConnectionsToServer : 8

    DistributedAccessCache
    ChannelInitializationTimeout : 60000
    ConnectionBufferSize : 131072
    MaxBufferPoolSize : 268435456
    MaxBufferSize : 8388608
    MaxOutputDelay : 2
    ReceiveTimeout : 82800000
    ChannelOpenTimeOut : 3000
    RequestTimeout : 3000
    MaxConnectionsToServer : 8

    DistributedActivityFeedCache
    ChannelInitializationTimeout : 120000
    ConnectionBufferSize : 131072
    MaxBufferPoolSize : 268435456
    MaxBufferSize : 16777216
    MaxOutputDelay : 2
    ReceiveTimeout : 82800000
    ChannelOpenTimeOut : 3000
    RequestTimeout : 3000
    MaxConnectionsToServer : 8

    DistributedBouncerCache
    ChannelInitializationTimeout : 60000
    ConnectionBufferSize : 131072
    MaxBufferPoolSize : 268435456
    MaxBufferSize : 8388608
    MaxOutputDelay : 2
    ReceiveTimeout : 82800000
    ChannelOpenTimeOut : 3000
    RequestTimeout : 3000
    MaxConnectionsToServer : 8

    DistributedLogonTokenCache
    ChannelInitializationTimeout : 60000
    ConnectionBufferSize : 131072
    MaxBufferPoolSize : 1073741824
    MaxBufferSize : 33554432
    MaxOutputDelay : 2
    ReceiveTimeout : 82800000
    ChannelOpenTimeOut : 3000
    RequestTimeout : 3000
    MaxConnectionsToServer : 16

    DistributedServerToAppServerAccessTokenCache
    ChannelInitializationTimeout : 60000
    ConnectionBufferSize : 131072
    MaxBufferPoolSize : 268435456
    MaxBufferSize : 8388608
    MaxOutputDelay : 2
    ReceiveTimeout : 82800000
    ChannelOpenTimeOut : 3000
    RequestTimeout : 3000
    MaxConnectionsToServer : 8

    DistributedSearchCache
    ChannelInitializationTimeout : 60000
    ConnectionBufferSize : 131072
    MaxBufferPoolSize : 268435456
    MaxBufferSize : 8388608
    MaxOutputDelay : 2
    ReceiveTimeout : 82800000
    ChannelOpenTimeOut : 3000
    RequestTimeout : 3000
    MaxConnectionsToServer : 8

    DistributedSecurityTrimmingCache
    ChannelInitializationTimeout : 60000
    ConnectionBufferSize : 131072
    MaxBufferPoolSize : 268435456
    MaxBufferSize : 8388608
    MaxOutputDelay : 2
    ReceiveTimeout : 82800000
    ChannelOpenTimeOut : 3000
    RequestTimeout : 3000
    MaxConnectionsToServer : 8

    DistributedActivityFeedLMTCache
    ChannelInitializationTimeout : 120000
    ConnectionBufferSize : 131072
    MaxBufferPoolSize : 268435456
    MaxBufferSize : 16777216
    MaxOutputDelay : 2
    ReceiveTimeout : 82800000
    ChannelOpenTimeOut : 3000
    RequestTimeout : 3000
    MaxConnectionsToServer : 8

    DistributedViewStateCache
    ChannelInitializationTimeout : 120000
    ConnectionBufferSize : 131072
    MaxBufferPoolSize : 268435456
    MaxBufferSize : 33554432
    MaxOutputDelay : 2
    ReceiveTimeout : 82800000
    ChannelOpenTimeOut : 3000
    RequestTimeout : 3000
    MaxConnectionsToServer : 8