I was looking at our DevOps dashboards and saw some really weird patterns:

So I pinged my colleague who owns this service and he noticed it was actual very predictable:

Like clockwork, once a minute - he went further and got a PerfView which showed high contention on a newly added ConcurrentDictionary:

He then asked me to take a look since that ConcurrentDictionary was added on my suggestion to work around another issue (which I will blog about one day). Having had that problem before, I figured we either had a hot spot or a hashing function problem - so I got a dump of the process to see which (I could have saved some time and looked at the source...but as they say, there's nothing like a good dump).

0:000> !do 0000015d19e676f0
Name: System.Collections.Concurrent.ConcurrentDictionary`2
MethodTable: 00007ff8e18f9728
EEClass: 00007ff8e18c5de0
Size: 64(0x40) bytes
File: D:WindowsMicrosoft.NetassemblyGAC_64mscorlibv4.0_4.0.0.0__b77a5c561934e089mscorlib.dll
Fields:
 MT              Field   Offset Type                 VT Attr     Value            Name
00007ff8e187d930 4001830 8      ....Byte, mscorlib]] 0  instance 0000015e3a819078 m_tables
00007ff93b3c4300 4001831 10     ...Canon, mscorlib]] 0  instance 0000000000000000 m_comparer
00007ff93b3b1f28 4001832 30     System.Boolean       1  instance 1                m_growLockArray
00007ff93b3a9288 4001833 20     System.Int32         1  instance 0                m_keyRehashCount
00007ff93b3a9288 4001834 24     System.Int32         1  instance 256              m_budget
0000000000000000 4001835 18     SZARRAY              0  instance 0000000000000000 m_serializationArray
00007ff93b3a9288 4001836 28     System.Int32         1  instance 0                m_serializationConcurrencyLevel
00007ff93b3a9288 4001837 2c     System.Int32         1  instance 0                m_serializationCapacity
00007ff93b3b1f28 400183b 10     System.Boolean       1  static   
0:000> !do 0000015e3a819078
Name: System.Collections.Concurrent.ConcurrentDictionary`2+Tables
MethodTable: 00007ff8e18fafd0
EEClass: 00007ff8e18c6ab0
Size: 48(0x30) bytes
File: D:WindowsMicrosoft.NetassemblyGAC_64mscorlibv4.0_4.0.0.0__b77a5c561934e089mscorlib.dll
Fields:
 MT              Field   Offset Type            VT Attr     Value            Name
0000000000000000 400341d 8      SZARRAY         0  instance 0000015e3a816ca8 m_buckets
00007ff93b3a6fc0 400341e 10     System.Object[] 0  instance 0000015e3a816290 m_locks
00007ff93b3a9220 400341f 18     System.Int32[]  0  instance 0000015e3a817e28 m_countPerLock
00007ff93b3c4300 4003420 20...Canon, mscorlib]] 0  instance 0000015d19e677a0 m_comparer
0:000> !DumpArray 0000015e3a816ca8
Name: System.Collections.Concurrent.ConcurrentDictionary`2+Node
MethodTable: 00007ff8e18fae50
EEClass: 00007ff93ad6aa00
Size: 4480(0x1180) bytes
Array: Rank 1, Number of elements 557, Type CLASS
Element Methodtable: 00007ff8e18fad88
[0] null
[1] null
<...>
[428] null
[429] 0000015d46de8280
[430] null
<...>
[556] null
0:000> !do 0000015d46de8280
Name: System.Collections.Concurrent.ConcurrentDictionary`2+Node
MethodTable: 00007ff8e18fad88
EEClass: 00007ff8e18c6990
Size: 40(0x28) bytes
File: D:WindowsMicrosoft.NetassemblyGAC_64mscorlibv4.0_4.0.0.0__b77a5c561934e089mscorlib.dll
Fields:
 MT              Field   Offset Type             VT Attr     Value            Name
00007ff93b3abf10 4003421 8      System.__Canon   0  instance 0000015d46de8240 m_key
00007ff93b3a8940 4003422 1c     System.Byte      1  instance 0                m_value
00007ff8e187d800 4003423 10 ....Byte, mscorlib]] 0  instance 0000015e47c672e8 m_next
00007ff93b3a9288 4003424 18     System.Int32     1  instance 37103870         m_hashcode
0:000> !do 0000015e47c672e8
Name: System.Collections.Concurrent.ConcurrentDictionary`2+Node
MethodTable: 00007ff8e18fad88
EEClass: 00007ff8e18c6990
Size: 40(0x28) bytes
File: D:WindowsMicrosoft.NetassemblyGAC_64mscorlibv4.0_4.0.0.0__b77a5c561934e089mscorlib.dll
Fields:
 MT              Field   Offset  Type            VT Attr    Value            Name
00007ff93b3abf10 4003421 8       System.__Canon  0 instance 0000015e47c672a8 m_key
00007ff93b3a8940 4003422 1c      System.Byte     1 instance 0                m_value
00007ff8e187d800 4003423 10 ....Byte, mscorlib]] 0 instance 0000015e4781f3f8 m_next
00007ff93b3a9288 4003424 18      System.Int32    1 instance 37103870         m_hashcode

Sure enough, there's only one bucket occupied, and all of the items in that bucket have the same hash code, so our ConcurrentDictionary is really a giant linked list, with a giant lock on top...

Our ConcurrentDictionary's keys are System.EventHandler which is really a delegate - which default HashCode implementation is...the hash code of the underlying type which means all of our delegates have the same hashcode, hence the same bucket....DOH!