RPC averaged latency performance counter really high…

Share this:

Hi folks,

Today I got a weird issue that took me quite a long time to figure it out and I would like to share with you guys. Everything started when a brand new DAG was deployed and everything went just fine during the installation process however during the initial testing I noticed a RPC averaged latency performance counter really high.

In the figure below you can see how bad was it and I the server had just a few users that were part of my pre-pilot phase.

image

I’ve done a couple of tests, as follows:

– First, I ran the Get-MailboxDatabaseCopyStatus and everything was golden

– I moved the databases to a single node, and then noticed that the utilization kept the same

– I restarted the RPC Client access and the same issue were the performance counter didn’t change (expected)

– I restarted the Information Store and then the performance counter went to 0 but as soon as I moved any DB the number would increase again.

– I ran Test-ReplicationHealth and no issues at all

– I use PAL tool to analyse performance and the disks were just fine

– I ran netstat to see the connections and nothing was outstanding

– Exchange Best Practices Analyzer

In this environment that I was working they have a LTM (Local Traffic Manager) and then I noticed that I could access the Domain Controller (\unc) however I couldn’t ping. I double check and there was a SNAT and a Virtual Server were in place, the missing piece was the ability to ping the Domain Controllers, so in a F5 we should go to System / Configuration / Local Traffic and change the default behaviour of the box to All traffic in the SNAT Packet Forwarding setting.

image_thumb[1]

After performing that change I moved the databases around and the RPC averaged latency came back to 0 Smile

Cheers,
Anderson Patricio
http://www.andersonpatricio.ca
http://www.andersonpatricio.org (Portuguese)
Twitter: @apatricio

Written by Anderson Patricio

Anderson Patricio

Anderson Patricio is a Canadian MVP in Cloud and Datacenter Management, and Office Server and Services, besides the Microsoft Award he also holds a Solutions Master (MCSM) in Exchange and several other certifications. Anderson has been contributing to the Microsoft Community with articles, tutorials, blog posts, twitter, forums and book reviews. He is a regular contributor here at ITPROCentral.com, MSExchange.org, Techgenix.com and Anderson Patricio.org (Portuguese).