Performance Unpredictability

I was hearing a talk from Joshua Bloch on Performance Anxiety and my key observations from the discussion was

  1. It has become impossible to estimate performance
  2. Performance is becoming more abstract
  3. Measure and use statistics with measures

Joshua talks about various aspects of the current systems and how these systems today have lead to a situation where we can not predict or estimate performance of a section of code. And is simply because we have so many layers of code, libraries, patterns, app servers and JVM and what not. The same set of code can have varied performance on a different machine just because how a JVM is going to interpret the code.

He also speaks very clearly on the current known facts and how can they simply be myths – profilers, app servers. There are studies and papers he mentions that speak about this area of uncertainty.

He sumarizes

Our results are disturbing because they indicate that profiler incorrectness is pervasive—occurring in most of our seven benchmarks and in two production JVM—-and significant—all four of the state-of-the-art profilers produce incorrect profiles. Incorrect profiles can easily cause a performance analyst to spend time optimizing cold methods that will have minimal effect on performance. We show that a proof-of-concept profiler that does not use yield points for sampling does not suffer from the above problems.

My Perspective

I chose to blog about this, is because for many years now, I have  been working on making applications performant in many ways – Databases, Application layers, Web – client and servers. In my experience, I can not more agree with Joshua and agree that many times, I have been baffled myself where performance models that I have used to measure performance in a QA or a staging environment came back to be different in production of the various constants I had with the hardware and many times, I had to go back to Production and see what is happening in the real world – One of the biggest factors that I simply could not replicate was the “user base” of the systems in any of the previous systems. I just could not ever stop measuring the performance of the systems and analyze the data only to find myself tuning certain areas of the application.

I will strongly recommend you all to go through his talk

%d bloggers like this: