Perfology Manual [Draft]
In this manual, I collect information on performance of complex systems. Performance issues emerge in software development, businesses, plants, research, and other areas of life. Mathematical models and approaches are similar across different domains. Spotting similarities helps to develop a good decision-making intuition. Discussed topics involve different disciplines, so I join related content under the name “perfology.”
Indirect Control
In complex system, it’s impossible to adjust the system directly. Instead, we manipulate the environment to nudge the system to the desired state. To improve the system performance, we should focus on the factors that define the performance derivatives.
A car drives on a road. The driver wants to reach the destination as fast as possible.
Let $f(t)$ be a function of drived distance at time $t$. Then the first derivative1 $f'(t) = v(t)$ is the car speed. It’s impossible to instantly change the speed to a higher value. The second derivative $f''(t) = a(t)$ is the car acceleration. The driver can control the acceleration by pressing the gas and brake pedals.
The traffic police wants to limit the max speed to reduce the risk of accidents. They can’t directly control the car acceleration like the driver. But they can put some road signs and indirectly affect the car acceleration near them. They have impact on the third derivative $f'''(t) = j(t)$, the car jerk.
The people want to have optimal driving regulations. They want to have the max allowed speed that guarantees the road safety and high-capacity traffic flow. The goverment introduces unified road rules with the help of traffic safety experts, transportation engineers, legal experts, behavioral scientists, and others. While all these people do not have phisical control over the car pedals, they contribute to the rules and indirectly affect the final driver speed. They have impact on the snap, crackle, and pop and higher derivatives2 of f(t).
While we often can’t directly control the low-order derivatives, we can modify the future system performance by influencing the higher-order derivatives. If some of these derivatives are permanently negative, the system is in a divergent state and will collapse one day. Our goal is to make high derivatives permanently positive and ensure decent long-term system performance.
When we publish marketing materials, we can’t control how many people will read them. But we optimize the content for search engines.
At some level of abstraction, we always meet people who implement the system. People are the $n^\textrm{th}$ performance derivative. Always consider the human factor; acknowledge cognitive biases and emotional states. To achieve great performance of the system, we should achieve great performance of the people who are working on it.
Indirect Measurements
The internet attributes the following phrase to Lord Kelvin (William Thomson):
If you can’t measure it, you can’t improve it.
When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of science, whatever the matter may be.
The most existing challenges arise in the areas where direct measurements are not available. It shouldn’t prevent us from improving the system. Firstly, the knowledge of the system often allows us to make right steps using intuition and common sense. Secondly, we can perform indirect measurements of related metrics. Beware: Correlation does not imply causation. Indirect measurements is not the reliable and precise source of truth. But it’s a useful source of insights that allows us to proceed through the fog of war.
A ship captain without a compass can navigate by the stars.
In a large organization, it’s impossible to measure the employee mindset and culture directly. But we can conduct employee surveys and collect some data. These data don’t accurately describe people thoughts, but it provides approximate aggregation of people sentiment and gives initial insights on major pain points.
Use this guides to conduct effective surveys:
How To Conduct Organizational Surveys · 1997
· Jack E. Edwards
et al.
Employee surveys that work · 2014
· Alec Levenson
A software application works slowly. A software developer can reproduce the slow use-case, profile the application, and find the bottleneck to fix. Profilers are lying and we cannot trust obtained snapshots. However, it’s a good point to start an investigation.
Acknowledge the observer effect: the act of measurements affects the system state and changes the results.
When you want to measure voltage in an electric circuit, you connect a voltmeter to the circuit, but this changes the circuit and, therefore, affects the original voltage.
When you profile an application, the profiler instruments the executed code and affect the measurements.
The question statements in a survey affect the mood of the respondents.
Team Effectiveness
Focus on building an efficient team around you. It doesn’t matter how well a single person is performing while the whole team is dysfunctional. It’s in everyone’s best interest to invest in building healthy and productive teams and communities. Efficient collaboration is a necessity.
Boeing 777 Flight Manual · 2002
Selected advice on the team effectiveness (slightly edited):
Crew Effectiveness Markers
The markers were structured in a checklist format for ease of use and recall. Crewmembers should use the markers as a checklist for decision-making and as a guide for crew briefings. They should be reviewed periodically.
The Crew Effectiveness Markers can also serve as a debriefing tool after a line flight or training event. A debrief should always be conducted after a flight which challenged a crew in some manner. Potential exists for valuable new learning if a crew conducts a frank yet positive self-evaluation following significant flight events. Debriefings should be conducted by the Captain, but may be initiated by anyone in the crew. Frequent, open communications and active listening are key characteristics of the most effective flight crews.
Overall Technical Proficiency
• Set a professional example
• Demonstrate high level of flying skills
• Be adept at normal and non-normal procedures
• Maintain thorough systems knowledgeBriefing and Communication
• Set an open tone
• Fully brief operational/safety issues
• Explicitly encourage participation
• All are obligated to seek and give information
• Include cabin crewLeadership and Teamwork
• Balance authority and assertiveness
• Promote continual dialogue
• Adapt to the personalities of others
• Must share doubts with othersSituational Awareness
• Anticipate required actions
• Ask the right questions
• Test assumptions, confirm understanding
• Monitor workload distribution and fellow crewmembers
• Report fatigue, stress, and overload in self and othersDecision Making
• Fly the aircraft
• Obtain all pertinent information
• All crewmembers state recommendations
• Better idea suggested? Abandon yours
• Clearly state plan or intentions
• Establish Bottom Lines
• Resolve conflicts and doubts quicklyCrew Self-Evaluation
• Debrief key events
• Continuously provide information to self-correct
• Openly discuss successes and mistakes
• Ask, “How could we have done better?”
• Discuss what is right, not who is right
Sabotage
Sabotage is an activity performed against the goals. Should be avoided.
Unintentional sabotage and self-sabotage are frequent in the presence of stress factors. Often caused by cognitive biases, emotional states, lack of feedback, and inbalanced rules. Research sabotage techniques and train the brain to recognize their patterns.
The Billion Dollar Spy · 2015
· David E. Hoffman
How a single unhappy employee can nullify all the efforts.
Simple Sabotage Field Manual · 1944
· United States Office of Strategic Services
The US intelligence crafted this manual during WW2.
Read “General Interference with Organizations and Production” carefully.
Think about the authors’s mindset.
They were intentionally striving to invent the best sabotage techniques.
This guide is the best of what they could come up with.
The Trial · 1925
· Franz Kafka
The Castle · 1926
· Franz Kafka
The consequences of bureaucratic sabotage.
Advantages of incompetent management · 2024-07-04
· Yossi Kreinin
Sabotage by incompetent management.
The Art of Always Being Right · 1831
· Arthur Schopenhauer
The 48 Laws of Power · 1998
· Robert Greene
Praise of Folly · 1508
· Erasmus
Work-In-Progress
!!! More sections are to be added soon.
References
Statistics
Statistics Manual [Draft]
Statistics Done Wrong · 2015
· Alex Reinhart
The Lady Tasting Tea · 2002
· David Salsburg
Statistical Consequences of Fat Tails · 2020
· Nassim Nicholas Taleb
Introduction to Robust Estimation and Hypothesis Testing · 2021
· Rand R. Wilcox
Software development
Systems Performance · 2020
· Brendan Gregg
Designing Data-Intensive Applications · 2017
· Martin Kleppmann
High Performance Web Sites · 2007-09-01
· Steve Souders
Performance Analysis and Tuning on Modern CPUs · 2020
· Denis Bakhvalov
Optimizing subroutines in assembly language. An optimization guide for x86 platforms · 2023
· Agner Fog
Instruction tables. Lists of instruction latencies, throughputs and micro-operation breakdowns ... · 2022
· Agner Fog
Intel 64 and ia-32 architectures software developer’s manual · 2011
· Part Guide
.NET
Pro .NET Benchmarking · 2019
Pro .NET Memory Management · 2018
· Konrad Kokosa
Pro .NET Performance · 2012
· Sasha Goldshtein
A curated list of awesome .NET Performance stuff
· Adam Sitnik
Performance Improvements in Organizations
Out of the Crisis · 1982
· W. Edwards Deming
The New Economics · 1991
· W. Edwards Deming
Gemba Kaizen · 1997
· Masaaki Imai
The Goal · 1984
· Eliyahu M Goldratt
et al.
The Logical Thinking Process · 2007
· H. William Dettmer
The Deadline · 1997
· Tom DeMarco
Managing Corporate Lifecycles · 1998
· Ichak Kalderon Adizes
Development Performance
Accelerate · 2018
· Nicole Forsgren
et al.
Effective Devops · 2015
· Jennifer Davis
et al.
Culture in Organization
Performance Culture · 2016-04-10
· Joe Duffy
The Secrets of High Performing Technology Organizations
· Jez Humble
A typology of organisational cultures · 2004
· R Westrum
On failure and resilience · 2017-07-17
· Ryn Daniels
Netflix Culture — The Best Work of Our Lives
Organizational culture · 2013
· Samson Ibidunni
et al.
The Culture Code · 2017
· Daniel Coyle
The Power of Company Culture · 2023
· Chris Dyer
Epistemology
The Structure of Scientific Revolutions · 1996
· Thomas S. Kuhn
Against Method · 1975
· Paul Karl Feyerabend
The Logic of Scientific Discovery · 1934
· Karl Popper
The Psychology of Invention in the Mathematical Field · 1945
· Jacques Hadamard
Misc
Nudge · 2009
· Richard H. Thaler
Atomic Habits · 2018
· James Clear
Flow · 1990
· Mihály Csíkszentmihályi
The Design of Everyday Things · 1988
· Donald A. Norman
Fooled by Randomness · 2004
· Nassim Nicholas Taleb
Antifragile · 2012
· Nassim Nicholas Taleb
The Black Swan · 2010
· Nassim Nicholas Taleb
Skin in the Game · 2018
· Nassim Nicholas Taleb
Mistakes Were Made, but Not by Me · 2007
· Carol Tavris
et al.
"Surely You're Joking, Mr. Feynman!" · 1985
· Richard P. Feynman
"What Do You Care What Other People Think?" · 2018
· Richard P. Feynman
et al.
Meditations · 181
· Marcus Aurelius
Zen and the Art of Motorcycle Maintenance · 1974
· Robert M. Pirsig
Zen in the Art of Archery · 1948
· Eugen Herrigel
Gödel, Escher, Bach · 1999
· Douglas R. Hofstadter
Escape from Freedom · 1941
· Erich Fromm
The Courage to Be Disliked · 2019
· Ichiro Kishimi
Complexity · 2009
· Melanie Mitchell
Predictably Irrational · 2008
· Dan Ariely
Footnotes
Watch Essence of calculus by 3Blue1Brown ↩︎
Watch Higher order derivatives by 3Blue1Brown ↩︎