Perfology Manual [Draft]

by Andrey Akinshin

In this manual, I collect information on performance of complex systems. Performance issues emerge in software development, businesses, plants, research, and other areas of life. Mathematical models and approaches are similar across different domains. Spotting similarities helps to develop a good decision-making intuition. Discussed topics involve different disciplines, so I join related content under the name “perfology.”

Indirect Control

In complex system, it’s impossible to adjust the system directly. Instead, we manipulate the environment to nudge the system to the desired state. To improve the system performance, we should focus on the factors that define the performance derivatives.

Example

A car drives on a road. The driver wants to reach the destination as fast as possible.

Let $f(t)$ be a function of drived distance at time $t$. Then the first derivative¹ $f'(t) = v(t)$ is the car speed. It’s impossible to instantly change the speed to a higher value. The second derivative $f''(t) = a(t)$ is the car acceleration. The driver can control the acceleration by pressing the gas and brake pedals.

The traffic police wants to limit the max speed to reduce the risk of accidents. They can’t directly control the car acceleration like the driver. But they can put some road signs and indirectly affect the car acceleration near them. They have impact on the third derivative $f'''(t) = j(t)$, the car jerk.

The people want to have optimal driving regulations. They want to have the max allowed speed that guarantees the road safety and high-capacity traffic flow. The goverment introduces unified road rules with the help of traffic safety experts, transportation engineers, legal experts, behavioral scientists, and others. While all these people do not have phisical control over the car pedals, they contribute to the rules and indirectly affect the final driver speed. They have impact on the snap, crackle, and pop and higher derivatives² of f(t).

While we often can’t directly control the low-order derivatives, we can modify the future system performance by influencing the higher-order derivatives. If some of these derivatives are permanently negative, the system is in a divergent state and will collapse one day. Our goal is to make high derivatives permanently positive and ensure decent long-term system performance.

Example

When we publish marketing materials, we can’t control how many people will read them. But we optimize the content for search engines.

At some level of abstraction, we always meet people who implement the system. People are the $n^\textrm{th}$ performance derivative. Always consider the human factor; acknowledge cognitive biases and emotional states. To achieve great performance of the system, we should achieve great performance of the people who are working on it.

Indirect Measurements

The internet attributes the following phrase to Lord Kelvin (William Thomson):

If you can’t measure it, you can’t improve it.

In fact, he said this:

When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of science, whatever the matter may be.

The most existing challenges arise in the areas where direct measurements are not available. It shouldn’t prevent us from improving the system. Firstly, the knowledge of the system often allows us to make right steps using intuition and common sense. Secondly, we can perform indirect measurements of related metrics. Beware: Correlation does not imply causation. Indirect measurements is not the reliable and precise source of truth. But it’s a useful source of insights that allows us to proceed through the fog of war.

Example

A ship captain without a compass can navigate by the stars.

Example

In a large organization, it’s impossible to measure the employee mindset and culture directly. But we can conduct employee surveys and collect some data. These data don’t accurately describe people thoughts, but it provides approximate aggregation of people sentiment and gives initial insights on major pain points.

Use this guides to conduct effective surveys:
How To Conduct Organizational Surveys · 1997 · Jack E. Edwards et al.
Employee surveys that work · 2014 · Alec Levenson

Example

A software application works slowly. A software developer can reproduce the slow use-case, profile the application, and find the bottleneck to fix. Profilers are lying and we cannot trust obtained snapshots. However, it’s a good point to start an investigation.

Acknowledge the observer effect: the act of measurements affects the system state and changes the results.

Example

When you want to measure voltage in an electric circuit, you connect a voltmeter to the circuit, but this changes the circuit and, therefore, affects the original voltage.

Example

When you profile an application, the profiler instruments the executed code and affect the measurements.

Example

The question statements in a survey affect the mood of the respondents.

Team Effectiveness

Focus on building an efficient team around you. It doesn’t matter how well a single person is performing while the whole team is dysfunctional. It’s in everyone’s best interest to invest in building healthy and productive teams and communities. Efficient collaboration is a necessity.

Boeing 777 Flight Manual · 2002
Selected advice on the team effectiveness (slightly edited):

Crew Effectiveness Markers
The markers were structured in a checklist format for ease of use and recall. Crewmembers should use the markers as a checklist for decision-making and as a guide for crew briefings. They should be reviewed periodically.
The Crew Effectiveness Markers can also serve as a debriefing tool after a line flight or training event. A debrief should always be conducted after a flight which challenged a crew in some manner. Potential exists for valuable new learning if a crew conducts a frank yet positive self-evaluation following significant flight events. Debriefings should be conducted by the Captain, but may be initiated by anyone in the crew. Frequent, open communications and active listening are key characteristics of the most effective flight crews.
Overall Technical Proficiency
• Set a professional example
• Demonstrate high level of flying skills
• Be adept at normal and non-normal procedures
• Maintain thorough systems knowledge
Briefing and Communication
• Set an open tone
• Fully brief operational/safety issues
• Explicitly encourage participation
• All are obligated to seek and give information
• Include cabin crew
Leadership and Teamwork
• Balance authority and assertiveness
• Promote continual dialogue
• Adapt to the personalities of others
• Must share doubts with others
Situational Awareness
• Anticipate required actions
• Ask the right questions
• Test assumptions, confirm understanding
• Monitor workload distribution and fellow crewmembers
• Report fatigue, stress, and overload in self and others
Decision Making
• Fly the aircraft
• Obtain all pertinent information
• All crewmembers state recommendations
• Better idea suggested? Abandon yours
• Clearly state plan or intentions
• Establish Bottom Lines
• Resolve conflicts and doubts quickly
Crew Self-Evaluation
• Debrief key events
• Continuously provide information to self-correct
• Openly discuss successes and mistakes
• Ask, “How could we have done better?”
• Discuss what is right, not who is right

Sabotage

Sabotage is an activity performed against the goals. Should be avoided.

Unintentional sabotage and self-sabotage are frequent in the presence of stress factors. Often caused by cognitive biases, emotional states, lack of feedback, and inbalanced rules. Research sabotage techniques and train the brain to recognize their patterns.

The Billion Dollar Spy · 2015 · David E. Hoffman
How a single unhappy employee can nullify all the efforts.

Simple Sabotage Field Manual · 1944 · United States Office of Strategic Services
The US intelligence crafted this manual during WW2. Read “General Interference with Organizations and Production” carefully. Think about the authors’s mindset. They were intentionally striving to invent the best sabotage techniques. This guide is the best of what they could come up with.

The Trial · 1925 · Franz Kafka
The Castle · 1926 · Franz Kafka
The consequences of bureaucratic sabotage.

Advantages of incompetent management · 2024-07-04 · Yossi Kreinin
Sabotage by incompetent management.

The Art of Always Being Right · 1831 · Arthur Schopenhauer
The 48 Laws of Power · 1998 · Robert Greene
Praise of Folly · 1508 · Erasmus