https://DevOpsCloud.io -- Cloud Monk Losang Jinpa, Ph.D., MCSE/MCT, GitOps DevOps Engineer

Return to Systems Performance Index, Systems Performance Glossary, Systems Performance, 2nd Edition, Performance Bibliography, Systems Performance, Performance DevOps, IT Bibliography, DevOps Bibliography

“ (SysPrfBGrg 2021)

• Systems Performance Table of Contents

• Systems Performance Preface

• Acknowledgments

• About the Author

• 1 Systems Performance Introduction

• 2 Systems Performance Methodologies

• 3 Systems Performance Operating Systems

• 4 Systems Performance Observability Tools

• 5 Systems Performance Applications

• 6 Systems Performance CPUs

• 7 Systems Performance Memory

• 8 Systems Performance File Systems

• 9 Systems Performance Disks

• 10 Systems Performance Network

• 11 Systems Performance Cloud Computing

• 12 Systems Performance Benchmarking

• 13 Systems Performance perf

• 14 Systems Performance Ftrace

• 15 Systems Performance BPF

• 16 Systems Performance Case Study

• A Systems Performance USE Method: Linux

• B Systems Performance sar Summary

• C Systems Performance bpftrace One-Liners

• D Systems Performance Solutions to Selected Exercises

• E Systems Performance Who’s Who

• Systems Performance Glossary

• Systems Performance Index

• Systems Performance Preface

• Acknowledgments

• About the Author

• 1 Systems Performance Introduction

• 1.1 Systems Performance

• 1.2 Roles

• 1.3 Activities

• 1.4 Perspectives

• 1.5 Performance Is Challenging

• 1.5.1 Subjectivity

• 1.5.2 Complexity

• 1.5.3 Multiple Causes

• 1.5.4 Multiple Performance Issues

• 1.6 Latency

• 1.7 Observability

• 1.7.1 Counters, Statistics, and Metrics

• 1.7.2 Profiling

• 1.7.3 Tracing

• 1.8 Experimentation

• 1.9 Cloud Computing

• 1.10 Methodologies

• 1.10.1 Linux Perf Analysis in 60 Seconds

• 1.11 Case Studies

• 1.11.1 Slow Disks

• 1.11.2 Software Change

• 1.11.3 More Reading

• 1.12 References

• 2 Systems Performance Methodologies

• 2.1 Terminology

• 2.2 Models

• 2.2.1 System Under Test

• 2.2.2 Queueing System

• 2.3 Concepts

• 2.3.1 Latency

• 2.3.2 Time Scales

• 2.3.3 Trade-Offs

• 2.3.4 Tuning Efforts

• 2.3.5 Level of Appropriateness

• 2.3.6 When to Stop Analysis

• 2.3.7 Point-in-Time Recommendations

• 2.3.8 Load vs. Architecture

• 2.3.9 Scalability

• 2.3.10 Metrics

• 2.3.11 Utilization

• 2.3.12 Saturation

• 2.3.13 Profiling

• 2.3.14 Caching

• 2.3.15 Known-Unknowns

• 2.4 Perspectives

• 2.4.1 Resource Analysis

• 2.4.2 Workload Analysis

• 2.5 Methodology

• 2.5.1 Streetlight Anti-Method

• 2.5.2 Random Change Anti-Method

• 2.5.3 Blame-Someone-Else Anti-Method

• 2.5.4 Ad Hoc Checklist Method

• 2.5.5 Problem Statement

• 2.5.6 Scientific Method

• 2.5.7 Diagnosis Cycle

• 2.5.8 Tools Method

• 2.5.9 The USE Method

• 2.5.10 The RED Method

• 2.5.11 Workload Characterization

• 2.5.12 Drill-Down Analysis

• 2.5.13 Latency Analysis

• 2.5.14 Method R

• 2.5.15 Event Tracing

• 2.5.16 Baseline Statistics

• 2.5.17 Static Performance Tuning

• 2.5.18 Cache Tuning

• 2.5.19 Micro-Benchmarking

• 2.5.20 Performance Mantras

• 2.6 Modeling

• 2.6.1 Enterprise vs. Cloud

• 2.6.2 Visual Identification

• 2.6.3 Amdahl’s Law of Scalability

• 2.6.4 Universal Scalability Law

• 2.6.5 Queueing Theory

• 2.7 Capacity Planning

• 2.7.1 Resource Limits

• 2.7.2 Factor Analysis

• 2.7.3 Scaling Solutions

• 2.8 Statistics

• 2.8.1 Quantifying Performance Gains

• 2.8.2 Averages

• 2.8.3 Standard Deviation, Percentiles, Median

• 2.8.4 Coefficient of Variation

• 2.8.5 Multimodal Distributions

• 2.8.6 Outliers

• 2.9 Monitoring

• 2.9.1 Time-Based Patterns

• 2.9.2 Monitoring Products

• 2.9.3 Summary-Since-Boot

• 2.10 Visualizations

• 2.10.1 Line Chart

• 2.10.2 Scatter Plots

• 2.10.3 Heat Maps

• 2.10.4 Timeline Charts

• 2.10.5 Surface Plot

• 2.10.6 Visualization Tools

• 2.11 Exercises

• 2.12 References

• 3 Systems Performance Operating Systems

• 3.1 Terminology

• 3.2 Background

• 3.2.1 Kernel

• 3.2.2 Kernel and User Modes

• 3.2.3 System Calls

• 3.2.4 Interrupts

• 3.2.5 Clock and Idle

• 3.2.6 Processes

• 3.2.7 Stacks

• 3.2.8 Virtual Memory

• 3.2.9 Schedulers

• 3.2.10 File Systems

• 3.2.11 Caching

• 3.2.12 Networking

• 3.2.13 Device Drivers

• 3.2.14 Multiprocessor

• 3.2.15 Preemption

• 3.2.16 Resource Management

• 3.2.17 Observability

• 3.3 Kernels

• 3.3.1 Unix

• 3.3.2 BSD

• 3.3.3 Solaris

• 3.4 Linux

• 3.4.1 Linux Kernel Developments

• 3.4.2 systemd

• 3.4.3 KPTI (Meltdown)

• 3.4.4 Extended BPF

• 3.5 Other Topics

• 3.5.1 PGO Kernels

• 3.5.2 Unikernels

• 3.5.3 Microkernels and Hybrid Kernels

• 3.5.4 Distributed Operating Systems

• 3.6 Kernel Comparisons

• 3.7 Exercises

• 3.8 References

• 3.8.1 Additional Reading

• 4 Systems Performance Observability Tools

• 4.1 Tool Coverage

• 4.1.1 Static Performance Tools

• 4.1.2 Crisis Tools

• 4.2 Tool Types

• 4.2.1 Fixed Counters

• 4.2.2 Profiling

• 4.2.3 Tracing

• 4.2.4 Monitoring

• 4.3 Observability Sources

• 4.3.1 /proc

• 4.3.2 /sys

• 4.3.3 Delay Accounting

• 4.3.4 netlink

• 4.3.5 Tracepoints

• 4.3.6 kprobes

• 4.3.7 uprobes

• 4.3.8 USDT

• 4.3.9 Hardware Counters (PMCs)

• 4.3.10 Other Observability Sources

• 4.4 sar

• 4.4.1 sar(1) Coverage

• 4.4.2 sar(1) Monitoring

• 4.4.3 sar(1) Live

• 4.4.4 sar(1) Documentation

• 4.5 Tracing Tools

• 4.6 Observing Observability

• 4.7 Exercises

• 4.8 References

• 5 Systems Performance Applications

• 5.1 Application Basics

• 5.1.1 Objectives

• 5.1.2 Optimize the Common Case

• 5.1.3 Observability

• 5.1.4 big O Notation

• 5.2 Application Performance Techniques

• 5.2.1 Selecting an I/O Size

• 5.2.2 Caching

• 5.2.3 Buffering

• 5.2.4 Polling

• 5.2.5 Concurrency and Parallelism

• 5.2.6 Non-Blocking I/O

• 5.2.7 Processor Binding

• 5.2.8 Performance Mantras

• 5.3 Programming Languages

• 5.3.1 Compiled Languages

• 5.3.2 Interpreted Languages

• 5.3.3 Virtual Machines

• 5.3.4 Garbage Collection

• 5.4 Methodology

• 5.4.1 CPU Profiling

• 5.4.2 Off-CPU Analysis

• 5.4.3 Syscall Analysis

• 5.4.4 USE Method

• 5.4.5 Thread State Analysis

• 5.4.6 Lock Analysis

• 5.4.7 Static Performance Tuning

• 5.4.8 Distributed Tracing

• 5.5 Observability Tools

• 5.5.1 perf

• 5.5.2 profile

• 5.5.3 offcputime

• 5.5.4 strace

• 5.5.5 execsnoop

• 5.5.6 syscount

• 5.5.7 bpftrace

• 5.6 Gotchas

• 5.6.1 Missing Symbols

• 5.6.2 Missing Stacks

• 5.7 Exercises

• 5.8 References

• 6 Systems Performance CPUs

• 6.1 Terminology

• 6.2 Models

• 6.2.1 CPU Architecture

• 6.2.2 CPU Memory Caches

• 6.2.3 CPU Run Queues

• 6.3 Concepts

• 6.3.1 Clock Rate

• 6.3.2 Instructions

• 6.3.3 Instruction Pipeline

• 6.3.4 Instruction Width

• 6.3.5 Instruction Size

• 6.3.6 SMT

• 6.3.7 IPC, CPI

• 6.3.8 Utilization

• 6.3.9 User Time / Kernel Time

• 6.3.10 Saturation

• 6.3.11 Preemption

• 6.3.12 Priority Inversion

• 6.3.13 Multiprocess, Multithreading

• 6.3.14 Word Size

• 6.3.15 Compiler Optimization

• 6.4 Architecture

• 6.4.1 Hardware

• 6.4.2 Software

• 6.5 Methodology

• 6.5.1 Tools Method

• 6.5.2 USE Method

• 6.5.3 Workload Characterization

• 6.5.4 Profiling

• 6.5.5 Cycle Analysis

• 6.5.6 Performance Monitoring

• 6.5.7 Static Performance Tuning

• 6.5.8 Priority Tuning

• 6.5.9 Resource Controls

• 6.5.10 CPU Binding

• 6.5.11 Micro-Benchmarking

• 6.6 Observability Tools

• 6.6.1 uptime

• 6.6.2 vmstat

• 6.6.3 mpstat

• 6.6.4 sar

• 6.6.5 ps

• 6.6.6 top

• 6.6.7 pidstat

• 6.6.8 time, ptime

• 6.6.9 turbostat

• 6.6.10 showboost

• 6.6.11 pmcarch

• 6.6.12 tlbstat

• 6.6.13 perf

• 6.6.14 profile

• 6.6.15 cpudist

• 6.6.16 runqlat

• 6.6.17 runqlen

• 6.6.18 softirqs

• 6.6.19 hardirqs

• 6.6.20 bpftrace

• 6.6.21 Other Tools

• 6.7 Visualizations

• 6.7.1 Utilization Heat Map

• 6.7.2 Subsecond-Offset Heat Map

• 6.7.3 Flame Graphs

• 6.7.4 FlameScoped

• 6.8 Experimentation

• 6.8.1 Ad Hoc

• 6.8.2 SysBench

• 6.9 Tuning

• 6.9.1 Compiler Options

• 6.9.2 Scheduling Priority and Class

• 6.9.3 Scheduler Options

• 6.9.4 Scaling Governors

• 6.9.5 Power States

• 6.9.6 CPU Binding

• 6.9.7 Exclusive CPU Sets

• 6.9.8 Resource Controls

• 6.9.9 Security Boot Options

• 6.9.10 Processor Options (BIOS Tuning)

• 6.10 Exercises

• 6.11 References

7 Systems Performance Memory

7.1 Terminology

7.2 Concepts

7.2.1 Virtual Memory

7.2.2 Paging

7.2.3 Demand Paging

7.2.4 Overcommit

7.2.5 Process Swapping

7.2.6 File System Cache Usage

7.2.7 Utilization and Saturation

7.2.8 Allocators

7.2.9 Shared Memory

7.2.10 Working Set Size

7.2.11 Word Size

7.3 Architecture

7.3.1 Hardware

7.3.2 Software

7.3.3 Process Virtual Address Space

7.4 Methodology

7.4.1 Tools Method

7.4.2 USE Method

7.4.3 Characterizing Usage

7.4.4 Cycle Analysis

7.4.5 Performance Monitoring

7.4.6 Leak Detection

7.4.7 Static Performance Tuning

7.4.8 Resource Controls

7.4.9 Micro-Benchmarking

7.4.10 Memory Shrinking

7.5 Observability Tools

7.5.1 vmstat

7.5.2 PSI

7.5.3 swapon

7.5.4 sar

7.5.5 slabtop

7.5.6 numastat

7.5.7 ps

7.5.8 top

7.5.9 pmap

7.5.10 perf

7.5.11 drsnoop

7.5.12 wss

7.5.13 bpftrace

7.5.14 Other Tools

7.6 Tuning

7.6.1 Tunable Parameters

7.6.2 Multiple Page Sizes

7.6.3 Allocators

7.6.4 NUMA Binding

7.6.5 Resource Controls

7.7 Exercises

7.8 References

8 Systems Performance File Systems

8.1 Terminology

8.2 Models

8.2.1 File System Interfaces

8.2.2 File System Cache

8.2.3 Second-Level Cache

8.3 Concepts

8.3.1 File System Latency

8.3.2 Caching

8.3.3 Random vs. Sequential I/O

8.3.4 Prefetch

8.3.5 Read-Ahead

8.3.6 Write-Back Caching

8.3.7 Synchronous Writes

8.3.8 Raw and Direct I/O

8.3.9 Non-Blocking I/O

8.3.10 Memory-Mapped Files

8.3.11 Metadata

8.3.12 Logical vs. Physical I/O

8.3.13 Operations Are Not Equal

8.3.14 Special File Systems

8.3.15 Access Timestamps

8.3.16 Capacity

8.4 Architecture

8.4.1 File System I/O Stack

8.4.2 VFS

8.4.3 File System Caches

8.4.4 File System Features

8.4.5 File System Types

8.4.6 Volumes and Pools

8.5 Methodology

8.5.1 Disk Analysis

8.5.2 Latency Analysis

8.5.3 Workload Characterization

8.5.4 Performance Monitoring

8.5.5 Static Performance Tuning

8.5.6 Cache Tuning

8.5.7 Workload Separation

8.5.8 Micro-Benchmarking

8.6 Observability Tools

8.6.1 mount

8.6.2 free

8.6.3 top

8.6.4 vmstat

8.6.5 sar

8.6.6 slabtop

8.6.7 strace

8.6.8 fatrace

8.6.9 LatencyTOP

8.6.10 opensnoop

8.6.11 filetop

8.6.12 cachestat

8.6.13 ext4dist (xfs, zfs, btrfs, nfs)

8.6.14 ext4slower (xfs, zfs, btrfs, nfs)

8.6.15 bpftrace

8.6.17 Other Tools

8.6.18 Visualizations

8.7 Experimentation

8.7.1 Ad Hoc

8.7.2 Micro-Benchmark Tools

8.7.3 Cache Flushing

8.8 Tuning

8.8.1 Application Calls

8.8.2 ext4

8.8.3 ZFS

8.9 Exercises

8.10 References

9 Systems Performance Disks

9.1 Terminology

9.2 Models

9.2.1 Simple Disk

9.2.2 Caching Disk

9.2.3 Controller

9.3 Concepts

9.3.1 Measuring Time

9.3.2 Time Scales

9.3.3 Caching

9.3.4 Random vs. Sequential I/O

9.3.5 Read/Write Ratio

9.3.6 I/O Size

9.3.7 IOPS Are Not Equal

9.3.8 Non-Data-Transfer Disk Commands

9.3.9 Utilization

9.3.10 Saturation

9.3.11 I/O Wait

9.3.12 Synchronous vs. Asynchronous

9.3.13 Disk vs. Application I/O

9.4 Architecture

9.4.1 Disk Types

9.4.2 Interfaces

9.4.3 Storage Types

9.4.4 Operating System Disk I/O Stack

9.5 Methodology

9.5.1 Tools Method

9.5.2 USE Method

9.5.3 Performance Monitoring

9.5.4 Workload Characterization

9.5.5 Latency Analysis

9.5.6 Static Performance Tuning

9.5.7 Cache Tuning

9.5.8 Resource Controls

9.5.9 Micro-Benchmarking

9.5.10 Scaling

9.6 Observability Tools

9.6.1 iostat

9.6.2 sar

9.6.3 PSI

9.6.4 pidstat

9.6.5 perf

9.6.6 biolatency

9.6.7 biosnoop

9.6.8 iotop, biotop

9.6.9 biostacks

9.6.10 blktrace

9.6.11 bpftrace

9.6.12 MegaCli

9.6.13 smartctl

9.6.14 SCSI Logging

9.6.15 Other Tools

9.7 Visualizations

9.7.1 Line Graphs

9.7.2 Latency Scatter Plots

9.7.3 Latency Heat Maps

9.7.4 Offset Heat Maps

9.7.5 Utilization Heat Maps

9.8 Experimentation

9.8.1 Ad Hoc

9.8.2 Custom Load Generators

9.8.3 Micro-Benchmark Tools

9.8.4 Random Read Example

9.8.5 ioping

9.8.6 fio

9.8.7 blkreplay

9.9 Tuning

9.9.1 Operating System Tunables

9.9.2 Disk Device Tunables

9.9.3 Disk Controller Tunables

9.10 Exercises

9.11 References

10 Systems Performance Network

10.1 Terminology

10.2 Models

10.2.1 Network Interface

10.2.2 Controller

10.2.3 Protocol Stack

10.3 Concepts

10.3.1 Networks and Routing

10.3.2 Protocols

10.3.3 Encapsulation

10.3.4 Packet Size

10.3.5 Latency

10.3.6 Buffering

10.3.7 Connection Backlog

10.3.8 Interface Negotiation

10.3.9 Congestion Avoidance

10.3.10 Utilization

10.3.11 Local Connections

10.4 Architecture

10.4.1 Protocols

10.4.2 Hardware

10.4.3 Software

10.5 Methodology

10.5.1 Tools Method

10.5.2 USE Method

10.5.3 Workload Characterization

10.5.4 Latency Analysis

10.5.5 Performance Monitoring

10.5.6 Packet Sniffing

10.5.7 TCP Analysis

10.5.8 Static Performance Tuning

10.5.9 Resource Controls

10.5.10 Micro-Benchmarking

10.6 Observability Tools

10.6.1 ss

10.6.2 ip

10.6.3 ifconfig

10.6.4 nstat

10.6.5 netstat

10.6.6 sar

10.6.7 nicstat

10.6.8 ethtool

10.6.9 tcplife

10.6.10 tcptop

10.6.11 tcpretrans

10.6.12 bpftrace

10.6.13 tcpdump

10.6.14 Wireshark

10.6.15 Other Tools

10.7 Experimentation

10.7.1 ping

10.7.2 traceroute

10.7.3 pathchar

10.7.4 iperf

10.7.5 netperf

10.7.6 tc

10.7.7 Other Tools

10.8 Tuning

10.8.1 System-Wide

10.8.2 Socket Options

10.8.3 Configuration

10.9 Exercises

10.10 References

11 Systems Performance Cloud Computing

11.1 Background

11.1.1 Instance Types

11.1.2 Scalable Architecture

11.1.3 Capacity Planning

11.1.4 Storage

11.1.5 Multitenancy

11.1.6 Orchestration (Kubernetes)

11.2 Hardware Virtualization

11.2.1 Implementation

11.2.2 Overhead

11.2.3 Resource Controls

11.2.4 Observability

11.3 OS Virtualization

11.3.1 Implementation

11.3.2 Overhead

11.3.3 Resource Controls

11.3.4 Observability

11.4 Lightweight Virtualization

11.4.1 Implementation

11.4.2 Overhead

11.4.3 Resource Controls

11.4.4 Observability

11.5 Other Types

11.6 Comparisons

11.7 Exercises

11.8 References

12 Systems Performance Benchmarking

12.1 Background

12.1.1 Reasons

12.1.2 Effective Benchmarking

12.1.3 Benchmarking Failures

12.2 Benchmarking Types

12.2.1 Micro-Benchmarking

12.2.2 Simulation

12.2.3 Replay

12.2.4 Industry Standard]]s

12.3 Methodology

12.3.1 Passive Benchmarking

12.3.2 Active Benchmarking

12.3.3 CPU Profiling

12.3.4 USE Method

12.3.5 Workload Characterization

12.3.6 Custom Benchmarks

12.3.7 Ramping Load

12.3.8 Sanity Check

12.3.9 Statistical Analysis

12.3.10 Benchmarking Checklist

12.4 Benchmark Questions

12.5 Exercises

12.6 References

13 Systems Performance perf

13.1 Subcommands Overview

13.2 One-Liners

13.3 perf Events

13.4 Hardware Events

13.4.1 Frequency Sampling

13.5 Software Events

13.6 Tracepoint Events

13.7 Probe Events

13.7.1 kprobes

13.7.2 uprobes

13.7.3 USDT

13.8 perf stat

13.8.1 Options

13.8.2 Interval Statistics

13.8.3 Per-CPU Balance

13.8.4 Event Filters

13.8.5 Shadow Statistics

13.9 perf record

13.9.1 Options

13.9.2 CPU Profiling

13.9.3 Stack Walking

13.10 perf report

13.10.1 TUI

13.10.2 STDIO

13.11 perf script

13.11.1 Flame Graphs

13.11.2 Trace Scripts

13.12 perf trace

13.12.1 Kernel Versions

13.13 Other Commands

13.14 perf Documentation

13.15 References

14 Systems Performance Ftrace

14.1 Capabilities Overview

14.2 tracefs (/sys)

14.2.1 tracefs Contents

14.3 Ftrace Function Profiler

14.4 Ftrace Function Tracing

14.4.1 Using trace

14.4.2 Using trace_pipe

14.4.3 Options

14.5 Tracepoints

14.5.1 Filter

14.5.2 Trigger

14.6 kprobes

14.6.1 Event Tracing

14.6.2 Arguments

14.6.3 Return Values

14.6.4 Filters and Triggers

14.6.5 kprobe Profiling

14.7 uprobes

14.7.1 Event Tracing

14.7.2 Arguments and Return Values

14.7.3 Filters and Triggers

14.7.4 uprobe Profiling

14.8 Ftrace function_graph

14.8.1 Graph Tracing

14.8.2 Options

14.9 Ftrace hwlat

14.10 Ftrace Hist Triggers

14.10.1 Single Keys

14.10.2 Fields

14.10.3 Modifiers

14.10.4 PID Filters

14.10.5 Multiple Keys

14.10.6 Stack Trace Keys

14.10.7 Synthetic Events

14.11 trace-cmd

14.11.1 Subcommands Overview

14.11.2 trace-cmd One-Liners

14.11.3 trace-cmd vs. perf(1)

14.11.4 trace-cmd function_graph

14.11.5 KernelShark

14.11.6 trace-cmd Documentation

14.12 perf ftrace

14.13 perf-tools

14.13.1 Tool Coverage

14.13.2 Single-Purpose Tools

14.13.3 Multi-Purpose Tools

14.13.4 perf-tools One-Liners

14.13.5 Example

14.13.6 perf-tools vs. BCC/BPF

14.13.7 Documentation

14.14 Ftrace Documentation

14.15 References

15 Systems Performance BPF

15.1 BCC

15.1.1 Installation

15.1.2 Tool Coverage

15.1.3 Single-Purpose Tools

15.1.4 Multi-Purpose Tools

15.1.5 One-Liners

15.1.6 Multi-Tool Example

15.1.7 BCC vs. bpftrace

15.1.8 Documentation

15.2 bpftrace

15.2.1 Installation

15.2.2 Tools

15.2.3 One-Liners

15.2.4 Programming

15.2.5 Reference

15.2.6 Documentation

15.3 References

16 Systems Performance Case Study

16.1 An Unexplained Win

16.1.1 Problem Statement

16.1.2 Analysis Strategy

16.1.3 Statistics

16.1.4 Configuration

16.1.5 PMCs

16.1.6 Software Events

16.1.7 Tracing

16.1.8 Conclusion

16.2 Additional Information

16.3 References

A USE Method: Linux

B sar Summary

C bpftrace One-Liners

D Solutions to Selected Exercises

• E Systems Performance Who’s Who

• Systems Performance Glossary

• Systems Performance Index

Fair Use Sources:

• B08J5QZPNC (SysPrfBGrg 2021)