https://DevOpsCloud.io -- Cloud Monk Losang Jinpa, Ph.D., MCSE/MCT, GitOps DevOps Engineer

Systems Performance Index - is the MASTER.

DO indeed EDIT HERE.

Temporary Backup is Systems Performance, 2nd Edition, by Brendan Gregg Index

Return to Systems Performance Glossary, Systems Performance, 2nd Edition, Performance Bibliography, Systems Performance, Performance DevOps, IT Bibliography, DevOps Bibliography

“ (SysPrfBGrg 2021)

• Accelerated receive flow steering, 523

• Accelerators in USE method, 49

• accept system calls, 95

• Access timestamps, 371

• ACK detection in TCP, 512

• Actions in bpftrace, 769

• Active benchmarking]], 657–660

• Active listening in three-way handshakes, 511

• Active pages in page caches, 318

• Activities overview, 3–4

• Ad hoc checklist method, 43–44

• Adaptive mutex locks, 198

• Adaptive Replacement Cache (ARC), 381

• Address space, 304

• guests, 603

• kernel, 90

• memory, 304, 310

• processes, 95, 99–102, 319–322

• threads, 227–228

• virtual memory, 104, 305

• Address space layout randomization (ASLR), 723

• Advanced Format for magnetic rotational disks, 437

• AF_NETLINK address family, 145–146

• Agents

• monitoring software, 137–138

• product monitoring, 79

• AKS (Azure [[Kubernetes Service), 586

• Alerts, 8

• Algorithms

• caching, 36

• congestion control, 115, 118, 513–514

• big O notation, 175–176

• Allocation groups in XFS, 380

• Allocators

• memory, 309

• multithreaded applications, 353

• process virtual address space, 320–321

• Amazon EKS (Elastic Kubernetes Service]]), 586

• Amdahl’s Law of Scalability, 64–65

• Analysis

• benchmarking, 644–646, 665–666

• capacity planning, 38, 71–72

• drill-down, 55–56

• I/O traces, 478–479

• latency, 56–57, 384–386, 454–455

• off-CPU, 188–192

• resource, 38–39

• thread state, 193–197

• workload, 4–5, 39–40

• Analysis step in scientific method, 44–45

• Analysis strategy in case study]], 784

• annotate subcommand for perf, 673

• Anonymous memory, 304

• Anonymous paging, 305–307

• Anti-methods

• blame-someone-else, 43

• random change, 42–43

• streetlight, 42

• Apdex (application performance index), 174

• Application calls, tuning, 415–416

• Application I/O, 369, 435

• Application instrumentation in off-CPU analysis, 189

• Application internals, 213

• Application layer, file system latency in, 384

• Application performance index (Apdex), 174

• Applications, 171

• basics, 172–173

• big O notation, 175–176

• bpftrace for, 765

• common case optimization, 174

• common problems, 213–215

• exercises, 216–217

• internals, 213

• latency documentation, 385

• methodology. See Applications methodology

• missing stacks, 215–216

• missing symbols, 214

• objectives, 173–174

• observability, 174

• observability tools. See Applications observability tools

• performance techniques. See Applications performance techniques

• programming languages. See Applications programming languages

• references, 217–218

• Applications methodology

• CPU profiling, 187–189

• distributed tracing, 199

• lock analysis, 198

• off-CPU analysis, 189–192

• overview, 186–187

• static performance tuning, 198–199

• syscall analysis, 192

• thread state analysis, 193–197

• USE method, 193

• Applications observability tools

• bpftrace, 209–213

• execsnoop, 207–208

• offcputime, 204–205

• overview, 199–200

• perf, 200–203

• profile, 203–204

• strace, 205–207

• syscount, 208–209

• Applications performance techniques

• buffers, 177

• caching, 176

• concurrency and parallelism, 177–181

• I/O size selection, 176

• non-blocking I/O, 181

• Performance Mantras, 182

• polling, 177

• processor binding, 181–182

• Applications programming languages, 182–183

• compiled, 183–184

• garbage collection]], 184–185

• Interpreted, 184–185

• virtual machines, 185

• Appropriateness level in methodologies, 28–29

• ARC (Adaptive Replacement Cache), 381

• Architecture

• CPUs. See CPUs architecture

• disks. See Disks architecture

• file systems. See File [[systems architecture

• vs. loads, 581–582

• memory. See Memory architecture

• networks. See Networks architecture

• scalable, 581–582

• archive subcommand for perf, 673

• arcstat.pl tool, 410

• arg variables for bpftrace, 778

• argdist tool, 757–759

• Arguments

• kprobes, 152

• networks, 507

• tracepoints, 148–149

• uprobes, 154

• Arithmetic mean, 74

• Arrival process in queueing systems, 67

• ASG (auto scaling]] group)

• capacity planning, 72

• cloud computing, 583–584

• ASLR (address space layout randomization), 723

• Associativity in caches, 234

• Asynchronous disk I/O, 434–435

• Asynchronous interrupts, 96–97

• Asynchronous writes, 366

• atop tool, 285

• Auto scaling]] group (ASG)

• capacity planning, 72

• cloud computing, 583–584

• available_filter_functions file, 710

• Available swap, 309

• available_tracers file, 710

• Averages, 74–75

• avg function, 780

• await metric, 461

• Axes

• flame graphs, 10, 187, 290

• heat maps, 289, 410, 488–489

• line charts, 59, 80

• scalability tests, 62

• scatter plots, 81–82, 488

• Azure [[Kubernetes Service (AKS), 586

• Back-ends in instruction pipeline, 224

• Background color in flame graphs, 291

• Backlogs in network connections, 507, 519–520, 556–557, 569

• Bad paging, 305

• Balloon drivers, 597

• Bandwidth

• disks, 424

• interconnects, 237

• networks, 500, 508, 532–533

• OS virtualization, 614–615

• Bare-metal hypervisors, 587

• Baseline statistics, 59

• BATCH scheduling policy, 243

• BBR (Bottleneck Bandwidth and RTT) algorithm, 118, 513

• bcache technology, 117

• BCC (BPF Compiler Collection), 12

• vs. bpftrace, 760

• disks, 450

• documentation, 760–761

• installing, 754

• multi-purpose tools, 757

• multi-tool example, 759

• networks, 526

• one-liners, 757–759

• overview, 753–754

• vs. perf-tools, 747–748

• single-purpose tools, 755–757

• slow disks case study]], 17

• system-wide tracing, 136

• tool overview, 754–755

• bcc-tools tool package, 132

• BEGIN probes in bpftrace, 774

• bench subcommand for perf, 673

• Benchmark paradox, 648–649

• Benchmarket]]ing, 642

• Benchmarking, 641–642

• analysis, 644–646

• capacity planning, 70

• CPUs, 254

• effective, 643–644

• exercises, 668

• failures, 645–651

• industry standards, 654–656

• memory, 328

• micro-benchmarking. See Micro-benchmarking

• questions, 667–668

• reasons, 642–643

• references, 669–670

• replay, 654

• simulation, 653–654

• specials, 650

• SysBench system, 294

• types, 13, 651–656

• Benchmarking methodology

• active, 657–660

• checklist, 666–667

• CPU profiling, 660–661

• custom benchmarks, 662

• overview, 656

• passive, 656–657

• ramping load, 662–664

• sanity checks, 664–665

• statistical analysis, 665–666

• USE method, 661

• workload characterization, 662

• Berkeley Packet Filter (BPF), 751–752

• BCC compiler. See BCC (BPF Compiler Collection)

• bpftrace. See bpftrace tool

• description, 12–13

• extended. See Extended BPF

• iterator, 562

• JIT compiler, 117

• kernels, 92

• OS virtualization tracing, 620, 624–625, 629

• vs. perf-tools, 747–748

• program, 90

• Berkeley Software Distribution (BSD)]], 113

• BFQ (Budget Fair Queueing) I/O schedulers, 119, 449

• Big kernel lock (BKL) performance bottleneck, 116

• big O notation, 175–176

• Billing in cloud computing, 584

• Bimodal performance, 76

• Binary executable files, 183

• Binary translations in hardware virtualization, 588, 590

• Binding

• CPU, 253, 297–298

• NUMA, 353

• processor, 181–182

• bioerr tool, 487

• biolatency tool

• BCC, 753–755

• disks, 450, 468–470

• example, 753–754

• biopattern tool, 487

• BIOS, tuning, 299

• biosnoop tool

• BCC, 755

• disks, 470–472

• event tracing, 58

• hardware virtualization, 604–605

• outliers, 471–472

• queued time, 472

• system-wide tracing, 136

• biostacks tool, 474–475

• biotop tool

• BCC, 755

• disks, 450, 473–474

• Bit width in CPUs, 229

• bitesize tool

• BCC, 755

• perf-tools, 743

• blame command, 120

• Blame-someone-else anti-method, 43

• Blanco, Brenden, 753

• Blind faith benchmarking, 645

• blk tracer, 708
• blkio control group]], 610, 617

• blkreplay tool, 493

• blktrace tool

• action filtering, 478

• action identifiers, 477

• analysis, 478–479

• default output, 476–477

• description, 116

• disks, 475–479

• RWBS description, 477

• visualizations, 479

• Block-based file systems, 375–376

• Block caches in disk I/O, 430

• Block device interface, 109–110, 447

• Block I/O state in delay accounting, 145

• Block I/O times for disks, 427–428, 472

• Block interleaving, 378

• Block [[size

• defined, 360

• FFS, 378

• Block stores in cloud computing, 584

• Blue-green cloud computing deployments, 3–4

• Bonnie and Bonnie++ benchmarking tools

• active benchmarking]], 657–660

• file systems, 412–414

• Boolean expressions in bpftrace, 775–776

• Boot options, security, 298–299

• Boot-time tracing, 119

• Borkmann, Daniel, 121

• Borrowed virtual time (BVT) schedulers, 595

• Bottleneck Bandwidth and RTT (BBR) algorithm, 118, 513

• Bottlenecks

• capacity planning, 70–71

• complexity, 6

• defined, 22

• USE method, 47–50, 245, 324, 450–451

• BPF. See Berkeley Packet Filter (BPF)

• bpftrace tool, 12–13

• application internals, 213

• vs. BCC, 752–753, 760

• block I/O events, 625, 658–659

• description, 282

• disk I/O errors, 483

• disk I/O latency, 482–483

• disk I/O size, 480–481

• event sources, 558

• examples, 284, 761–762

• file system internals, 408

• hardware virtualization, 602

• I/O profiling, 210–212

• installing, 762

• lock tracing, 212–213

• malloc() bytes flame graph, 346

• memory internals, 346–347

• one-liners for CPUs, 283, 803–804

• one-liners for disks, 479–480, 806–807

• one-liners for file systems, 402–403, 805–806

• one-liners for memory, 343–344, 804–805

• one-liners for networks, 550–552, 807–808

• one-liners overview, 763–765

• package contents, 132

• packet inspection]], 526

• page fault flame graphs, 346

• programming. See bpftrace tool programming

• references, 782

• scheduling internals, 284–285

• signal tracing, 209–210

• socket tracing, 552–555

• stacks viewing, 450

• syscall tracing, 403–405

• system-wide tracing, 136

• TCP tracing, 555–557

• tracepoints, 149

• user allocation stacks, 345

• VFS tracing, 405–408

• bpftrace tool programming

• actions, 769

• comments, 767

• documentation, 781

• example, 766

• filters, 769

• flow [[control, 775–777

• functions, 770–772, 778–781

• Hello, World! program, 770

• operators, 776–777

• probe arguments, 775

• probe format, 768

• probe types, 774–775

• probe wildcards, 768–769

• program structure, 767

• timing, 772–773

• usage, 766–767

• variables, 770–771, 777–778

• BQL (Byte Queue Limits)

• driver queues, 524

• tuning, 571

• Branch prediction in instruction pipeline, 224

• Breakpoints in perf, 680

• brk system calls, 95

• brkstack tool, 348

• Broadcast]] network messages, 503

• BSD ([[Berkeley Software Distribution)]], 113

• b[[trace tool, 476, 478

• btrfs file system, 381–382, 399

• btrfsdist tool, 755

• btrfsslower tool, 755

• btt tool, 478

• Buckets

• hash table]]s, 180

• heat maps, 82–83

• Buddy allocators, 317

• Budget Fair Queueing (BFQ) I/O schedulers, 119, 449

• buf function, 778

• Buffer caches, 110, 374

• Bufferbloat, 507

• Buffers

• applications, 177

• block devices, 110, 374

• networks, 507

• ring, 522

• TCP, 520, 569

• bufgrow tool, 409
• Bug database systems

• applications, 172

• case studies, 792–793

• buildid-cache subcommand for perf, 673

• Built-in bpftrace variables, 770, 777–778

• Bursting in cloud computing, 584, 614–615

• Buses, memory, 312–313

• BVT (borrowed virtual time) schedulers, 595

• Bypass, kernel, 94

• Byte Queue Limits (BQL)

• driver queues, 524

• tuning, 571

• Bytecode, 185

• C, C++

• compiled languages, 183

• symbols, 214

• stacks, 215

• C-states in CPUs, 231

• c2c subcommand for perf, 673, 702

• Cache Allocation Technology (CAT), 118, 596

• Cache miss rate, 36

• Cache warmth, 222

• cachegrind tool, 135

• Caches and caching

• applications, 176

• associativity, 234

• block devices, 110, 374

• cache line size, 234

• coherency, 234–235

• CPUs, hardware virtualization, 596

• CPUs, memory, 221–222, 314

• CPUs, OS virtualization, 615–616

• CPUs, processors, 230–235

• CPUs, vs. GPUs, 240

• defined, 23

• dentry, 375

• disks, I/O, 430

• disks, on-disk, 437

• disks, tuning, 456

• file systems, flushing, 414

• file systems, OS virtualization, 613

• file systems, overview, 361–363

• file systems, tuning, 389, 414–416

• file systems, types, 373–375

• file systems, usage, 309

• inode, 375

• methodologies, 35–37

• micro-benchmarking test, 390

• operating systems, 108–109

• page, 315, 374

• perf events, 680

• RAID, 445

• tuning, 60

• write-back, 365

• cachestat tool

• file systems, 399, 658–659

• memory, 348

• perf-tools, 743

• slow disks case study]], 17

• Caching disk model, 425–426

• Canary testing, 3

• Capacity-based utilization, 34

• Capacity of file systems, 371

• Capacity planning

• benchmarking for, 642

• cloud computing, 582–584

• defined, 4

• factor analysis, 71–72

• micro-benchmarking, 70

• overview, 69

• resource analysis, 38

• resource limits, 70–71

• scaling solutions, 72–73

• CAPI (Coherent Accelerator Processor Interface), 236

• Carrier sense multiple access with collision detection (CSMA/CD) algorithm, 516

• CAS (column address strobe) latency, 311

• Cascading failures, 5

• Case studies

• analysis strategy, 784

• bug database systems, 792–793

• conclusion, 792

• configuration, 786–788

• PMCs, 788–789

• problem statement, 783–784

• references, 793

• slow disks, 16–18

• software change, 18–19

• software events, 789–790

• statistics, 784–786

• tracing, 790–792

• Casual benchmarking, 645

• CAT (Cache Allocation Technology), 118, 596

• cat function, 779

• CAT (Intel Cache Allocation Technology), 118, 596

• CFQ (completely fair queueing), 115, 449

• CFS (completely fair scheduler), 116–117

• CPU scheduling, 241

• CPU shares, 614–615

• description, 243

• cgroup file, 141

• cgroup variable, 778

• cgroupid function, 779

• cgroups

• block I/O, 494

• description, 116, 118

• Linux kernel, 116

• memory, 317, 353

• OS virtualization, 606, 608–611, 613–620, 630

• resource management, 111, 298

• statistics, 139, 141, 620–622, 627–628

• cgtop tool, 621

• Character devices, 109–110

• Characterizing memory usage, 325–326

• Cheating in benchmarking, 650–651

• Checklists

• ad hoc checklist method, 43–44

• benchmarking, 666

• CPUs, 247, 527

• disks, 453

• file systems, 387

• Linux 60-second analysis, 15

• memory, 325

• Chip-level multiprocessing (CMP), 220

• chrt command, 295

• Cilium, 509, 586, 617

• Circular buffers for applications, 177

• CISCs (complex instruction set computers), 224

• clang complier, 122

• Classes, scheduling

• CPUs, 242–243

• I/O, 493

• kernel, 106, 115

• priority, 295

• Clean memory, 306

• clear function in bpftrace, 780

• clear subcommand in trace-cmd, 735

• clock routine, 99

• Clocks

• CPUs, 223, 230

• CPUs vs. GPUs, 240

• operating systems, 99

• clone system calls, 94, 100

• Cloud APIs, 580

• Cloud computing, 579–580

• background, 580–581

• capacity planning, 582–584

• comparisons, 634–636

• vs. enterprise, 62

• exercises, 636–637

• hardware virtualization. See Hardware virtualization

• instance types, 581

• lightweight virtualization, 630–633

• multitenancy, 585–586

• orchestration, 586

• OS virtualization. See OS virtualization

• overview, 14

• PMCs, 158

• proof-of-concept testing, 3

• references, 637–639

• scalable architecture, 581–582

• storage, 584–585

• types, 634

• Cloud-[[native databases, 582

• Clue-based approach in thread state analysis, 196

• Clusters in cloud computing, 586

• CMP (chip-level multiprocessing), 220

• CNI (container network interface]]) software, 586

• Co-routines in applications, 178

• Coarse view in profiling, 35

• Code changes in cloud computing, 583

• Coefficient of variation (CoV), 76

• Coherence

• caches, 234–235

• models, 63

• Coherent Accelerator Processor Interface (CAPI), 236

• Cold caches, 36

• collectd agent, 138

• Collisions

• hash, 180

• networks, 516

• Colors in flame graphs, 291

• Column address strobe (CAS) latency, 311

• Column quantizations, 82–83

• comm variable in bpftrace, 778

• Comma-separated values (CSV) format for sar, 165

• Comments in bpftrace, 767

• Common case optimization in applications, 174
• Communication in multiprocess vs. multithreading, 228

• Community applications, 172–173

• Comparing benchmarks, 648

• Competition, benchmarking, 649

• Compiled programming languages

• optimizations, 183–184

• overview, 183

• Compilers

• CPU optimization, 229

• options, 295

• Completely fair queueing (CFQ), 115, 449

• Completely fair scheduler (CFS), 116–117

• CPU scheduling, 241

• CPU shares, 614–615

• description, 243

• Completion target in workload analysis, 39

• Complex benchmark tools, 646

• Complex instruction set computers (CISCs), 224

• Complexity, 5

• Comprehension in flame graphs, 249

• Compression

• btrfs, 382

• disks, 369

• ZFS, 381

• Compute kernel, 240

• Compute Unified Device Architecture (CUDA), 240

• Concurrency

• applications, 177–181

• micro-benchmarking, 390, 456

• CONFIG options, 295–296

• CONFIG_TASK_DELAY_ACCT option, 145

• Configuration

• applications, 172

• case study]], 786–788

• network options, 574

• Congestion avoidance and control

• Linux kernel, 115

• networks, 508

• TCP, 510, 513

• tuning, 570

• connect system calls, 95

• Connections for networks, 509

• backlogs, 507, 519–520, 556–557, 569

• characteristics, 527–528

• firewalls, 517

• latency, 7, 24–25, 505–506, 528

• life span, 507

• local, 509

• monitoring, 529

• NICs, 109

• QUIC, 515

• TCP queues, 519–520

• three-way handshakes, 511–512

• UDP, 514

• Container network interface]] (CNI) software, 586

• Containers

• lightweight virtualization, 631–632

• orchestration, 586

• observability, 617–630

• OS virtualization, 605–630

• resource controls, 52, 70, 613–617, 626

• Contention

• locks, 198

• models, 63

• Context switches

• defined, 90

• kernels, 93

• Contributors to system performance technologies, 811–814

• Control group]]s (cgroups). See cgroups

• Control paths in hardware virtualization, 594

• Control units in CPUs, 230

• Controllers

• caches, 430

• disk, 426

• mechanical]] disks, 439

• micro-benchmarking, 457

• network, 501–502, 516

• solid-state drives, 440–441

• tunable, 494–495

• USE method, 49, 451

• Controls, resource. See Resource controls

• Cookies, TCP, 511, 520

• Copy-on-write (COW) file systems, 376

• btrfs, 382

• ZFS, 380

• Copy-on-write (COW) process strategy, 100

• CoreLink Interconnects, 236

• Cores

• CPUs vs. GPUs, 240

• defined, 220

• Corrupted file [[system data, 365

• count function in bpftrace, 780

• Counters, 8–9

• fixed, 133–135

• hardware, 156–158

• CoV (coefficient of variation), 76

• COW (copy-on-write) file systems, 376

• btrfs, 382

• ZFS, 380

• COW (copy-on-write) process strategy, 100

• CPCs (CPU performance counters), 156

• CPI (cycles per instruction), 225

• CPU affinity, 222

• CPU-bound applications, 106

• cpu control group]], 610

• CPU mode for applications, 172

• CPU performance counters (CPCs), 156

• CPU registers, perf-tools for, 746–747

• cpu variable in bpftrace, 777

• cpuacct control group]], 610

• cpudist tool

• BCC, 755

• case study]], 790–791

• threads, 278–279

• cpufreq tool, 285

• cpuinfo tool, 142

• cpupower tool, 286–287

• CPUs, 219–220

• architecture. See CPUs architecture

• benchmark questions, 667–668

• binding, 181–182

• bpftrace for, 763, 803–804

• clock rate, 223

• compiler optimization, 229

• cross calls, 110

• exercises, 299–300

• experiments, 293–294

• feedback]]-directed optimization, 122

• flame graphs. See Flame graphs

• FlameScoped tool, 292–293

• garbage collection]], 185

• hardware virtualization, 589–592, 596–597

• I/O wait, 434

• instructions, defined, 220

• instructions, IPC, 225

• instructions, pipeline, 224

• instructions, size, 224

• instructions, steps, 223

• instructions, width, 224

• memory caches, 221–222

• memory tradeoffs with, 27

• methodology. See CPUs methodology

• models, 221–222

• multiprocess and multithreading, 227–229

• observability tools. See CPUs observability tools

• OS virtualization, 611, 614, 627, 630

• preemption, 227

• priority inversion, 227

• profiling. See CPUs profiling

• references, 300–302

• run queues, 222

• saturation, 226–227

• scaling in networks, 522–523

• schedulers, 105–106

• scheduling classes, 115

• simultaneous multithreading, 225

• statistic accuracy, 142–143

• subsecond-offset heat maps, 289

• terminology, 220

• thread pools, 178

• tuning. See CPUs tuning

• USE method, 49–51, 795–797

• user time, 226

• utilization, 226

• utilization heat maps, 288–289

• virtualization support, 588

• visualizations, 288–293

• volumes and pools, 383

• word size, 229

• CPUs architecture, 221, 229

• accelerators, 240–242

• associativity, 234

• caches, 230–235

• GPUs, 240–241

• hardware, 230–241

• idle threads, 244

• interconnects, 235–237

• latency, 233–234

• memory management units, 235

• NUMA grouping, 244

• P-states and C-states, 231

• PMCs, 237–239

• processors, 230

• schedulers, 241–242

• scheduling classes, 242–243

• software, 241–244

• CPUs methodology

• CPU binding, 253

• cycle analysis, 251

• micro-benchmarking, 253–254

• overview, 244–245

• performance monitoring, 251

• priority tuning, 252–253

• profiling, 247–250

• resource controls, 253

• sample processing, 247–248

• static performance tuning, 252

• tools method, 245

• USE, 245–246

• workload characterization, 246–247

• CPUs observability tools, 254–255

• bpftrace, 282–285

• cpudist, 278–279

• GPUs, 287

• hardirqs, 282

• miscellaneous, 285–286

• mpstat, 259

• perf, 267–276

• pidstat, 262

• pmcarch, 265–266

• profile, 277–278

• ps, 260–261

• ptime, 263–264

• runqlat, 279–280

• runqlen, 280–281

• sar, 260

• showboost, 265

• softirqs, 281–282

• time, 263–264

• tlbstat, 266–267

• top, 261–262

• turbostat, 264–265

• uptime, 255–258

• vmstat, 258

• CPUs profiling

• applications, 187–189

• benchmarking, 660–661

• perf, 200–201

• record, 695–696

• steps, 247–250

• system-wide, 268–270

• CPUs tuning

• compiler options, 295

• CPU binding, 297–298

• exclusive CPU sets, 298

• overview, 294–295

• power states, 297

• processor options, 299

• resource controls, 298

• scaling governors, 297

• scheduler options, 295–296

• scheduling priority and class, 295

• security boot options, 298–299

• Cpusets, 116

• CPU binding, 253

• exclusive, 298

• cpusets control group]], 610, 614, 627

• cpuunclaimed tool, 755

• Crash resilience, multiprocess vs. multithreading, 228

• Cr[[edit-based schedulers, 595

• Crisis tools, 131–133

• critical-chain command, 120

• Critical paths in systemd]] service manager, 120

• criticalstat tool, 756

• CSMA/CD (carrier sense multiple access with collision detection) algorithm, 516

• CSV (comma-separated values) format for sar, 165

• CUBIC algorithm for TCP congestion control, 513

• CUDA (Compute Unified Device Architecture), 240

• CUMASK values in MSRs, 238–239

• current_tracer file, 710

• curtask variable for bpftrace, 778

• Custom benchmarks, 662

• Custom load generators, 491

• Cycle analysis

• CPUs, 251

• memory, 326

• Cycles per instruction (CPI), 225

• Cylinder groups in FFS, 378

• Daily patterns, monitoring, 78

• Data [[Center TCP (DCTCP) congestion control, 118, 513

• Data deduplication in ZFS, 381

• Data integrity in magnetic rotational disks, 438

• Data paths in hardware virtualization, 594

• Data Plane Development Kit (DPDK), 523

• Data rate in throughput, 22

• Databases

• applications, 172

• case studies, 792–793

• cloud computing, 582

• Datagrams

• OSI model, 502

• UDP, 514

• DAX (Direct Access), 118

• dbslower tool, 756

• dbstat tool, 756

• Dcache (dentry cache), 375

• dcsnoop tool, 409

• dcstat tool, 409

• DCTCP (Data [[Center TCP) congestion control, 118, 513

• dd command

• disks, 490–491

• file systems, 411–412

• DDR SDRAM (double data rate synchronous dynamic random-access memory), 313

• Deadline I/O schedulers, 243, 448

• DEADLINE scheduling policy, 243

• DebugFS interface, 116

• Decayed average, 75

• Deflated disk I/O, 369

• Defragmentation in XFS, 380

• Degradation in scalability, 31–32

• Delay accounting

• kernel, 116

• off-CPU analysis, 197

• overview, 145

• Delayed ACKs algorithm, 513

• Delayed allocation

• ext4, 379

• XFS, 380

• delete function in bpftrace, 780

• Demand paging

• BSD kernel, 113

• memory, 307–308

• Dentry caches (dcaches), 375

• Dependencies in perf-tools, 748

• Development, benchmarking for, 642

• Development attribute, multiprocess vs. multithreading, 228

• Devices

• backlog tuning, 569

• disk I/O caches, 430

• drivers, 109–110, 522

• hardware virtualization, 588, 594, 597

• devices control group]], 610

• df tool, 409

• Dhrystone benchmark

• CPUs, 254

• simulations, 653

• Diagnosis cycle, 46

• diff subcommand for perf, 673

• Differentiated Services Code Points (DSCPs), 509–510

• Direct Access (DAX), 118

• Direct buses, 313

• Direct I/O, 366

• Direct mapped caches, 234

• Direct measurement approach in thread state analysis, 197

• Direct-reclaim memory method, 318–319

• Directories in file systems, 107

• Directory indexes in ext3, 379

• Directory name lookup cache (DNLC), 375

• Dirty memory, 306

• Disk commands, 424

• Disk controllers

• caches, 430

• magnetic rotational disks, 439

• tunable, 494–495

• USE method, 451

• Disk I/O state in thread state analysis, 194–197

• Disk request time, 428

• Disk response time]], 428

• Disk service time, 428–429

• Disk wait time, 428

• Disks, 423–424

• architecture. See Disks architecture

• exercises, 495–496

• experiments, 490–493

• I/O. See Disks I/O

• IOPS, 432

• latency analysis, 384–386

• methodology. See Disks methodology

• models. See Disks models

• non-data-[[transfer disk commands, 432

• observability tools. See Disks observability tools

• read/write ratio, 431

• references, 496–498

• resource controls, 494

• saturation, 434

• terminology, 424

• tunable, 494

• tuning, 493–495

• USE method, 451

• utilization, 433

• visualizations, 487–490

• Disks architecture

• interfaces, 442–443

• magnetic rotational disks, 435–439

• operating system disk I/O stack, 446–449

• persistent memory, 441

• solid-state drives, 439–441

• storage types, 443–446

• Disks I/O

• vs. application I/O, 435

• bpftrace for, 764, 806–807

• caching, 430

• errors, 483

• heat maps, 488–490

• latency, 428–430, 454–455, 467–472, 482–483

• operating system stacks, 446–449

• OS virtualization, 613, 616

• OS virtualization strategy, 630

• random vs. sequential, 430–431

• scatter plots, 488

• simple disk, 425

• size, 432, 480–481

• synchronous vs. asynchronous, 434–435

• time measurements, 427–429

• time scales, 429–430

• wait, 434

• Disks methodology

• cache tuning, 456

• latency analysis, 454–455

• micro-benchmarking, 456–457

• overview, 449–450

• performance monitoring, 452

• resource controls, 456

• scaling, 457–458

• static performance tuning, 455–456

• tools method, 450

• USE method, 450–451

• workload characterization, 452–454

• Disks models

• caching disk, 425–426

• controllers, 426

• simple disk, 425

• Disks observability tools, 484–486

• biolatency, 468–470

• biosnoop, 470–472

• biostacks, 474–475

• biotop, 473–474

• blktrace, 475–479

• bpftrace, 479–483

• iostat, 459–463

• iotop, 472–473

• MegaCli, 484

• miscellaneous, 487

• overview, 458–459

• perf, 465–468

• pidstat, 464–465

• PSI, 464

• sar, 463–464

• SCSI event logging, 486

• diskstats tool, 142, 487

• Dispatcher-queue latency, 222

• Distributed operating systems, 123–124

• Distributed tracing, 199

• Distributions

• multimodal, 76–77

• normal, 75

• dmesg tool

• CPUs, 245

• description, 15

• memory, 348

• OS virtualization, 619

• dmidecode tool, 348–349

• DNLC (directory name lookup cache), 375

• DNS latency, 24–25

• Docker 607, 620–622

• Documentation

• application latency, 385

• BCC, 760–761

• bpftrace, 781

• Ftrace, 748–749

• kprobes, 153

• perf, 276, 703

• perf-tools, 748

• PMCs, 158

• sar, 165–166

• trace-cmd, 740

• tracepoints, 150–151

• uprobes, 155

• USDT, 156

• Domains

• scheduling, 244

• Xen, 589

• Double data rate synchronous dynamic random-access memory (DDR SDRAM), 313

• Double-pumped data [[transfer for CPUs, 237

• DPDK (Data Plane Development Kit), 523

• DRAM (dynamic random-access memory), 311

• Drill-down analysis

• overview, 55–56

• slow disks case study]], 17

• Drivers

• balloon, 597

• device, 109–110, 522

• parameterized, 593–595

• drsnoop tool

• BCC, 756

• memory, 342

• DSCPs (Differentiated Services Code Points), 509–510

• D[[Trace tool

• description, 12

• Solaris kernel, 114

• Duplex for networks, 508

• Duplicate ACK detection, 512

• Duration in RED method, 53

• DWARF (debugging with attributed record formats) stack walking, 216, 267, 676, 696

• Dynamic instrumentation

• kprobes, 151

• latency analysis, 385

• overview, 12

• Dynamic priority in scheduling classes, 242–243

• Dynamic random-access memory (DRAM), 311

• Dynamic sizing in cloud computing, 583–584

• Dynamic tracers, 12

• Dynamic tracing

• D[[Trace, 114

• perf, 677–678

• tools, 12

• Dynamic USDT, 156

• DynTicks, 116

• e2fsck tool, 418

• Early Departure Time (EDT), 119, 524

• eBPF. See Extended BPF

• EBS (Elastic Block Store), 585

• ECC (error-correcting code) for magnetic rotational disks, 438

• ECN (Explicit Congestion Notification) field

• IP, 508–510

• TCP, 513

• tuning, 570

• EDT (Early Departure Time), 119, 524

• EFS (Elastic File [[System), 585

• EKS (Elastic Kubernetes Service]]), 586

• elasped variable in bpftrace, 777

• Elastic Block Store (EBS), 585

• Elastic File [[System (EFS), 585

• Elastic Kubernetes Service]] (EKS), 586

• Elevator seeking in magnetic rotational disks, 437–438

• ELF (Executable and Linking Format) binaries

• description, 183

• missing symbols in, 214

• Embedded caches, 232

• eMLC (enterprise multi-level cell) flash memory]], 440

• Encapsulation for networks, 504

• END probes in bpftrace, 774

• End-to-end network arguments, 507

• Enterprise models, 62

• Enterprise multi-level cell (eMLC) flash memory]], 440

• Environment

• benchmarking, 647

• processes, 101–102

• Ephemeral drives, 584

• Ephemeral ports, 531

• epoll system call, 115, 118

• EPTs (extended page tables), 593

• Erlang virtual machines, 185

• Error-correcting code (ECC) for magnetic rotational disks, 438

• Errors

• applications, 193

• benchmarking, 647

• CPUs, 245–246, 796, 798

• disk controllers, 451

• disk devices, 451

• I/O, 483, 798

• kernels, 798

• memory, 324–325, 796, 798

• networks, 526–527, 529, 796–797

• RED method, 53

• storage, 797

• task capacity, 799

• USE method overview, 47–48, 51–53

• user mutex, 799

• Ethernet congestion avoidance, 508

• ethtool tool, 132, 546–547

• Event-based concurrency, 178

• Event-based tools, 133

• Event-select MSRs, 238

• Event sources for Wireshark, 559

• Event tracing

• disks, 454

• file systems, 388

• Ftrace, 707–708

• kprobes, 719–720

• methodologies, 57–58

• perf-tools for, 745–746

• trace-cmd for, 737

• uprobes, 722–723

• Event worker threads, 178

• Events

• case study]], 789–790

• CPUs, 273–274

• frequency sampling, 682–683

• observability source, 159

• perf. See perf tool events

• SCSI logging, 486

• selecting, 274–275

• stat filters, 693–694

• synthetic, 731–733

• trace, 148

• events directory in tracefs, 710

• Eviction policies for caching, 36

• evlist subcommand for perf, 673

• Exceptions

• synchronous interrupts, 97

• user mode, 93

• Exclusive CPU sets, 298

• exec system calls

• kernel, 94

• processes, 100

• execsnoop tool

• BCC, 756

• CPUs, 285

• perf-tools, 743

• process tracing, 207–208

• static instrumentation, 11–12

• tracing, 136

• Executable and Linking Format (ELF) binaries

• description, 183

• missing symbols in, 214

• Executable data in process virtual address space, 319

• Executable text in process virtual address space, 319

• Execution in kernels, 92–93

• execve system call, 11

• exit function in bpftrace, 770, 779

• Experimentation-based performance gains, 73–74

• Experiments

• CPUs, 293–294

• disks, 490–493

• file systems, 411–414

• networks, 562–567

• observability, 7

• overview, 13–14

• scientific method, 45–46

• Experts for applications, 173

• Explicit Congestion Notification (ECN) field

• IP, 508–510

• TCP, 513

• tuning, 570

• Explicit logical metadata]] in file systems, 368

• Exporters for monitoring, 55, 79, 137

• Express Data Path (XDP) technology

• description, 118

• event sources, 558

• kernel bypass, 523

• ext3 file system, 378–379

• ext4 file system

• features, 379

• tuning, 416–418

• ext4dist tool, 399–401, 756

• ext4slower tool, 401–402, 756

• Extended BPF, 12

• BCC 751–761

• bpftrace 752–753, 761–781, 803–808

• description, 118

• firewalls, 517

• histograms, 744

• kernel-mode applications, 92

• overview, 121–122

• tracing tools, 166

• Extended page tables (EPTs), 593

• Extent-based file systems, 375–376

• Extents, 375–376

• btrfs, 382

• ext4, 380

• External caches, 232

• FaaS (functions as a service), 634

• FACK (forward acknowledgments) in TCP, 514

• Factor analysis in capacity planning, 71–72

• Failures, benchmarking, 645–651

• Fair-share schedulers, 595

• False sharing for hash table]]s, 181

• Families of instance types, 581

• Fast File [[System (FFS)

• description, 113

• overview, 377–378

• Fast open in TCP, 510

• Fast recovery in TCP, 510

• Fast retransmits in TCP, 510, 512

• Fast user-space mutex (Futex), 115

• Fastpath state in Mutex locks, 179

• fatrace tool, 395–396

• Faults

• in synchronous interrupts, 97

• page faults. See page faults

• faults tool, 348

• FC (Fibre Channel) interface, 442–443

• fd tool, 141

• Feedback]]-directed optimization (FDO), 122

• ffaults tool, 348

• FFS (Fast File [[System)

• description, 113

• overview, 377–378

• Fiber threads, 178

• Fibre Channel (FC) interface, 442–443

• Field-programmable gate arrays (FPGAs), 240–241

• FIFO scheduling policy, 243

• File descriptor capacity in USE method, 52

• File [[offset pattern, micro-benchmarking for, 390

• File stores in cloud computing, 584

• File [[system internals, bpftrace for, 408

• File [[systems

• access timestamps, 371

• ad hoc tools, 411–412

• architecture. See File [[systems architecture

• bpftrace for, 764, 805–806

• caches. See File [[systems caches

• capacity, OS virtualization, 616

• capacity, performance issues, 371

• exercises, 419–420

• experiments, 411–414

• hardware virtualization, 597

• I/O, logical vs. physical, 368–370

• I/O, non-blocking, 366–367

• I/O, random vs. sequential, 363–364

• I/O, raw and direct, 366

• I/O, stack, 107–108

• interfaces, 361

• latency, 362–363

• memory-mapped files, 367

• metadata]], 367–368

• methodology. See File [[systems methodology

• micro-benchmark tools, 412–414

• models, 361–362

• observability tools. See File [[systems observability tools

• operations, 370–371

• OS virtualization, 611–612

• overview, 106–107, 359–360

• paging, 306

• pre[[fetch, 364–365

• read-ahead, 365

• reads, micro-benchmarking for, 61

• record size tradeoffs, 27

• references, 420–421

• special, 371

• synchronous writes, 366

• terminology, 360

• tuning, 414–419

• types. See File [[systems types

• visualizations, 410–411

• volumes and pools, 382–383

• File [[systems architecture

• caches, 373–375

• features, 375–377

• I/O stacks, 107–108, 372

• VFS, 107, 373

• File [[systems caches, 361–363

• defined, 360

• flushing, 414

• hit ratio, 17

• OS virtualization, 616

• OS virtualization strategy, 630

• tuning, 389

• usage, 309

• write-back, 365

• File [[systems methodology

• cache tuning, 389

• disk analysis, 384

• latency analysis, 384–386

• micro-benchmarking, 390–391

• overview, 383–384

• performance monitoring, 388

• static performance tuning, 389

• workload characterization, 386–388

• workload separation, 389

• File [[systems observability tools

• bpftrace, 402–408

• cachestat, 399

• ext4dist, 399–401

• ext4slower, 401–402

• fatrace, 395–396

• filetop, 398–399

• free, 392–393

• LatencyTOP, 396

• miscellaneous, 409–410

• mount, 392

• opensnoop, 397

• overview, 391–392

• sar, 393–394

• slabtop, 394–395

• strace, 395

• top, 393

• vmstat, 393

• File [[systems types

• btrfs, 381–382

• ext3, 378–379

• ext4, 379

• FFS, 377–378

• XFS, 379–380

• ZFS, 380–381

• FileBench tool, 414

• filelife tool, 409, 756

• fileslower tool, 409

• filetop tool, 398–399

• filetype tool, 409

• Filters

• bpftrace, 769, 776

• event, 693–694

• kprobes, 721–722

• PID, 729–730

• tracepoints, 717–718

• uprobes, 723

• fio (Flexible IO Tester) tool

• disks, 493

• file systems, 413–414

• Firecracker project, 631

• Firewalls, 503

• misconfigured, 505

• overview, 517

• tuning, 574

• First-byte latency, 506, 528

• Five Whys in drill-down analysis, 56

• Fixed counters, 133–135

• Flame graphs

• automated, 201

• characteristics, 290–291

• colors, 291

• CPU profiling, 10–11, 187–188, 278, 660–661

• generating, 249, 270–272

• interactivity, 291

• interpretation, 291–292

• malloc() bytes, 346

• missing stacks, 215

• off-CPU time, 190–191, 205

• overview, 289–290

• page faults, 340–342, 346

• perf, 119

• performance wins, 250

• profiles, 278

• sample processing, 249–250

• scripts, 700

• FlameScoped tool, 292–293, 700

• Flash-memory-based SSDs, 439–440

• Flash translation layer (FTL) in solid-state drives, 440–441

• Flent (FLExible Network Tester) tool, 567

• Flexible IO Tester (fio) tool

• disks, 493

• file systems, 413–414

• FLExible Network Tester (Flent) tool, 567

• Floating point events in perf, 680

• floating-point operations per second (FLOPS) in benchmarking, 655

• Flow [[control in bpftrace, 775–777

• Flusher threads, 374

• Flushing caches, 365, 414

• fmapfault tool, 409

• Footprints, off-CPU, 188–189

• fork system calls, 94, 100

• forks.bt tool, 624–625

• Format string for tracepoints, 148–149

• Forward acknowledgments (FACK) in TCP, 514

• 4-wide processors, 224

• FPGAs (field-programmable gate arrays), 240–241

• Fragmentation

• FFS, 377

• file systems, 364

• memory, 321

• packets, 505

• reducing, 380

• Frames

• defined, 500

• networks, 515

• OSI model, 502

• Free memory lists, 315–318

• free tool

• description, 15

• file systems, 392–393

• memory, 348

• OS virtualization, 619

• FreeBSD

• jails, 606

• je[[malloc, 322

• kernel, 113

• TSA analysis, 217

• network stack, 514

• performance vs. Linux, 124

• TCP LRO, 523

• Freeing memory, 315–318

• Frequency sampling for hardware events, 682–683

• Front-ends in instruction pipeline, 224

• Front-side buses, 235–237

• fsck time in ext4, 379

• fsrwstat tool, 409

• FTL (flash translation layer) in solid-state drives, 440–441

• ftrace subcommand for perf, 673

• Ftrace, 13, 705–706

• capabilities overview, 706–708

• description, 166

• documentation, 748–749

• function_graph, 724–725

• function profiler, 711–712

• function tracer, 713–716

• hist triggers, 727–733

• hwlat, 726

• kprobes, 719–722

• options, 716

• OS virtualization, 629

• perf, 741

• perf-tools, 741–748

• references, 749

• trace-cmd, 734–740

• trace file, 713–715

• trace_pipe file, 715

• tracefs, 708–711

• tracepoints, 717–718

• tracing, 136

• uprobes, 722–723

• Full I/O distributions disk latency, 454

• Full stack in systems performance, 1

• Fully associative caches, 234

• Fully-preemptible kernels, 110, 114

• func variable in bpftrace, 778

• funccount tool

• BCC, 756–758

• example, 747

• perf-tools, 744, 748

• funcgraph tool

• Ftrace, 706–707

• perf-tools, 744, 748

• funclatency tool, 757

• funcslower tool

• BCC, 757

• perf-tools, 744

• function_graph tracer

• description, 708

• graph tracing, 724–725

• options, 725

• trace-cmd for, 737, 739

• function_profile_enabled file, 710

• Function profiling

• Ftrace, 707, 711–712

• observability source, 159

• Function tracer. See Ftrace tool

• Function tracing

• profiling, 248

• trace-cmd for, 736–737

• Functional block diagrams in USE method, 49–50

• Functional units in CPUs, 223

• Functions as a service (FaaS), 634

• Functions in bpftrace, 770, 778–781

• functrace tool, 744

• Futex (fast user-space mutex), 115

• futex system calls, 95

• Garbage collection]], 185–186

• gcc compiler

• optimizations, 183–184

• PGO kernels, 122

• gdb tool, 136

• Generic segmentation offload (GSO) in networks, 520–521

• Generic system performance methodologies, 40–41

• Geometric mean, 74

• getdelays.c tool, 286

• gethostlatency tool, 561, 756

• github.[[com tool package, 132

• GKE (Google Kubernetes Engine), 586

• glibc allocator, 322

• Glossary of terms, 815–823

• Golang

• goroutines, 178

• syscalls, 92

• Good/fast/cheap trade-off]]s, 26–27

• Google Kubernetes Engine (GKE), 586

• Goroutines for applications, 178

• gprof tool, 135

• Grafana, 8–9, 138

• Graph tracing, 724–725

• Graphics processing units (GPUs)

• vs. CPUs, 240

• tools, 287

• GRO (Generic Receive Offload), 119

• Growth

• big O notation, 175

• heap, 320

• memory, 185, 316, 327

• GSO (generic segmentation offload) in networks, 520–521

• Guests

• hardware virtualization, 590–593, 596–605

• lightweight virtualization, 632–633

• OS virtualization, 617, 627–629

• gVisor project, 631

• Hard disk drives (HDDs), 435–439

• Hard interrupts, 282

• hardirqs tool, 282, 756

• Hardware

• memory, 311–315

• networks, 515–517

• threads, 220

• tracing, 276

• Hardware-assisted virtualization, 590

• Hardware counters. See Performance monitoring counters (PMCs)

• Hardware events

• CPUs, 273–274

• frequency sampling, 682–683

• perf, 680–683

• selecting, 274–275

• Hardware instances in cloud computing, 580

• Hardware interrupts, 91

• Hardware latency detector (hwlat), 708, 726

• Hardware latency tracer, 118

• Hardware probes, 774

• Hardware RAID, 444

• Hardware resources in capacity planning, 70

• Hardware virtualization

• comparisons, 634–636

• CPU support, 589–592

• I/O, 593–595

• implementation, 588–589

• memory mapping, 592–593

• multi-tenant contention, 595

• observability, 597–605

• overhead, 589–595

• overview, 587–588

• resource controls, 595–597

• Harmonic mean, 74

• Hash fields in hist triggers, 728

• Hash table]]s in applications, 180–181

• HBAs (host bus adapters), 426

• HDDs (hard disk drives), 435–439

• hdparm tool, 491–492

• Head-based sampling in distributed tracing, 199

• Heads in magnetic rotational disks, 436

• Heap

• anonymous paging, 306

• description, 304

• growth, 320

• process virtual address space, 319

• Heat maps

• CPU utilization]], 288–289

• disk offset, 489–490

• disk utilization, 490

• file systems, 410–411

• FlameScoped, 292–293

• I/O latency, 488–489

• overview, 82–83

• subsecond-offset, 289

• Hello, World! program, 770

• hfaults tool, 348

• hist function in bpftrace, 780

• Hist triggers

• fields, 728–729

• modifiers, 729

• multiple keys, 730

• perf-tools, 748

• PID filters, 729–730

• single keys, 727–728

• stack trace keys, 730–731

• synthetic events, 731–733

• usage, 727

• hist triggers profiler, 707

• Histogram, 76–77

• Hits, cache, 35–36, 361

• Hold times for locks, 198

• Holistic approach, 6

• Horizontal pod autoscalers (HPAs), 73

• Horizontal scaling and scalability

• capacity planning, 72

• cloud computing, 581–582

• Host bus adapters (HBAs), 426

• Hosts

• applications, 172

• cloud computing, 580

• hardware virtualization, 597–603

• lightweight virtualization, 632

• OS virtualization, 617, 619–627

• Hot caches, 37

• Hot/cold flame graphs, 191

• Hourly patterns, monitoring, 78

• HPAs (horizontal pod autoscalers), 73

• HT (HyperTransport) for CPUs, 236

• htop tool, 621

• HTTP/3 protocol, 515

• Hubs in networks, 516

• Hue in flame graphs, 291

• Huge pages, 115–116, 314, 352–353

• hugetlb control group]], 610

• hwlat (hardware latency detector), 708, 726

• Hybrid clouds, 580

• Hybrid kernels, 92, 123

• Hyper-Threading Technology, 225

• Hyper-V, 589

• Hypercalls in paravirtualization, 588

• Hyperthreading-aware scheduling classes, 243

• HyperTransport (HT) for CPUs, 236

• Hypervisors

• cloud computing, 580

• hardware virtualization, 587–588

• kernels, 93

• Hypothesis step in scientific method, 44–45